Your client has been rate-limited

Hey kovpack , I omitted to mention that the Service Account feature is still in beta and is available for use on an invite base only. If you'd like to already try it out in beta, make sure to check out this thread (https://1password.community/discussion/131233/join-our-beta-test-for-1password-service-accounts-launching-mid-july#latest) here on the community forums. The last couple of replies are users who are were recently interested in joining the beta group.

IMPORTANT: Besides this, we have taken a look at your initial problem and we realised that op item list --long --format json might get all the info you want in one request only.

Best,
Andi

andi_t_1P cool, thank you.

I was going to try out your Service Accounts (this seems to be a nice thing I need). Unfortunately, I literally don't have it in the "Integrations" section anywhere, as you have described on https://developer.1password.com/docs/service-accounts/.

I do have integrations for events reporting, user provisioning, secrets automation (but Connect server setup only), etc. I definitely don't have a "Service Account" anywhere on any page from the Integrations section. How can I check it out?

Thank you for the feedback kovpack. In our latest version (2.6.0) we have enabled caching by default as well as brought some improvements to the caching functionality. In the near future, multiple types of commands such as create/edit/delete are to use the cache therefore reducing the server load in order to avoid these rate limits. We are investing effort in batching the requests for secret retrieval which should decrease the server load even further. Finally, you can keep an eye on or even try out our new Service Accounts (https://developer.1password.com/docs/service-accounts/ ) which are meant to be a nice alternative to Connect. Feel free to leave feedback regarding SA as well!

In the meantime, I will look into rate limiting documentation!

All the best,
Andi

OK, decided to test a Connect Server. And got disappointed again :(

The first problem:
- I can't even test a Docker container on M1 (it simply does not work). I suppose this problem was not addressed. 1 year has already passed.

Bunch of other problems:
- Connect Server API is very limited and does not provide all the info we need
- which means we'll have to use both the Connect Server & CLI app to get all the information we need
- which also means we'll have to maintain 2 servers now
- as you have extremely low request rates (which are not even documented)
- this means even if we use CLI, sooner or later we'll hit limits even when getting permissions for our vaults or access rights for users (based on things I've seen above - this will happen very soon, as the number of vaults grows over time)

Oh, come on...

I've just tested the rate limit.
- Total requests made until rate-limiting: 410 requests
- Time: 17 minutes (all requests were equally distributed, meaning around 24 requests per minute).

The second test showed the same numbers. Making requests faster will lead to faster blocking.

And Your client has been rate-limited means you are temporarily BLOCKED, not rate-limited. I was not able to make requests for at least 6 minutes after that error (and just decided to give up, so not sure for how long I was blocked). I still remember a case when I got blocked for 40 minutes.

This means that with the current CLI v2 we are not able to rebuild the functionality we need (and we had with CLI v1).

Not cool, not cool :(

I have a similar problem and got this (429) Too Many Requests: Too many requests. Your client has been rate-limited. when trying to get details of each item. It's not documented anywhere (at least I was not able to find the documentation).

Before fetching each item separately, I listed them all with --cache option. Then started getting details of each item (making a 2-seconds pause between each request just to be safe). Someone on the forum told me some time ago, that this should not make a real call (thus be rate-limited), but this does not seem to be true at all. In CLI 1 the information I need now was available in a list command, but now in CLI 2 I have to make a separate request per item to get the username, password, and URL for a LOGIN item. Basically, upgrading to CLI 2 broke a lot of things for us and made 1password less useful and more expensive.

Once a day I need to make a copy of 1password vaults, users, items & their permissions (we use this data internally and build some monitors and overview dashboards of permissions, which is not possible with the web UI at all). This is fine, as all the information can be fetched with a few list calls and a few dozens of calls when iterating over entities.

But we also need extra data (username, password & URL) to track item duplicates, password change/rotation dates, etc. Though, this will be around 800 requests more (maybe, even more in the future).

Setting up a Connect Server for us means +$300 to our monthly bills, which are already pretty high for a company of almost 150 employees (currently on a Business plan). Our InfoSec says, "you already did more for 1password, than 1password did for you", so we may soon consider alternatives.

If rate limits would be documented, this will at least give us enough information to make a daily sync possible not violating rate limits, and will give us the benefits we had before the CLI 2 release. Yes, this sync will take ages, but at least it will not decrease the value of 1password that much.

Yes, reducing the concurrency and updating to the latest CLI version has seemed to have mitigated the issue.

Thank you for adding the feature request.

Thank you @kormoc for taking the time to so clearly describe what it is you need. I'll make sure to track this request in our internal issue tracker.

For the time being and your immediate needs, did the following work for you?

I was running with 16 thread concurrency, which I presume is why I hit the rate limiter? I'll reduce the concurrency and see if that fixes the issue.

So the clients all sync state, and there's a ton of things that could happen that could cause every client to sync and remove everything. Database failures, corruptions, being hacked, ransomware, billing issues, access issues. Sure, they're unlikely, but this is my entire digital existence in one place. It would take literally months of my time to recover, and some things will not be able to be recovered at all.

I have no insight into your backup processes, nor does it really matter, cause at the end of the day, I need to take personal responsibility that I'm not relying on anyone else to ensure my data that I need is available.

This wasn't a problem with local vaults, I could just back them up in a way that makes me feel safe and secure.

But now we have the cloud. I can't control the cloud, and the app no longer makes backup records that are restorable.

So I'm left with needing to store a mirrored copy of the data in some other format I trust won't be impacted by the same catastrophic event that broke my 1p usage.

Self hosted is certainly a solution. That gives me the ability to backup and restore the raw data. But that doesn't exist right now.

Connect might be an option, but it's a lot of money (as I have too many vaults) for something that should be a basic feature (offsite backups). It also won't backup personal/private vaults, so my other family members aren't protected.

be sure that it's available

I'm thinking about this some more. What kind of events are you thinking about that could make the items unavailable? I was first thinking 1password.com downtime (which is historically quite rare: https://1password.statuspage.io/), but I just realized that the 1Password client apps will continue to have your items available, also in the event that 1password.com cannot be reached. You can confirm this yourself by disabling your internet connection and then unlocking your vault and checking out items. Are there any other events you're concerned about?