Hi @r00t ,

I'm Sadia, a Product Manager at 1Password, and have some news that may be interesting to you. I am looking for some developers and administrators that would be interested in chatting with me about a new feature our team has been working on: Service Accounts. Earlier this year, we introduced the CLI 2.0, where users can use “run” and “inject” commands to substitute secret references for secrets stored in 1Password vaults. With our new Service Account capabilities, organizations can use a separate non-user account to control and manage access to secrets without deploying additional services like Connect.

We are currently building out service accounts and want to understand your pain-points and experiences with secrets management, and gather some feedback, so we could deliver the best product for our customers.

If you are interested, please feel free to reach out to me at sadia.azmal@agilebits.com or sign-up for a 30 minute slot on Calendly. I look forward to hearing from you :)

💙

Do you have examples of scenarios where you ran into hard to debug problems?

While testing my script, I had a variable already set in my env without me realising it, and then was wondering why op seemed to fetch wrong value to it. :smile:

Do you agree that OS envvars should be opt-out or do you think it'd be better for them to be opt-in?

I think that the most common usage should dictate the defaults. Normally at least I would expect that already set variables override the ones from a config file (be that .env in this case).

Let's see where to go with the direnv extension. I'm more than happy to have all possible cases handled by op and maintained by you. :)
Maybe just revert to using just op read and support fetching one var at a time. Even though over-engineering is fun.

Thanks again for all the info and ideas. I also like Makefiles. And service accounts, 1Password Connect, SCIM, and whatnot really add options to choose from.

https://github.com/tmatilai/direnv-1password

🎉

I guess the OS env var issue has to be addressed in some way or the other. That can lead to hard-to-debug problems now.

Do you have examples of scenarios where you ran into hard to debug problems? Do you agree that OS envvars should be opt-out or do you think it'd be better for them to be opt-in?

I'm also planning to add opt-in sign-in support.

Just a heads up that there's a couple of changes coming up to the way op signin is used together with the new Biometric Unlock functionality. If you have it enabled, when you're not signed in, using any op command will prompt you to log in using your biometrics, so there's no need to have a separate op signin action in these cases. We're looking into adding the same behavior when you don't have the biometrics integration enabled.

So that'd mean you'll get the sign-in added to the direnv integration without any changes needed :)

the aim is that less and less secrets are needed on the workstations at all, and deployments, inter-service authentication, etc. are handled automatically

💯

We're planning on making the CLI available in these automated settings as well. op run, op inject andop read` already support using 1Password Connect (I see that's hidden in the https://developer.1password.com/docs/cli/reference/commands/run/, and there's no guide on using Connect for provisioning secrets using the CLI yet. I'll make sure we add more docs on that), and we're exploring adding Service Accounts. Here's a https://1password.community/discussion/125815 on authenticating these workflows. I'd love to hear your thoughts on that!

Besides this "raw" use of the CLI, we're also building integrations on top of it that integrate natively with workflows such as CI providers and orchestrators. Currently https://github.com/1Password/load-secrets-action and https://github.com/1Password/onepassword-operator are currently supported.

That said, just like you said I imagine there will continue to be a need to run some things locally. For example, if your developing an app that integrates with third-party services, you'll probably want to be able to run it locally with credentials to a development environment of those third-party services passed to the apps runtime. When doing incident response, you may need credentials to connect to the servers you're looking after.

The main issue with op run (or aws-vault which we had to use before Terraform supported AWS SSO natively) is that it is extra wrapper, which has to be remembered to type or scripted, and one extra component to debug and track issues with versions etc. Especially for a fast growing organisation.

I think this is one of those cases again where it's good to have multiple options. Depending on your workflow, one or the other fits better.

Personally, I'm a huge fan of Makefiles. They self-document what actions you can take in a repo and what commands to use for them. For fast growing organizations I really like the on-boarding aspect of that self-documentation. It also helps with standardizing, even beyond programming language barriers.
Using Makefiles, you don't even notice the new wrapper until you look in the contents of the file.

In your case, I imagine the dotenv integration or maybe using the https://github.com/1Password/terraform-provider-onepassword (backed by 1Password Connect) fits better?

OK, here is the first public version: https://github.com/tmatilai/direnv-1password
I'm more than happy to pass the ownership to you when things are a bit more stable. :)

I guess the OS env var issue has to be addressed in some way or the other. That can lead to hard-to-debug problems now. I'm also planning to add opt-in sign-in support. More suggestions and comments from everyone are of course warmly welcome.

With Terraform and other secrets, I'm happy to have multiple options available. All the improvements in 1Password are very nice, and of course can have impact on the setup. But of course the aim is that less and less secrets are needed on the workstations at all, and deployments, inter-service authentication, etc. are handled automatically. Still there will always be some personal secrets needed.

The main issue with op run (or aws-vault which we had to use before Terraform supported AWS SSO natively) is that it is extra wrapper, which has to be remembered to type or scripted, and one extra component to debug and track issues with versions etc. Especially for a fast growing organisation. But let's see how things evolve. :)

Thanks a lot for providing such a detailed answer! Sweet setup you've build there :sunglasses:

Sharing the environment with colleagues

I can't speak for my colleagues, as everyone uses direnv like they wish.

One advantage of secret references we had seen at SecretHub, (we had build something similar for SecretHub before https://blog.1password.com/secrethub-acquisition/), was that by replacing the plaintext secrets with references, the environment file now no longer needs to be kept secret. If you want this (and this is completely optional!), you could check in the .env file in source control together with your code. These are a couple of advantages users saw in that:

1. No need to manually update the environment files anymore when a colleague makes an environment change. With any pull of the source code, the corresponding environment file is pulled as well.
2. Your project is reproducible. If a colleague runs terraform plan or terraform apply on the same commit, they'll get the same results, as they'll use the same secrets.
3. It's easier for new colleagues to join the project, as they'll immediately have their setup configured when cloning the repo.
4. When introducing a change in your Terraform configuration that requires an environment variable to be set, you commit the secret reference along with it. And even the secrets used are now covered by any reviews you do on code changes.

Using op run

Running Terraform through op run doesn't feel like optimal, as not all terraform runs need secrets, and expiring 1password session is annoying, especially with long passphrase and without Touch ID

I'd love to learn more about what we can do to help.

What's the downside if op run runs, but doesn't add any secrets? Is there anything besides the expiring session? How's the speed of op run? Is that a concern? Any other downsides?

And when we have touch ID support ready, would you use op run or continue to use direnv?

Is it a concern to you that environment variables continue to be available to other commands that you run in your terminal in-between invocations of the Terraform commands?

Integrating with dotenv

Like you said, this is fun!

We had the idea of a dotenv integration tracked internally to consider looking into when some bandwidth frees up, so you can imagine it was pretty awesome to hear you are already using 1Password like this! :chuffed:

I think it'd indeed be awesome if we can publish the result for others to be able to use it as well. I'd love to eventually put it on https://github.com/1password/ (if more than a one-liner is required) and https://developer.1password.com so it will be easy for anyone to find.

Implementation

It looks like https://github.com/direnv/direnv/blob/cb5222442cb9804b1574954999f6073cc636eff0/man/direnv-stdlib.1.md#direnv_load-command-generating-dump-output requires the passed argument to be in the format of the shell used. I think it'd be great if we can re-use the functionality already built into direnv dump for that.

I think we'll want to make the sources used by op run further configurable. Specifically, we could make it possible to opt-out of passing the OS environment and use just the environment file(s).

Yeah, I totally understand the spirit of the EA. Better to aim for the best UI, no matter how many changes still.

I can't speak for my colleagues, as everyone uses direnv like they wish. Or don't use it. But I have found it to be really, really great for setting environment based on the directory. At the moment I mostly work with Terraform with wide variety of providers. And while waiting for a real CD pipeline, Terraform is normally run on my own machine. Most common variables to set are e.g. AWS_PROFILE for selecting the role and account, and different API secrets (from 1Password). Sometimes also Ansible Vault passphrases and what not.

Running Terraform through op run doesn't feel like optimal, as not all terraform runs need secrets, and expiring 1password session is annoying, especially with long passphrase and without Touch ID ;)
The variables set with direnv stay in the shell session (until changing directory out of the hierarchy). So far I've had something like this in an .envrc (requiring that op session is valid):

bash
source_up
export FOO="$(op get item --vault VAULT --item ITEM --fields FIELD)"


OK, now starts the fun part. :p

On the weekend I already made first version of a direnv extension. Based on v1 I started indeed with op read, but then thought that it might be more efficient to fetch all secrets with just one op call, as there seems to be significant latency.

So the next idea was to use op run similar to this:

bash
direnv_load op run --env-file=/dev/stdin -- direnv dump <<OP
FOO=op://vault/item/field
# ...
OP


So op would resolve the secrets and add the variables to the env, which is then dumped and read by direnv. But there are (at least) two problems:

1. OP_SESSION_* variables are removed from the run env, which leads direnv to unset them from the real shell session. But no problem, I can save them before the command and then export them back.
2. Existing env vars are not overridden with the --env-file vars (which is wanted in the real use case for the op run command). Unsetting all the variables would require parsing them, which I wanted to avoid (for v2).

Another smaller downside of op run in this case is that it needlessly scans through all the environment variables to find op:// refs, even though I only want processing for the --env-file ones.

So now I'm thinking using op inject by feeding the variables though it. That would only process the needed ones, but would require passing the variables in certain structure and either evaluating the output, or parse and export the vars one-by-one.

I'm happy to hear your thoughts on this. :)

I'll try to publish the code soon. I work on this mostly on my own time, so depends on the life, too.

That sounds great! Thanks for sharing. One note about the migration: although I don't foresee any big changes coming up, the version 2 contract is still in beta. We may still change things based on all of your feedback when needed. When the Early Access period ends and we ship a v2.0.0, you can expect us to keep all commands backwards compatible in all subsequent v2 releases.

I'd love to learn more about your direnv setup. What do you use them for? And what do your .env/.envrc files currently look like?

Have you had the chance to try out the new https://developer.1password.com/docs/cli/secrets-environment-variables support? Would that be useful for your direnv use cases as well? op run -- $SHELL will open a shell with the secrets passed as defined in your .env file and secret references resolved.

I haven't looked in the details yet, but it looks like integration with direnv is possible:

It’s also possible to create your own extensions by creating a bash file at ~/.config/direnv/direnvrc or ~/.config/direnv/lib/*.sh. This file is loaded before your .envrc and thus allows you to make your own extensions to direnv.

Source: https://direnv.net/

So maybe we can make it resolve 1Password secret references as well (e.g. using op read).

The CLI users are normally technical enough, so I would guess that upgrading to the new version is not an issue and could be done fast. But finding and fixing all the scripts and especially direnv (example) files is still work. So better if we can start adding this kind of migration support right now.

What's your overall impression on the migration @r00t ? Do you foresee any challenges in getting your organization migrated?

I'm also curious to learn how long you expect it will take for everyone in your organization to upgrade and how long you will be running both versions? As Horia also mentions, 2.0 is the way forward. Of course, it's important to us that you have sufficient time to migrate to the new version, so we'll continue to support version 1 with critical bug-fixes and security updates for some time after we've launched the stable version of 2.0. We're currently thinking 1 year will be enough, and I'd love to hear your thoughts and learn if this will work for you.