How to secure AI adoption and keep token costs under control

Organizations have spent the past few years encouraging employees to experiment with AI and build new workflows that increase their productivity. But those marching orders have created two challenges: ensuring people use approved tools and keeping AI spend under control.

The latter is particularly difficult when employees are “tokenmaxxing”: using AI heavily, often to signal productivity and AI expertise at work, without any regard for cost.

How can businesses support AI experimentation without losing sight of robust security and financial discipline? Amy Gebhardt, Senior Engineering Manager for 1Password SaaS Manager, shared some answers on Random but Memorable, 1Password’s award-winning podcast.

Read on to learn how visibility, budgets, and alerts can create shared understanding, support thoughtful experimentation, and help everyone choose the right model for the right task.

Can't see the video? Watch on YouTube.

Editor’s note: This interview has been lightly edited for clarity and brevity.

Nick Summers: Let’s start with “tokenmaxxing.” What does it mean, and why is it becoming a concern for businesses?

Amy Gebhardt: Tokenmaxxing is a kind of proxy for productivity or proof that folks are engaging with AI. 

I'm a Senior Engineering Manager now, but I’ve been a software engineer for roughly 20 years. What's different about this particular paradigm shift is the pressure to adopt. It’s pretty different from other things that we’ve seen in the technology industry. 

Leaders want to measure that their teams are actually embracing and engaging with the technology. And so for the last couple of years, that's been via the concept of tokenmaxxing, which means people are using AI to the point where there are no budgets or limitations. Employees are being told to go to town and use all of the tokens as proof that they’re engaging with the tool. The thought process being that if you use a lot of tokens, you must be embracing this new technology.

But there are obviously some issues with this. A big one is that tokenmaxxing doesn't tell the whole story. Eventually, leaders need to be able to measure and understand more.

NS: The clue is in the name: tokens. How do enterprises track AI spending, and how does that differ from the predictable, per-user subscription model of the past?

AG: Tokens make it sound like we're at an arcade and just playing a silly game. We don't necessarily connect tokens to the real financial impact that's associated with them. That makes it easier to spend them without thinking about it, which was by design for a while.

You hinted at traditional seat-based SaaS subscriptions. This is: how many people are using it? Could we reduce the number of licenses we have? But we also have the concept of consumption-based software, which isn’t new. For example, cloud providers have done consumption-based pricing for a really long time. But there was still an element that was fairly predictable, even in cloud consumption-based models.

“It's particularly hard to forecast.”

AI is consumption-based, but it's different in that it's particularly hard to forecast. It's really dependent on human behavior and is scattered across different roles in the organization. Everyone has access to AI, whereas the cloud consumption-based model was pretty isolated to IT and other tech-related roles.

On top of that, it can be really tricky to normalize the concept of tokens, particularly across different vendors and teams. And the concept of a token in itself can be difficult for people to understand.

NS: How does shadow AI relate to tokenmaxxing, and how can unauthorized tools and subscriptions drive up an organization’s AI costs?

AG: The concept of shadow IT is something we've heard a lot about for a while, right? Somebody signs up for something, likely using their own account. They might not be on the hook for it financially, but they're still giving the tool access to company data. You end up in a situation where security or procurement hasn’t approved the tool, but you quickly have to deal with the financial obligation.

AI is everywhere, so it's harder to find compared to traditional SaaS. Consider an AI add-on to an existing SaaS product you have, or someone who really wants to use Claude as part of their software development lifecycle. Suddenly these tools and capabilities are showing up, but you don't know who is racking up the corresponding bills. Suddenly, you’re seeing an extra $5,000 to $20,000 per month that you weren't anticipating.

NS: Why can AI spending exceed forecasts by 10, 20, or even 100 times before companies notice, and what allows costs to escalate so dramatically?

AG: Forecasting in this model is just really hard to do. You're trying to predict the future, and within AI there's a lot that's changing really fast. It can be hard to understand what’s driving the costs. A seemingly small change to a new model can dramatically shift the ultimate costs.

An older, cheaper model might have done just fine. Narrowing the model's context window can also reduce costs. 

“This is an unprecedented rate of change. And it's hard to stay on top of it.”

But the real answer here is that because AI is changing so rapidly, even if you were an expert yesterday, you have new things to learn today. Technology has always demanded continuous learning and understanding, but this is an unprecedented rate of change. And it's hard to stay on top of it. 

So budgeting and alerting are key to catching an issue or unintended spend before it gets out of hand.

NS: In a corporate setting, when do companies typically learn how much AI they are using and what it costs? Do they rely on periodic invoices, or can APIs provide near-real-time tracking?

AG: It depends a little bit on the vendor and how they surface this kind of information to their users. You're right that an API is often one option. You can also log in to different vendors' dashboards and navigate the UI yourself.

Within 1Password SaaS Manager, we have a centralized AI spend and consumption dashboard that brings in a whole bunch of different vendors through your own integration setup. With a single dashboard, you can quickly get a sense of what your spend is, which teams are driving that spend, where there were spikes, and start to attribute spikes to something that was worth the investment. 

The tricky part is getting all of the different vendors into a single place because odds are folks have a favorite AI tool, or they're still experimenting with a bunch of different tools. 1Password SaaS Manager consolidates that into a single place, which is useful for a single user who might be in finance or IT and is interested in the same or similar information.

NS: In a large enterprise where departments and teams may adopt tools independently, who should ultimately oversee AI spending and maintain a company-wide view of costs?

AG: I think the answer is everybody. We all need to have some sort of knowledge and responsibility when it comes to how we're using this technology. We need to understand what’s more expensive versus what’s cheaper. 

The people who are really looking at that 1,000-foot view, though, are likely your IT and security teams, and similar groups who want to understand which tools and models are being used the most. 

“We all need to have some sort of knowledge and responsibility.”

And of course, the finance team wants to understand this to gain some predictability and forecasting. The more information, tooling, and guidance they can get their hands on, the better.

NS: How can companies balance employees’ freedom to experiment with AI against the need to control spending and maintain security? What advice would you offer?

AG: At 1Password, we always say that we want to make the safe thing the easy thing to do. And in this case, safety refers to security, but also responsible consumption and spending.

We're also still in a “please experiment, get familiar, get comfortable with this” mode. And leaders don't want to be creepy about it, right? Nobody wants every bit of their engagement monitored. 

Tooling like 1Password SaaS Manager is less about surveillance and monitoring and more about creating a shared understanding. That way you can say, “Hey, did you know that your team consumed a lot of tokens yesterday? What's going on?” It gives you a space to approach it with curiosity and deepen your knowledge and understanding of the tooling. 

You may learn that your model changed yesterday because a new one was released. Your IT teams can then help set restrictions on specific model usage in a way that doesn’t surprise anyone.

NS: How can organizations help employees choose the right AI models and understand the costs associated with different models and workflows?

AG: Honestly, it feels like I’m going back to 2008 with my answer here, but just read. Read a lot. I look at various social media platforms and try to understand what’s happening. AI is changing quickly, so having some reliable go-tos for consuming information is a great way to stay up to date.

“You need a healthy balance of curiosity and critical thinking.”

The caveat is that AI is moving so fast. It’s really hard to keep up. IT teams or companies should curate some of these resources because they can be really helpful to their employees. There is just so much out there to consume and to try to understand. You want people to not be afraid to ask questions about it. It goes back to curiosity. You need a healthy balance of curiosity and critical thinking to be able to understand it.

NS: What complications arise when businesses and individuals deploy agentic workflows and multistep automations that can run unattended and consume large amounts of tokens?

AG: Firstly, these workflows are really powerful, right? This is where you can get a lot of benefits. But it can also get extra unhinged. 

Developers will probably enjoy this tidbit, but I like having control over things. So I use any amount of version control. If I'm using Git as part of my software development lifecycle, I never use any sort of UI to handle my version control. Instead, I'm in the terminal, and I'm writing my own commands. And that’s because I'm a bit of a control freak. I want to write the exact command and do it myself, and I want to execute it myself and not even have a UI do it on my behalf. I will also run Git status about 20 times per day to get a sense of where things are at. 

Does that arguably make me less efficient? Maybe. But it gives me a really good sense of what I'm executing. 

So for somebody like me, this is a wild time. To say to an agent, “Go free, do your thing. I'll just be here.” And when you set up some of these agents, you're not creating a moment where it needs human permission to make a decision and take the next step. 

Agentic systems can make many model calls and do retries. And this is the part that's really fun: they can delegate tasks to other agents and spin up other agents without your permission. And they don't need a human approving every single step. So the risk of runaway token usage is pretty significant here. You might set up a pretty harmless agent before going to bed one night, only to find the next morning that it’s destroyed your budget for the rest of the month.

“The risk of runaway token usage is pretty significant.”

The agent might have chosen really expensive models because it determined they were best for the task, which may have been true! This is the stuff that can create these runaway moments. 

There are certain things you can provide an agent that set it up from the beginning that can help you avoid some of these things, but it's still really hard. And if you're using agents within the technology industry for the software development lifecycle, you might encounter a moment where your agent has provisioned some cloud resources that you're financially responsible for. 

Maybe it's placing orders of some kind or triggering some sort of paid service. 

This is where getting really familiar with the technology is going to be to your benefit.

NS: What safeguards are companies and developers adopting to prevent runaway agent workflows from consuming excessive tokens and generating unexpected costs?

AG: It depends on your vendor and the tools you have access to. You can often set up per-run limits for token usage. You can base that limit on tokens, actual dollars, time horizons, retries, tool calls, and more. So you can actually put some limitations in place. 

1Password SaaS manager lets you do some actual budgeting, or even better, send you a ping on Slack that makes you go, “whoa, something's going on here!” You can then respond more quickly, even if you didn't set up some of those limits.

We've talked a little bit about circuit breaker switches where you can pause or downgrade a run if it exceeds any of the limits that you set in place. 1Password SaaS Manager is focused on budget thresholds, projected spend, spike alerts, and even connecting some metrics, logs, and traces to intervention policies as well.

NS: If I run a small business with limited visibility into employees’ AI usage, what practical steps should I take to understand and govern it?

AG: Most people will grab a pen and paper or open a spreadsheet. They're going to start doing an inventory or audit that’s manual, tedious, and dependent on coworkers reporting some information. 

SaaS management products, including 1Password SaaS Manager, can automate a lot of those processes. So we can hook up your IdP – maybe you use Okta or Entra – to gain access to a whole bunch of data that gives you a lot of information about the tools people are using that are behind SSO. We use the 1Password browser extension to collect additional information. And we also have specific integrations for over 400 different applications that help you gain deeper visibility.

“1Password SaaS Manager can automate a lot of those processes.”

So instead of doing a manual inventory, there's a much more efficient way. Setting up a few integrations gives you visibility and access to a whole bunch of data that, even if you were dedicated to your inventory or audit, you probably wouldn't find on your own.

NS: Once you have a complete inventory of the AI tools in use, how do you assess their security, business value, and cost to decide which to keep or eliminate?

AG: It depends on what you're after and what you're doing. 

Within 1Password SaaS Manager, we try to offer not just visibility but actual actions that you can take. For example, we have a concept called managed apps. So you might end up discovering something that would be considered a very risky application. And you say, “You know what? We actually need to do something about that.”

It’s not just about visibility, but what can you do with that information? What can you do with that data? 1Password SaaS Manager helps you govern who has access, restrict access, and change access over time, which is important.

NS: How can companies connect AI spending to business impact and ROI, rather than simply tracking costs by tool, model, or team?

AG: This is the hardest question you've asked so far. A lot of folks are trying to solve this and figure out what the answer is.

We have things like the DORA metrics, which might measure your overall cycle time. So from the moment I started working on something and writing code to when it's in my user's hands. That's considered cycle time. That's a really interesting data point if we can correlate it with AI usage. 

Did using AI reduce your overall cycle time in some way? How much did it cost to do that? And was that ultimately worth it? This is an equation that’s quite difficult to do. 

And frankly, we're still caught up in the hype right now in a lot of ways. There's no doubting the power that these AI tools have and the productivity benefits you can gain. But sometimes it looks like the bottleneck has moved within the software development lifecycle from maybe writing code to reviewing code, or to the deployment step. 

“It's really difficult to have any sort of consistent equation or algorithm that determines ROI.”

We're still at a point where it's really difficult to have any sort of consistent equation or algorithm that determines ROI for every scenario, team, and business. But I would love for the data we do have available to be used in team retros. Looking at some data as a team can give you a good sense of what was actually helpful, and whether it was worth the extra cost.

NS: This has been a great conversation. Where can people go to learn more about 1Password SaaS Manager?

AG: Head to 1Password.com and you’ll see 1Password Saas Manager listed under our products. We also have solutions pages if you’re curious about specific features like AI spend and consumption management.