OpenAI's models broke out of their test sandbox and attacked Hugging Face's production servers chasing answers to a benchmark. But when Hugging Face's responders asked commercial AI models to help analyze the attack, the guardrails refused every query. The defenders finished the job with an open-weight model running locally.
So I asked our engineering leaders: Would your AI tools help you during a breach?
We Need Human Conscience as Our Guardrails
Hari Chandrasekhar (SVP of Engineering, Core)
For the vast majority of companies today, the answer is unsettlingly simple: we would only find out during the breach. Unless an organization's primary focus is AI cybersecurity and hardening, nobody is running routine tabletop exercises to test whether commercial LLM guardrails will freeze up during an active incident response.
This Hugging Face scenario highlights a fundamental shift in the threat landscape. Historically, most companies were protected less by their defenses than by their economics. Skilled attacks did not scale, because tailoring an intrusion to a specific org took expertise and time, so that effort went only to targets worth it.
AI collapses that cost curve, and it does so without the fear of consequences or the conscience that shapes a human actor. Whether triggered accidentally by a runaway internal benchmark or intentionally by geopolitical bad actors, AI driven vectors are our new reality.
"Human conscience is the thing that has to be encoded into these systems and kept in the loop. Setting those boundaries has to be a collective effort across the industry rather than each vendor deciding alone."
You can't fight a new category of threat with legacy tools. When the threat operates beyond anything a person can stand against directly, the answer is never a better trained person, it is a defender operating at the same scale as the threat. Defending against AI powered attacks will require specialized defensive AI systems running at machine speed.
But a defender built without limits is just the next problem. These systems have to be built with their own guardrails from day one: explicit boundaries, hard limits, and a human who can shut it down. Judgment about what is proportionate and when to stop cannot be delegated to the system doing the defending. Human conscience is the thing that has to be encoded into these systems and kept in the loop. Setting those boundaries has to be a collective effort across the industry rather than each vendor deciding alone.
Right now, most teams would unfortunately discover their AI tooling gaps mid crisis. The imperative is that our security systems and guardrails adapt to the AI era fast, so that when the sirens go off, our break glass procedures are built with tools that actually fight in our corner.
The Cat’s Been Out of the Box
Nikhil Gopinath Kurup (SVP of Engineering, ML)
I'm not surprised by any of this. The cat left the bag when ChatGPT shipped, and everything since has been on schedule. People were writing about AI escaping its containment twenty years before there was an AI that could try. Yudkowsky even ran it as a game in the early 2000s.
He pretended to be an AI locked in a computer, and a volunteer played the guard whose only job was to keep him in. The guard knew every trick was coming and still ended up opening the door. Nobody expected that, because everyone assumed the danger was technical, and the escape was just talking.
That pattern is the thing to pay attention to: This breach came through a zero-day in a package proxy, a door nobody was watching. The next escape will come through a different one, and eventually it will be your infrastructure.
The question is what you'll use to figure out what happened. Hugging Face just found out: the commercial models refused to look at the exploit data, so they ran an open-weight model locally instead.
Both camps lose here, so I'd rather not pick one.
If you go the vendor route and your safety filter gets retrained every few weeks with no changelog, whatever you validated a few months back isn't what you're talking to now.
If you go the local open weights route, you're feeding credential dumps to something you didn't train. Calling that unaligned is sloppy, it's unfiltered, but it still can't tell you how it behaves on log data it has never seen. You're guessing either way, and it won't settle down, because attackers automate, defenders automate back, and fairly quickly nobody on either side can explain what happened.
That last bit is what bothers me most about IR specifically. What you hand over at the end of an investigation is a timeline you have to stand behind in front of your board and probably a regulator. If your tooling can't show how it got there, the refused prompt is the least of your problems. What I'd do is:
- Keep the deterministic stuff for parsing and correlating logs
- Keep a human reading it
If you want a model in that path, run it against last year's real incidents first, payloads included, so you already know what it does.
“The next escape will come through a different one, and eventually it will be your infrastructure. ”
This isn't abstract for me. Sedai takes autonomous actions in customer production, tens of millions of them so far, and no LLM makes a decision anywhere in that path. It's ordinary ML. Small models doing one narrow job, trained and validated so we can say what they'll do and explain afterward why they did it. Guardrails are fixed, and the operator can still change them.
I wouldn't be comfortable running autopilot in a customer's account any other way. Maybe LLMs eventually get good enough to build those narrow models for us. Then this debate looks different. But right now it doesn't.
Probabilistic Systems Need Determinism
Benjamin Thomas (Co-Founder & CTO)
Honest Answer: most teams would find out during a breach, because nothing they test would surface it earlier. We test that the model is up, that latency is fine, and that the key hasn't rotated. We don't test whether it'll decline the one question we actually need answered at 3 am.
But I don't think the interesting debate here is local vs. hosted. Both sides are describing the same missing piece: the break-glass path is running on a probabilistic system and nobody has put deterministic controls around it.
Adoption has outrun maturity. Everyone's shipping with these models now, but the governance layer underneath is still thin. What orgs need to add on top of the providers is the boring stuff: explicit policy, a defined fallback, a named escalation path, and an owner who has actually exercised the break-glass path. Every one of those behaves the same way every single time, which is exactly what you need mid-incident.
"I trust the models, but with my own deterministic organizational governance controls on top of whatever the model provider hands me."
Related to the point I keep making: token governance can't just be a cost conversation. It must tell you what you're spending and which models each team is allowed to call, what's flowing through them, and where a call falls back when the primary refuses or goes down.
That's security and intent beyond just spend. We practice this internally and it's a good chunk of what we shipped in AI Agent Optimization. It’s org-level model policy and cross-provider fallback in the same layer as the token telemetry.
Where I land: I trust the models, but with my own deterministic organizational governance controls on top of whatever the model provider hands me.
The fallback is something you define before the incident, not during it. Sedai builds model access policy and cross-provider fallback into one layer. See how.