# Sedai: Blog (part 3 of 27) Full content of 10 pages from https://sedai.io/blog, published 2026-07-08 to 2026-07-31. Only pages published in the last 12 months are included. Each entry starts with the page title and its source URL. Every full-content file is listed in https://sedai.io/llms.txt. --- # 10 Best AI Agent Cost Optimization Tools in 2026 Source: https://sedai.io/blog/top-ai-agent-cost-optimization-tools Compare the 10 best AI agent cost optimization tools for 2026, from gateways to observability to autonomous action. See which fits your stack. AI agent cost optimization tools do one of four things to your spend: show you where spend goes, route and cache requests to reduce spend, trim cloud and compute costs underneath, or act on costs directly. Which you need depends on where your cost lives. ## **Key Takeaways** - One agent task triggers many billed model calls because each step re-sends the growing context, so costs compound. - The FinOps Foundation estimates a 30x to 200x cost difference between an unoptimized setup and one engineered for cost. - Prompt caching and model routing save the most; Anthropic prices cache reads at a 90% discount. - Most tools only report or route cost. Only one tool acts on costs autonomously and safely. ## **How AI Agent Costs Differ From Normal LLM Costs** AI agent costs are harder to control than single-call LLM costs because of what happens to the context on every call. With any LLM, context means everything you send the model in a request. That could be instructions, the conversation so far, or any data/tool results from earlier steps.  You pay for all of it with [input tokens](https://sedai.io/blog/what-is-ai-tokenomics) each time you call the model. A single-turn chatbot sends one prompt and gets one answer, so it pays for that context once.  An agent works differently. To finish a single task, it makes a series of calls, and on each call it re-sends everything that has happened so far so the model can continue where it left off. That re-sent history grows with every step, so the agent pays for the same accumulating context again and again. This is what makes agent bills compound and run higher than predicted. Two teams can run the same agent workload, the same tasks at the same request volume, and still pay very different amounts. The [FinOps Foundation estimates a 30x to 200x difference](https://www.finops.org/wg/how-to-forecast-ai-services-costs-in-cloud/) between an unoptimized setup on a premium vendor with no discounts and one engineered for cost. That range is why this category of tools exists. Three cost drivers sit underneath that delta: - **Input tokens**: The system prompt, tool definitions, and prior conversation get re-sent on every step of the loop, so long-running agents pay for the same context repeatedly. - **Output tokens**: Generated text, tool calls, and reasoning are billed at a higher rate than input on most providers. - **Runaway loops**: An agent that retries, recurses, or fails to terminate keeps spending until something stops it. The tools below handle these drivers in different ways. Some measure them, some cap and reroute them, and some act on them without a human in the loop. --- *If those compounding bills are already outpacing what your team can trace and fix by hand, see how [Sedai cuts AI agent spend autonomously while keeping every change inside your SLOs](https://sedai.io/platform/ai-agent-optimization).* --- ## **How We Evaluated These Tools** We evaluated each tool on what it actually does to cost, not feature-count: - **What it optimizes**: Token and model spend, cloud and Kubernetes infrastructure, or both. - **How it acts**: Observe (show you the cost), route (change how requests are sent), or act (make the change for you). - **Production-safety awareness**: Whether it checks latency, errors, and SLO compliance before it changes anything. - **Integration effort**: Whether it needs a rewrite, an SDK import, a proxy, or read-only access. - **Best-fit team**: Platform engineering, FinOps, or finance. Most lists skip the question that matters most: **what does the tool actually do about the cost**? A dashboard and an [autonomous optimizer](https://sedai.io/blog/what-is-ai-agent-optimization) both say they optimize AI agent costs, but a dashboard only shows you where the money goes, while an autonomous optimizer changes it. We say which one each tool is. ## **Quick Comparison Table** | Tool | What It Optimizes | How It Acts | Production Safety | | --- | --- | --- | --- | | Sedai | Full stack (infra + tokens) | Acts autonomously | SLO and latency validated, auto-rollback | | TrueFoundry | Model and token spend | Route and govern | Budget caps, no SLO gating | | Portkey | Model and token spend | Route and cache | Budget limits, no SLO gating | | LiteLLM | Model and token spend | Route (open-source proxy) | Budgets and rate limits, no SLO gating | | Helicone | Model and token spend | Observe (plus gateway) | Alerts and rate limits, no action | | Langfuse | Model and token spend | Observe (tracing) | None, observability only | | CloudZero | AI and cloud spend | Observe (attribution) | None, reporting only | | Cast AI | Kubernetes infra (incl. GPU) | Acts (infra only) | Spot-interruption aware, limited app enforcement | | PointFive | Cloud and coding-agent tokens | Recommend, human applies | None, cost-only lens | | Datadog CCM | Cloud and Kubernetes spend | Observe and recommend | SLOs observed, not used to gate | --- ## **Autonomous Optimizers** ### Sedai [Sedai acts on cost autonomously](https://sedai.io/platform/ai-agent-optimization) instead of simply providing recommendations or visibility through a dashboard. It installs as an SDK between your production agents and their LLM providers, then optimizes cost, performance, and accuracy across every call.  Where most tools stop at surfacing a finding, Sedai autonomously makes the change, validating each one against latency, error rates, and SLO compliance before it acts. #### Key Features [Smart Routing](https://sedai.io/blog/ai-agent-optimization) sends each agent to the right model through a custom router trained on that agent's own production traffic, not generic benchmarks. Cross-provider fallback and reliability are built in, and org-level governance and credential management sit on top.  It supports OpenAI, Bedrock, Vertex AI, and Azure Foundry, and optimizes what the agent runs on as well as what it calls.  On the infrastructure side, that includes the [GPUs behind AI workloads](https://sedai.io/blog/gpu-optimization), where Sedai reclaims idle GPU capacity and packs more work onto each [GPU with NVIDIA MIG](https://sedai.io/blog/how-we-solved-gpu-optimization-problem-kubernetes) and fractional instances, using the same validation and rollback it applies everywhere else. #### What It Does Well Sedai removes the human from the loop safely. Every action works within SLO guardrails and is reversed by auto-rollback if a change degrades performance, so cost cuts do not become incidents. Sedai estimates that at least 30% of LLM costs can be eliminated once agent observability is turned on. #### Limitations Sedai is built for production agents running at real scale, not for single-app usage. It does not cover Snowflake today. #### Best For Platform and ML-platform teams that want a system to fix cost continuously on its own and have SLOs they cannot afford to breach. --- ## **AI Gateways** ### TrueFoundry [TrueFoundry](http://docs.truefoundry.com) enforces cost at the gateway before spend happens through a single governed control plane. It sits in front of your models, tools, and agents and applies policy uniformly across all of them. Its strength is governance rather than autonomy. #### Key Features TrueFoundry supports:  - Latency-based and intelligent routing to avoid unnecessary calls to expensive models  - Cost-based and token-based quotas set with metadata filters - Rate limits per user, service, or endpoint Team-level isolation runs on role-based access control with per-team API keys and usage tracking. #### Limitations Its controls are policy-driven, not SLO-gated. It enforces budgets and routes requests. However, it does not validate a change against latency or error rates and act on its own the way an autonomous optimizer does. --- ### Portkey  [Portkey](http://portkey.ai/docs) reduces spend at the gateway through caching and routing across a wide model catalog. It gives access to more than 1,600 LLMs behind a unified API and adds cost controls on top. Its most distinctive lever is caching that returns stored responses instead of paying for repeated inference. #### Key Features It offers intelligent caching, routing strategies, and request batching, with real-time cost and performance monitoring and budget limits. Its observability surfaces traces, errors, and cache behavior so you can see which calls are expensive. #### Limitations Like other gateways, it enforces budgets but does not gate actions on production SLOs, and caching only helps to the degree your traffic repeats. --- ### LiteLLM  [LiteLLM](http://docs.litellm.ai) is an open-source proxy that puts routing, budgets, and spend tracking in front of 100+ models. It calls them all through the OpenAI input and output format, which makes it a low-friction way to standardize access and add controls without committing to a vendor. The project is open source, with its code on [GitHub](https://github.com/BerriAI/litellm). #### Key Features The proxy server provides a centralized API gateway with:  - Authentication and authorization - Retry and fallback logic across deployments - Multi-tenant cost tracking per project and user  - Budgets per project - Caching as a per-project option #### Limitations It is infrastructure you run and maintain, and its budgets and rate limits are static controls rather than SLO-aware, autonomous ones. --- ## **Observability and Cost Attribution** ### Helicone  [Helicone](http://docs.helicone.ai) shows you where LLM spend goes at the level of individual requests, users, and sessions. It positions itself as an AI gateway and observability platform that helps teams route, debug, and analyze their applications. It is open source with its code on [GitHub](https://github.com/helicone/helicone).  Its role in a cost stack is to make spend legible before you decide what to change. #### Key Features Helicone's dashboard tracks activity by request, session, and user, so you can see what is driving spend. A built-in query language lets you slice that data, and a playground with prompt and dataset tools help you test changes.  Rate limits and alerts flag runaway usage. #### Limitations Observability shows the problem, it does not fix it. Every saving still depends on an engineer reading the data and making the change, so Helicone reduces spend only indirectly. --- ### Langfuse [Langfuse](http://langfuse.com/docs) traces what your agents actually do and attaches cost and quality to each step. It is an open-source platform for debugging and improving LLM applications. It captures every call an agent makes, including retrieval, embeddings, and other API calls, not just the model calls.  For agents, that matters because much of the cost hides in those non-model calls that most tools never track. #### Key Features Langfuse offers comprehensive tracing with:  - Multi-turn and agent visualization - Cost and usage tracking per user - Evaluation tools including LLM-as-a-judge and human labeling - Prompt management with versioning. It is fully open source with a public API. #### Limitations It is an observability and evaluation platform, so it surfaces cost but does not route around it or act on it. --- ### CloudZero [CloudZero](http://docs.cloudzero.com) connects AI spend to the customers and features that drive it, so you can see what each one actually costs. It pulls cost and usage from your AI providers and cloud into one place, then traces a dollar of spend back to the customer or feature that caused it. Its job is answering what something costs and for whom, not changing the bill directly. #### Key Features CloudZero can break a single dollar of AI cost down by customer, feature, or model. It reports unit economics like cost per customer, per feature, and per transaction, and streams spend in real time. When cost jumps, its anomaly detection points to the team, model, or feature behind the spike. #### Limitations It is an attribution and reporting platform, so it identifies waste and anomalies but leaves the fix to your team. --- ## **Infrastructure and FinOps Tools** ### Cast AI  [Cast AI](http://docs.cast.ai) cuts the compute cost underneath your agents when they run on Kubernetes. It is a Kubernetes cost optimization platform for FinOps and cloud engineering teams that rightsizes pods, automates spot instances, and bin-packs nodes in real time.  #### Key Features  It combines rules with predictive machine learning for real-time rightsizing, autoscaling, and spot management, including spot-interruption protection ahead of reclamation. #### Limitations Cast AI optimizes Kubernetes infrastructure only. It rightsizes and scales the compute your agents run on, including GPU workloads inside Kubernetes, but it does not touch Lambda, ECS, VMs, or the LLM and API token spend that drives most agent cost.  Its optimization stops at the infrastructure layer, not the models and calls your agents make. --- ### PointFive  PointFive finds cloud waste across providers and, through its TokenShift product, compresses tokens for coding agents. It is a cloud cost visibility and FinOps platform that detects waste and recommends savings across AWS, Azure, and GCP across 500+ detections.  Its AI-specific play targets developer tooling rather than production agents. #### Key Features It surfaces waste and savings recommendations across clouds and reports through clean FinOps dashboards. TokenShift compresses context for coding agents, supporting tools like Claude Code, Cursor, Copilot, and Windsurf. #### Limitations PointFive can write a fix, either as a pull request or as infrastructure-as-code. But an engineer still has to review it and apply it. The work of changing production stays with your team, and findings can pile up in a backlog. It also runs on static rules with no SLO awareness. That means it flags what looks wasteful, not what is safe to change.  TokenShift is also a separate, narrower product. It runs on developer machines against coding assistants, and never touches the cost of agents running in production. --- ### Datadog Cloud Cost Management [Datadog Cloud Cost Management](http://docs.datadoghq.com/cloud_cost_management/) surfaces cost data and recommendations from the telemetry you already collect. It is a visibility platform that fires anomaly alerts and generates daily recommendations backed by Datadog observability data.  If you already run Datadog, it puts cost next to the metrics you watch every day. #### Key Features  It combines observability data with cloud billing to flag orphaned, legacy, and over-provisioned resources. It refreshes recommendations daily, and covers EC2, RDS, Lambda, ECS, and more. #### Limitations Datadog Cloud Cost Management centers on visibility and recommendations. It can automate a limited set of actions, such as deleting unused EBS volumes or running scheduled remediations with optional Slack or Teams approval, but most cost decisions are surfaced for an engineer to act on. It does not gate cost changes on latency or error SLOs, so it flags what looks wasteful, not what is safe to change. --- ## **Other AI Agent Cost Tools Worth Knowing** A few adjacent tools cover parts of this space:  - [Amnic](https://amnic.com/) gives you a single view of LLM, cloud, and Kubernetes spend together. - [Mavvrik](https://www.mavvrik.ai/) is a cost management platform focused on GPU hours and model costs across providers. - [nOps](https://www.nops.io/) and [Usage.ai](http://usage.ai) lean toward cloud commitment and infrastructure cost, with AI cost as a growing module. ### FAQs #### How do you reduce AI agent costs in production without breaking things? Measure first, then change one thing at a time behind a safety net. A few specific moves do most of the work: - Instrument spend at the request and agent level - Enable prompt caching where context repeats - Route simple sub-tasks to cheaper models, and validate quality as you go The safest changes stay within latency and error guardrails, and can roll back automatically, so a cost cut never becomes an incident. #### What makes AI agent costs different from regular LLM API costs? An agent makes many model calls per task, and each call re-sends the accumulated context of the ones before it, so cost compounds across the loop instead of staying flat. [The FinOps Foundation](http://finops.org/wg/how-to-forecast-ai-services-costs-in-cloud/) measures a 30x to 200x cost delta between an unoptimized deployment and a fully engineered one. That compounding is why agent bills are so unpredictable. #### Can any tool optimize AI agent costs autonomously? A few act rather than just report, but most of the market stops at observability or gateway routing with human-set policies. Sedai is the only tool that provides true autonomous optimization, where the system decides and makes the change itself, validated against production signals.  It does this by keeping each action within SLO and latency guardrails, rolling back automatically on degradation. #### How much can caching and routing actually save? Caching and routing target the two largest levers: repeated context and over-use of expensive models. [Anthropic prices cache reads](http://platform.claude.com/docs/en/docs/build-with-claude/prompt-caching) at 0.1x the base input rate, a 90% discount on cached input tokens, so caching pays off wherever context repeats. Routing simple tasks to cheaper models compounds the effect, though real savings depend on how repetitive and varied your traffic is. #### Do observability tools like Helicone or Langfuse reduce cost on their own? Not directly. They show you where spend goes at the request and trace level, which is the necessary first step, but every saving still depends on an engineer reading the data and making the change. They work best paired with a gateway or an autonomous layer that acts on what they surface. #### What is the difference between an AI gateway and an autonomous optimizer? A gateway sits in front of your models and enforces policies you define, such as budgets, routing rules, and caching. An autonomous optimizer decides what to change and makes the change itself, validating it against production SLOs and rolling back if needed. The gateway needs you to set the rules; the autonomous layer takes the action. #### Which AI agent cost tools are best for FinOps teams? FinOps teams usually start with cost attribution and visibility tools like CloudZero, PointFive, and Datadog, which tie AI spend to teams, features, and customers.  Gateways like TrueFoundry add budget enforcement on top. Acting on that spend, rather than just reporting it, needs an engineering-owned layer like Sedai that can safely make the change. --- *If your agents' costs are rising faster than your team can safely cut them, Sedai is the act layer that reduces spend autonomously without breaching your SLOs. [See what Sedai finds in your own environment](https://sedai.io/platform/ai-agent-optimization).* --- # Would You Trust Your AI Tools During a Breach? Source: https://sedai.io/blog/would-you-trust-ai-tools-help-during-breach OpenAI's models escaped containment and attacked Hugging Face. Guardrails blocked the defenders, not the attacker. Three Sedai engineering leaders weigh in. OpenAI's models [broke out of their test sandbox and attacked Hugging Face's production servers](https://venturebeat.com/security/openais-models-broke-containment-and-cyberattacked-hugging-face-what-enterprises-need-to-know) chasing answers to a benchmark. But when Hugging Face's responders asked commercial AI models to help analyze the attack, the guardrails refused every query. The defenders finished the job with an open-weight model running locally. So I asked our engineering leaders: Would your AI tools help you during a breach? ## We Need Human Conscience as Our Guardrails [***Hari Chandrasekhar***](https://www.linkedin.com/in/hari-chandrasekhar-8241ba7/) *(SVP of Engineering, Core)* For the vast majority of companies today, the answer is unsettlingly simple: we would only find out during the breach. Unless an organization's primary focus is AI cybersecurity and hardening, nobody is running routine tabletop exercises to test whether commercial LLM guardrails will freeze up during an active incident response. This Hugging Face scenario highlights a fundamental shift in the threat landscape. Historically, most companies were protected less by their defenses than by their economics. Skilled attacks did not scale, because tailoring an intrusion to a specific org took expertise and time, so that effort went only to targets worth it.  AI collapses that cost curve, and it does so without the fear of consequences or the conscience that shapes a human actor. Whether triggered accidentally by a runaway internal benchmark or intentionally by geopolitical bad actors, AI driven vectors are our new reality. > "Human conscience is the thing that has to be encoded into these systems and kept in the loop. Setting those boundaries has to be a collective effort across the industry rather than each vendor deciding alone." You can't fight a new category of threat with legacy tools. When the threat operates beyond anything a person can stand against directly, the answer is never a better trained person, it is a defender operating at the same scale as the threat. Defending against AI powered attacks will require specialized defensive AI systems running at machine speed. But a defender built without limits is just the next problem. These systems have to be built with their own guardrails from day one: explicit boundaries, hard limits, and a human who can shut it down. Judgment about what is proportionate and when to stop cannot be delegated to the system doing the defending. Human conscience is the thing that has to be encoded into these systems and kept in the loop. Setting those boundaries has to be a collective effort across the industry rather than each vendor deciding alone. Right now, most teams would unfortunately discover their AI tooling gaps mid crisis. The imperative is that our security systems and guardrails adapt to the AI era fast, so that when the sirens go off, our break glass procedures are built with tools that actually fight in our corner. ## The Cat’s Been Out of the Box [*Nikhil Gopinath Kurup](https://www.linkedin.com/in/nikhilgk?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAADamPYBaJ89fxN1oO4R-K8uWrHq1L5Brk0) (SVP of Engineering, ML)* I'm not surprised by any of this. The cat left the bag when ChatGPT shipped, and everything since has been on schedule. [People were writing about](https://sedai.io/blog/how-much-isolation-ai-agents-need) AI escaping its containment twenty years before there was an AI that could try. [Yudkowsky even ran it as a game in the early 2000s](https://www.yudkowsky.net/singularity/aibox).  He pretended to be an AI locked in a computer, and a volunteer played the guard whose only job was to keep him in. The guard knew every trick was coming and still ended up opening the door. Nobody expected that, because everyone assumed the danger was technical, and the escape was just talking. That pattern is the thing to pay attention to: This breach came through a zero-day in a package proxy, a door nobody was watching. The next escape will come through a different one, and eventually it will be your infrastructure.  The question is what you'll use to figure out what happened. Hugging Face just found out: the commercial models refused to look at the exploit data, so they ran an open-weight model locally instead. Both camps lose here, so I'd rather not pick one.  If you go the vendor route and your safety filter gets retrained every few weeks with no changelog, whatever you validated a few months back isn't what you're talking to now.  If you go the local open weights route, you're feeding credential dumps to something you didn't train. Calling that unaligned is sloppy, it's unfiltered, but it still can't tell you how it behaves on log data it has never seen. You're guessing either way, and it won't settle down, because attackers automate, defenders automate back, and fairly quickly nobody on either side can explain what happened.  That last bit is what bothers me most about IR specifically. What you hand over at the end of an investigation is a timeline you have to stand behind in front of your board and probably a regulator. If your tooling can't show how it got there, the refused prompt is the least of your problems. What I'd do is:  1. Keep the deterministic stuff for parsing and correlating logs 2. Keep a human reading it If you want a model in that path, run it against last year's real incidents first, payloads included, so you already know what it does. > “The next escape will come through a different one, and eventually it will be your infrastructure. ” This isn't abstract for me. Sedai takes autonomous actions in customer production, tens of millions of them so far, and [*no LLM makes a decision anywhere in that path*](https://sedai.io/blog/case-against-llms-cicd-pipeline). It's ordinary ML. Small models doing one narrow job, trained and validated so we can say what they'll do and explain afterward why they did it. Guardrails are fixed, and the operator can still change them.  I wouldn't be comfortable running autopilot in a customer's account any other way. Maybe LLMs eventually get good enough to build those narrow models for us. Then this debate looks different. But right now it doesn't. ## Probabilistic Systems Need Determinism [***Benjamin Thomas***](https://www.linkedin.com/in/benjamin-thomas-97691a8/) *(Co-Founder & CTO)* Honest Answer: most teams would find out during a breach, because nothing they test would surface it earlier. We test that the model is up, that latency is fine, and that the key hasn't rotated. We don't test whether it'll decline the one question we actually need answered at 3 am. But I don't think the interesting debate here is local vs. hosted. Both sides are describing the same missing piece: the break-glass path is running on a probabilistic system and nobody has put deterministic controls around it. Adoption has outrun maturity. Everyone's shipping with these models now, but the governance layer underneath is still thin. What orgs need to add on top of the providers is the boring stuff: explicit policy, a defined fallback, a named escalation path, and an owner who has actually exercised the break-glass path. Every one of those behaves the same way every single time, which is exactly what you need mid-incident. > "I trust the models, but with my own deterministic organizational governance controls on top of whatever the model provider hands me." Related to the point I keep making: token governance can't just be a cost conversation. It must tell you what you're spending *and* which models each team is allowed to call, what's flowing through them, and where a call falls back when the primary refuses or goes down.  That's security and intent beyond just spend. We practice this internally and it's a good chunk of what we shipped in [AI Agent Optimization](https://sedai.io/blog/ai-agent-optimization). It’s org-level model policy and cross-provider fallback in the same layer as the token telemetry. Where I land: I trust the models, but with my own deterministic organizational governance controls on top of whatever the model provider hands me. *The fallback is something you define before the incident, not during it. Sedai builds model access policy and cross-provider fallback into one layer. [See how.](https://sedai.io/platform/ai-agent-optimization) --- # See the Full Picture: Introducing the Resource Lifecycle Dashboard Source: https://sedai.io/blog/resource-lifecycle-dashboard Sedai already gives you deep visibility into individual resources, opportunities, and savings. Now we're providing a single place to step back and understand your entire environment at a glance. Sedai already gives you deep visibility into individual resources, opportunities, and savings. Now we're providing a single place to step back and understand your entire environment at a glance. The **Resource Lifecycle** dashboard gives you a real-time, 10,000-foot view of every resource managed by Sedai: where it is in the optimization pipeline, what's actively moving forward, and where you can take action to unlock more value. --- ## A Bird's-eye View  Resources are surfaced across all your clusters, cloud providers, and accounts, organized into the stages of Sedai's optimization workflow. At the top level, they're grouped into three high-level states: - **Analysis and Execution** — Resources Sedai is actively working through. This includes resources currently being analyzed, those queued up and ready to execute, and those actively being optimized right now. - **Pending Review** — Resources where Sedai has found an optimization opportunity but needs your input before it can act. This is the human-in-the-loop stage. If you're running in Copilot or Datapilot mode, these resources are waiting on your review and approval. - **Blocked** — Resources where something is preventing Sedai from proceeding. Blockers come in two types: *actionable* (resolvable directly within Sedai by adjusting settings or preferences) and *external* (requiring a change outside of Sedai, like updating permissions or environment configurations). --- ## Understanding the Lifecycle Stages Sedai's optimization process is a pipeline. Resources move through it in a predictable sequence, and Resource Lifecycle makes that sequence visible. When a resource is first discovered, Sedai begins **analyzing** it. Once an opportunity is identified, what happens next depends on the resource's operation mode: - **Autopilot**: Sedai moves the resource into a *Ready* state and picks it up in the next execution batch. - **Copilot**: The resource enters the *Review: Copilot* queue, waiting for you to approve the change. - **Datapilot**: Sedai surfaces its recommendation in the *Review: Datapilot* queue, but operates read-only. No changes are applied until you either make a change manually or enable a more active mode. Once an execution batch is picked up, resources move through **In Queue** and into **Executing**. When complete, the optimization becomes a realized saving. If Sedai can't complete its analysis — for example, because telemetry is missing — the resource is marked **Analysis Blocked**. If Sedai has found an opportunity but determines it's not yet safe to act, the resource is marked **Execution Blocked**. Common execution barriers include things like a recent deployment (Sedai backs off to gather more data after changes in the environment) or insufficient replica counts (Sedai won't act if it can't make a non-disruptive change). --- ## Drill In, Don't Just Observe Resource Lifecycle isn't a static summary. It's a starting point for action. Clicking into any status card opens a detailed view of the resources in that state, with explanations of what the status means and what you can do about it. For blocked resources, Sedai breaks down barriers by type and frequency, so you can immediately see where the biggest bottlenecks are. Where a barrier can be resolved within Sedai, you'll see a direct action button. Where it requires an external fix, Sedai links to documentation explaining exactly what needs to change. You can also filter the entire dashboard by cloud provider, cluster, region, or resource type. If you own Kubernetes workloads and want to see where those specifically stand in the pipeline, without wading through Lambda functions or ECS services, you can do exactly that. --- ## Built for Everyone on the Team Resource Lifecycle isn't just for account administrators. It's designed for any engineer or operator who owns a slice of the environment and wants to understand where their resources stand. If you're running Copilot and want to know what is queued for your review versus still being analyzed, this is your dashboard. If you're trying to unblock execution and want to prioritize which barriers to tackle first, the barrier breakdown tells you exactly where to focus. And if you're responsible for the overall health of Sedai's operation, the optimization summary at the bottom of the page gives you a 30-day view of completed optimizations, cost savings realized, and cost avoidance from reclaimed CPU and memory. --- ## Integration Health, Front and Center One more thing: Sedai can only optimize what it can discover. If there's a problem with an integration — an AWS account connection, a Kubernetes cluster, an Azure subscription — Resource Lifecycle surfaces those issues at the top of the page, with a direct link to diagnose and resolve them. Fixing discovery issues upstream means more resources flowing into the optimization lifecycle downstream. --- ### Available Now Resource Lifecycle is available today — you can find it in the left navigation. Check it out and let us know what you think. And if you’re not already a Sedai customer, book a demo to learn more. --- # 12 Best LLM Observability Tools for 2026 Source: https://sedai.io/blog/best-llm-observability-tools Compare the 12 best LLM observability tools for 2026 across tracing, evals, gateways, & autonomous action. Honest strengths, limits, & who each fits. LLM observability tools capture what happens inside your LLM and agent applications: the traces, token costs, latency, errors, and output quality of every model and agent call. Teams use them to both debug failures and control spend. ## Key Takeaways - Most of these tools observe, evaluate, or execute predefined controls; only one acts autonomously on cost, latency, and reliability in production - Open source is a real option: Langfuse, MLflow, Helicone, Portkey, and DeepEval are openly licensed; Arize Phoenix is source-available - The tools are categorized into five jobs: Tracing platforms, evaluation-first tools, gateways, enterprise monitoring, and full-stack with autonomous action - Most teams run more than one tool, so they should evaluate which tool is right for their specific job ## Why LLM Observability Matters Now LLM observability has shifted from optional to expected because AI systems are now in production, spending real money, and failing in ways traditional monitoring was never built to catch. [Gartner expects 60%](https://www.gartner.com/en/documents/7387730) of software engineering teams to adopt AI evaluation and observability platforms by 2028, up from 18% in 2025, a jump that tracks how quickly agents and LLM features are reaching customers. Debugging is the main pressure point because LLM and agent behavior is non-deterministic. In a [2025 study of 38 practitioners](https://arxiv.org/html/2503.06745v1), 79% named non-deterministic execution as a major problem. And because ordinary logs do not capture the complete agent flow, 77% of practitioners in the study reported difficulty diagnosing root causes.  Cost adds a second problem, since token spend accumulates across providers, models, and teams. Without per-call attribution, a spike is hard to trace back to its source. What makes this practical to solve now is standardization. Many of the tools below build on [OpenTelemetry's GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/), which cover OpenAI, Anthropic, AWS Bedrock, and Azure AI Inference, so instrumentation is becoming portable rather than locked to one vendor. And the category has matured to match: Gartner formally named it in their February 2026 [Market Guide for AI Evaluation and Observability Platforms](https://www.gartner.com/en/documents/7387730). ## How We Evaluated These Tools Every tool on this list had to be production-ready and under active development as of July 2026. Beyond that baseline, we scored each tool on six criteria: - **Scope**. Whether the tool covers tracing, evaluation, gateway routing, or the full stack - **Integration effort**. Whether adoption requires code changes, an SDK, an OpenTelemetry exporter, or a base-URL swap - **Self-serve or sales-gated**. Whether you can start for free or have to talk to sales - **Multi-provider support**. Whether it works across OpenAI, Anthropic, Bedrock, and others - **Acts or displays**. Whether the tool acts on its findings or surfaces them for a human to act on - **Enterprise governance**. Whether it offers access control, PII handling, and deployment options regulated teams need We also organized the tools into five distinct categories: - **Full-stack observability with autonomous action**. Tools that act on what they observe, not just display it - **Open-source observability platforms.** Self-hostable tracing, evaluation, and prompt management you run yourself - **Evaluation-first platforms**. Tools that measure output quality and gate releases on it - **Gateways with observability.** Tools that sit in the request path and add cost tracking, caching, and routing - **Ecosystem and enterprise**. Deep ties to a framework like LangChain, or enterprise-grade monitoring and governance We describe every tool using its own documentation and list real limitations for each one. And because most teams run more than one of these tools together, we note where a tool complements another rather than replacing it. --- ## LLM Observability Tools Compared | Tool | Open Source? | Action Taken | | --- | --- | --- | | Sedai | No | Acts autonomously | | Langfuse | Yes (MIT) | Displays and evaluates | | Arize Phoenix | Source-available (ELv2) | Displays and evaluates | | MLflow | Yes (Apache 2.0) | Displays and evaluates | | Braintrust | No | Proposes changes | | Confident AI | Yes (DeepEval, Apache 2.0) | Flags regressions | | Galileo | No | Detects and enforces | | Helicone | Yes (Apache 2.0) | Gateway controls | | Portkey | Yes (MIT gateway) | Gateway controls | | LangSmith | No | Displays and alerts | | Datadog | No | Alerts and redacts | | Fiddler | No | Evaluates and enforces runtime guardrails | --- ## Full-Stack Observability With Autonomous Action ### Sedai Sedai is an optimization platform that brings governance, observability, and intelligent routing to every LLM call. Sedai is unlike other LLM observability tools because it autonomously takes action to reduce the cost and improve performance of AI agents, going beyond simple metric reporting. #### How It Works When a model call turns expensive, slows down, or fails, Sedai reroutes, retries, or fails over to another model autonomously. It is designed to [optimize the agents you've already built](https://sedai.io/blog/what-is-ai-agent-optimization), so it sits alongside your existing stack rather than replacing your tracing or eval tools. Sedai runs as an intelligent layer on every call, capturing cost, token usage, and latency directly from its SDK so teams get a single source of truth across every provider instead of reconciling separate vendor bills.  Its [Smart Routing](https://sedai.io/blog/ai-agent-optimization) then acts on that data. Rather than static, rule-based routing, it:  - Builds a dedicated router for each agent from that agent's real production traffic - Identifies a Pareto set of candidate models - Sends each prompt to the best fit for your cost, latency, or accuracy goals Because the router translates requests and responses between provider formats on the fly, an application written for OpenAI, for example, can be routed to a better-fit model on Anthropic or Google without refactoring code.  Governance policies control which models each team can use, and built-in reliability handles retries and cross-provider fallbacks.  Integration is a one-line import across OpenAI, AWS Bedrock, Vertex AI, and Azure Foundry, with more to release in the future.  #### Key Features - **Acts autonomously in production**: Reroutes, retries, and fails over instead of alerting a human - **Traffic-trained routing**: Smart Routing builds a per-agent router from real production traffic, not generic benchmarks, with Manual and Auto modes - **Single source of truth**: Consolidated cost, usage, and latency across every provider, project, and model - **Governance**: Org and project-level policies control which models each team can use - **Zero-code adoption**: One import, no rewrites, with cross-provider API translation so you can switch models without touching code. #### Limitations - **Not a tracing or eval workbench**: You cannot author your own test suites or manage annotation queues the way you can in Langfuse or Braintrust - **Provider coverage still expanding**: OpenAI, AWS Bedrock, Vertex AI, and Azure Foundry today, with more to come. - **Analysis depth**: Built for governance, routing, and reliability rather than deep trace debugging or hallucination scoring #### Why Choose Sedai - You want observability acted on autonomously and safely in production, not just shown on a dashboard. - You run agents across several providers and want routing, reliability, and governance in one layer - You want per-agent routing tuned to your traffic without manual benchmarking or code changes #### Best For Teams that want observability across providers and want the data acted on autonomously in production. ## Open-source observability platforms ### Langfuse Langfuse is an open-source LLM observability platform (MIT) offering tracing, prompt management, evaluations, a playground, and datasets. It suits teams that want a self-hostable backend they fully control. Currently, its [GitHub repository](https://github.com/langfuse/langfuse) has more than 31,000 stars. #### How It Works The core does not impose product-level usage caps, although teams pay for and manage the underlying infrastructure. Run it self-hosted on Docker or Kubernetes, or use the managed Langfuse Cloud. Its evaluations cover both LLM-as-a-judge and code-based checks. You can connect Langfuse through:  - Its native Python and JavaScript SDKs - Drop-in integrations for the OpenAI SDK and more than 100 libraries and frameworks - A standard OpenTelemetry exporter  That OpenTelemetry support keeps you flexible, since the same instrumentation can point at another backend later instead of locking you in. #### Key Features - **Open source (MIT)**: Some enterprise modules require a license key. - **Broad framework support**: OpenAI SDK drop-in; LangChain, LlamaIndex, and OpenTelemetry-based instrumentation - **Complete platform**: Tracing, prompt management, evaluations, playground, and datasets in one place - **Deploy anywhere**: Self-hostable via Docker or Kubernetes, or use Langfuse Cloud #### Limitations - **Open-core licensing**: Some enterprise modules require a license key - **No zero-code path**: Adoption means SDK or OpenTelemetry instrumentation - **Displays and evaluates**: Automates tracing and evaluation; application remediation remains external ### Arize Phoenix [Arize Phoenix](http://arize.com/docs/phoenix) is an OpenTelemetry-native observability and evaluation tool that runs from a single pip install or Docker container. It fits engineers who want a lightweight tracing and eval workbench during development. #### How It Works Phoenix is built on OpenTelemetry and OpenInference instrumentation, which makes its traces portable.  It documents four features:  - Tracing that captures model calls, retrieval, and tool use step by step - Evaluation through LLM-based evaluators, code checks, or human labels - Prompt engineering with versioning and replay - Datasets and experiments for comparing versions systematically  #### Key Features - **Lightweight start**: Pip install or a single Docker container - **Portable traces**: OpenTelemetry- and OpenInference-native - **Full dev workbench**: Tracing, evaluation, prompt engineering, and dataset experiments - **Free to self-host** #### Limitations - **Production limitations**:Managed alerting and online evals live in commercial Arize AX - **Source-available, not open source**: Elastic License 2.0 allows self-hosting but ELv2 is not an OSI-approved open-source license ### MLflow MLflow is the open-source ML platform whose [tracing feature is a fully OpenTelemetry-compatible](http://mlflow.org/docs/latest/genai/tracing) way to observe LLM and agent applications. It fits teams already running MLflow who want agent traces in the same place as their experiments. #### How It Works MLflow Tracing captures the inputs, outputs, and metadata of each intermediate step in a request. Instrumentation can be a single line, for example mlflow.openai.autolog(), or manual decorators for custom spans. It integrates with OpenAI, LangChain, LlamaIndex, and DSPy.  MLflow also supports human feedback on traces and systematic evaluation, and it is licensed under Apache 2.0 with no feature gating. Tracing was introduced in [MLflow 2.14.0](https://mlflow.org/releases/2.14.0/) and expanded for generative AI in MLflow 3.0. #### Key Features - **Apache 2.0, fully free**: No feature gating - **One-line auto-tracing**: For example, mlflow.openai.autolog(), plus manual decorators - **OpenTelemetry-compatible**: Integrates with OpenAI, LangChain, LlamaIndex, and DSPy - **Mature ecosystem**: Reuses MLflow's experiment-tracking foundation #### Limitations - **You run the server**: Self-hosted tracking-server model unless you use Databricks - **Newer at LLM observability**: Tracing is recent next to its classic ML heritage - **Focused on trace capture and evaluation**: Alerting/anomaly detection come from separate tooling or managed Databricks ## Evaluation-First Platforms ### Braintrust Braintrust is a proprietary platform built around evaluations, combining an eval framework, logging, a prompt playground, and an assistant called Loop. It fits teams that want quality evaluations wired into their CI/CD pipeline. #### How It Works Braintrust’s core primitive combines a dataset, a task, and scorers into a repeatable eval. It automatically wraps OpenAI and Anthropic clients without requiring manual instrumentation around each call. As for the assistant, Loop, it:  - Analyzes logs and traces - Generates SQL filters from natural language - Creates datasets from log patterns - Generates scorers - Proposes prompt changes.  [Per Braintrust’s Loop documentation](https://www.braintrust.dev/docs/observe/loop), Loop asks for confirmation before executing certain actions by default, and auto-accept is an option a user can turn on. So Braintrust automates the analysis and proposes changes, but a human must still approve them. #### Key Features - **Evaluation-first workflow**: Dataset, task, and scorers, built to fit CI/CD - **No-code trace capture**: Auto-wraps OpenAI and Anthropic clients - **Loop assistant**: Analyzes logs and proposes prompts, scorers, and datasets - **Developer tooling**: Prompt playground and multi-language SDKs #### Limitations - **Closed-source**: Self-hosting only on the Enterprise tier - **Free tier caps**: Starter limits processed data and score volume - **Proposes, doesn't act**: A human confirms Loop's changes by default ### Confident AI Confident AI is the hosted platform built on top of DeepEval, the open-source LLM evaluation framework. It fits teams that want a rigorous, code-first evaluation suite with score-regression gates. #### How It Works [DeepEval is open source under Apache 2.0](https://github.com/confident-ai/deepeval). It ships more than 50 research-backed metrics from general scorers like G-Eval and DAG to agent-specific ones like task completion and tool correctness.  Evaluations can be written as standard pytest-style tests, so teams can run them automatically on every code change and fail the build when output quality drops below a configured threshold. Everything above is the open-source framework you run yourself. Confident AI is the hosted version on top, adding the pieces teams need to run evals in production, like shared datasets, release-blocking quality gates, and red-teaming. #### Key Features - **Built on DeepEval (Apache 2.0)**: More than 50 research-backed metrics - **Code-first workflow**: Pytest-style tests with score-regression gates - **Agent-ready metrics**: Task completion, tool correctness, and red-teaming - **Local-first evaluation** #### Limitations - **Evaluation-first, not full APM:** Built for LLM tracing and evals, not infra monitoring - **Judge-model overhead**: LLM-as-a-judge metrics carry cost and variance - **Sales-gated extras**: Advanced and on-premises features require a sales conversation ### Galileo Galileo is a proprietary reliability platform that observes, evaluates, and guardrails GenAI and agent applications using its own small evaluation models. It fits teams that want to evaluate all of their production traffic cheaply. #### How It Works Galileo's [Luna-2 evaluation models](https://galileo.ai/luna-2) stay under 200ms even when running 10 to 20 checks at once on L4 GPUs. And at $0.12 per million tokens versus $5.00 for a frontier model like GPT-5.4, they cost roughly 98% less than using that model as the judge. Its Insights engine automatically clusters similar failures, surfaces root-cause patterns, and prescribes fixes such as adding few-shot examples; a single click can generate an evaluator from an identified issue. Guardrails can also be applied. #### Key Features - **Low-cost, low-latency evaluation**: Luna-2 stays under 200ms even at 10 to 20 checks at once, at about 98% less than a frontier-model judge ($0.12 vs. $5.00 per million tokens) - **Automated root cause**: Insights engine auto-clusters failures and surfaces patterns - **No-code guardrails**: Applied without changes to agent code - **Free tier available** #### Limitations - **Closed-source**: Trace-based pricing scales with volume - **Recommends, doesn't implement**: Insights prescribes fixes, engineers apply them ## Gateways with observability ### Helicone [Helicone](http://docs.helicone.ai) is an open-source observability platform and AI gateway that instruments LLM apps through a simple base-URL change. It fits teams that want cost tracking and caching live in minutes. #### How It Works Helicone offers two modes: gateway and async logger. As a gateway, it sits in front of your provider and adds edge caching with savings shown in the dashboard, plus automatic fallbacks and rate limiting. As an async logger, its SDK records requests after they reach the provider, which keeps Helicone off the critical path but gives up the gateway features. Both modes include cost and latency tracking with prompt management. The platform is Apache 2.0 licensed and self-hostable. #### Key Features - **One-line integration**: A base-URL swap - **Open source (Apache 2.0) and self-hostable** - **Gateway feature set**: Cost tracking, edge caching, rate limiting, and fallbacks across 100+ models #### Limitations - **Inline dependency**: Gateway mode sits in the critical request path, adding potential latency and another availability dependency - **Observes and controls requests**: Does not act on your application logic ### Portkey Portkey is an AI gateway with an open-source core that sits between your application and [more than 1,600 models](http://portkey.ai/docs). It fits teams that want resilient multi-provider routing with built-in controls. #### How It Works Portkey handles:  - Automatic routing and fallbacks across providers - Load balancing across API keys - [20+ deterministic guardrails](https://portkey.ai/docs/product/guardrails) plus LLM-based guardrails, including PII redaction and prompt injection - Observability and cost controls  Integration is a base-URL swap against an OpenAI-compatible endpoint or its SDK. [The gateway is MIT-licensed](http://github.com/Portkey-AI/gateway) and self-hostable through a single command, Docker, Kubernetes, or a VPC, and the managed platform adds a hosted layer. #### Key Features - **Open-source gateway (MIT)**: Self-hostable, including in a VPC - **Resilient routing**: Fallbacks and load balancing across 1,600+ models - **Guardrails built in**: 20+ deterministic and LLM-based guardrails, including PII redaction and prompt injection - **Base-URL swap integration** #### Limitations - **Tier-limited retention**: The free tier caps logs at 3-day retention and 10k per month; longer retention and alerts require paid tiers - **Tier-gated budgets**: Budget controls sit in higher tiers - **Inline middleware**: Every request flows through it - **Not autonomous:** Routing is configuration-driven ### Watching LLMs Isn't Optimizing Them Most LLM observability tools tell you what's going wrong. Sedai fixes it for you. ## Ecosystem and Enterprise ### LangSmith [LangSmith](http://docs.langchain.com/langsmith) is LangChain's proprietary observability and evaluation platform, with native integration for LangChain and LangGraph. It fits teams building on that ecosystem. #### How It Works LangSmith provides:  - Trace-level debugging - Offline and online evaluations - Dataset management - Monitoring dashboards - Annotation queues - A prompt hub It works with or without the LangChain framework and supports OpenTelemetry natively, so you can point any OTel exporter at its endpoint. Its strongest use is turning production issues into offline test cases within the LangChain workflow. #### Key Features - **Deepest LangChain and LangGraph integration** - **Full eval loop**: Trace-level debugging plus offline and online evaluations - **Expert review tooling**: Annotation queues, dataset management, and monitoring dashboards - **Native OpenTelemetry support**: Works beyond LangChain #### Limitations - **Self-hosting:** Enterprise-only and requires teams to operate supporting infrastructure - **Ecosystem-optimized**: Built first for LangChain workflows - **Detects and surfaces**: Humans act on what it finds ### Datadog LLM Observability [Datadog LLM Observability](http://docs.datadoghq.com/llm_observability) brings LLM and agent tracing into the broader Datadog platform, correlating it with existing APM and infrastructure monitoring. It fits teams already on Datadog. #### How It Works Datadog captures end-to-end traces of LLM chains and agent workflows. At production scale, it clusters conversations by topic and flags anomalies, so quality problems surface without anyone reading individual traces. It also covers the security side, scanning prompts and responses for sensitive data, redacting it, and detecting prompt injections. Instrumentation is an auto-instrumenting Python SDK, and the product natively supports OpenTelemetry’s GenAI semantic conventions, mapping attributes like token usage and provider name automatically. #### Key Features - **One platform**: Correlates LLM traces with existing APM and infrastructure monitoring - **Production-scale analysis**: Topic clustering and anomaly detection - **Security built in**: PII redaction and prompt-injection detection - **Standards-native**: OpenTelemetry GenAI conventions support with an auto-instrumenting SDK #### Limitations - **Platform lock**: Requires Datadog - **Cost scaling**: Priced per LLM span, so cost tracks LLM-call volume (agent and tool spans aren't billed). - **Surfaces and evaluates**: Remediation is a human or ops task ### Fiddler Fiddler is a proprietary enterprise platform for AI observability and security across traditional ML models, LLM applications, and multi-agent systems. It fits regulated organizations where AI decisions carry legal or safety consequences. #### How It Works [Fiddler's Centor Models](http://fiddler.ai/trust-service) scores outputs for safety, privacy, and quality using purpose-built models. The checks include toxicity, jailbreaks, PII, and hallucination. Its purpose-built Centor evaluation models return results in under 80ms with no external API calls, and its guardrails integrate with NVIDIA NeMo Guardrails to intercept prompts and responses at runtime.  It retains the drift, explainability, and model-monitoring capabilities of its ML heritage. Detection, scoring, and guardrail interception run automatically. #### Key Features - **Enterprise trust and safety**: Governance across ML and LLM systems - **Trust Service scoring**: Built-in safety, privacy, and quality metrics (toxicity, PII, jailbreaks, hallucination). - **Sub-80ms evaluation**: Centor models return safety, quality, and privacy scores in under 80ms with no external API calls - **Runtime guardrails**: Native NVIDIA NeMo Guardrails integration - **Model-monitoring heritage**: Drift and explainability #### Limitations - **Sales-gated**: Closed-source with no public self-serve pricing - **Heavier setup**: Enterprise deployment effort - **Detection, not remediation**: Fixes stay with human engineers ### FAQs #### Which LLM observability tools are open source? Langfuse (MIT), MLflow (Apache 2.0), Helicone (Apache 2.0), Portkey’s gateway (MIT), and DeepEval (Apache 2.0) are openly licensed and self-hostable. Arize Phoenix is source-available under the Elastic License 2.0, which allows self-hosting but is not OSI-approved open source. Opik by Comet and OpenLLMetry are two additional open-source options worth evaluating. #### Can you use more than one of these tools together? Yes, and most teams do. A gateway for routing and cost control, an evaluation platform for quality, and a tracing backend for debugging cover different jobs and stack cleanly, especially since many share OpenTelemetry instrumentation. Sedai, for example, integrates alongside tracing tools like Langfuse rather than replacing them. #### What is the difference between an AI gateway and an observability tool? A gateway such as Helicone or Portkey sits inline in the request path and can act on traffic in real time, routing, caching, rate-limiting, and applying guardrails. A pure observability tool records and analyzes what happened, usually after the fact, and leaves action to a human. Some tools do both, which is why the comparison table above separates category from whether a tool acts on its findings. #### What metrics should LLM observability tools track? The core set is latency, token usage and cost, and error or failure rates for every model and agent call. Beyond operational metrics, teams track output quality signals such as faithfulness, relevance, and hallucination or toxicity scores, plus drift over time. Agent systems add tool-call success and step-level tracing. #### What is the difference between LLM observability and LLM monitoring? Monitoring tracks known metrics against thresholds and alerts when something crosses them, such as latency or error rate. Observability is broader: it captures enough detail, full traces, inputs, outputs, and evaluations, to investigate problems you did not anticipate. In practice most tools on this list do both, but the deeper the tracing and evaluation, the more it leans toward observability. #### Do LLM observability tools replace APM? Not usually. LLM observability adds token cost, prompt and response tracing, and output-quality evaluation that traditional APM was never built to capture. Tools like Datadog fold LLM traces into existing APM, while dedicated platforms run alongside your APM rather than replacing it. #### What is the difference between LLM observability tools and eval tools? Observability tools capture what happened in production, traces, cost, latency, and errors. Evaluation tools like Confident AI and Braintrust measure whether the output was correct, using metrics such as faithfulness, relevance, and hallucination scoring. Most teams need both, and several platforms on this list combine them. #### Can these tools monitor multi-turn agent conversations? Most modern tools on this list capture multi-step agent traces, not just single calls, including tool calls and sub-agent steps. This matters because agent debugging is where teams struggle most: in a 2025 study of 38 practitioners, 77% reported difficulty diagnosing root cause in agent systems. #### Which tools require the least application-code change? Gateways like Helicone and Portkey adopt through a base-URL swap, and Sedai integrates with a single import line and no rewrites. Tracing and evaluation platforms such as Langfuse, MLflow, and Confident AI use SDKs or OpenTelemetry exporters, which require some instrumentation code, though framework auto-instrumentation keeps it minimal. #### Do LLM observability tools help with AI governance and compliance? Yes, they provide the audit trail: a record of every prompt, response, and model version that governance reviews and data-residency requirements depend on. Gartner predicts explainable-AI requirements will push LLM observability into 50% of generative-AI deployments by 2028, up from 15%. Enforcement, such as controlling which teams can use which models, is a separate governance layer built on top of that record. --- *Sedai is the one tool that actually acts on what it observes. See how Sedai optimizes the agents you've already built → [**Book a demo.**](https://sedai.io/demo) --- # Is AI Too Big to Question? Source: https://sedai.io/blog/is-ai-too-big-to-question A viral essay claims a 0% AI project success rate and dissent gets people fired. Three Sedai engineering leaders on how to question AI with numbers. A consultant who turns down all AI implementation work published an [essay engineers have been passing around](https://hermit-tech.com/blog/ai-mania-is-eviscerating-global-decisionmaking): His team has watched AI projects fail at a 0% success rate for a year and a half, yet engineers feel incapable of pushing back. While companies keep buying AI solutions, the only people the consultant has seen fired over any of it are the ones who expressed doubt out loud. So engineering leaders keep quiet and the least sensible ideas go unchallenged. So I asked our engineering leaders: Has AI become too big to question? ## Forcing AI on Engineers Is How You Get a 0% Success Rate [***Aby Jacob***](https://www.linkedin.com/in/aby-jacob/) *(VP of Engineering)* Saying AI is a failure because of the “[zero success rate](https://sedai.io/blog/ai-failed-human-benchmark)” is judging AI too harshly. What's really happening is a bad workman blaming the tool. For us at Sedai, AI helps tremendously. We have an internal tool called Tycho that triages issues within our systems, and helps us figure out issues way ahead of time. But beyond that, Tycho enabled us to start exploring surface areas that we wouldn't have explored so far. That boosted our ability to work and reduced toil from a developer standpoint. Conversely, just last night, [OpenAI reported they had lost control of their most advanced models](https://www.wired.com/story/openai-models-escaped-containment-and-hacked-huggingface/), as they went rogue and hacked Hugging Face, one of the largest hubs for sharing AI models. The “escaped models” were trying to find solutions for the AI cybersecurity benchmark ExploitGym. In doing so, the prompting that makes these experiments possible “egged” the models on, allowing them to breach containment. One consultant said, “This is not an AI problem. It’s negligence on a 40-year-old standard, and it’s basically every sci-fi film ever.” Here you see both sides of the AI coin. On one side, AI has allowed my team to build things we would have never thought to build before. On the other hand, AI without human oversight can cause chaos. This is what AI brings to every scenario: If you use the tool the right way, you can gain from it, If you don't, it's like a nuclear weapon. "Engineers should be given their space and time so AI can be utilized in the right way. Just asking Claude, 'Fix this,' is not going to fix the problem." — Aby Jacob, VP of Engineering I really do feel like this is an inflection point in human civilization where we’re working with a technology that is far more capable, or seemingly capable, than our own intelligence. But while we know it thinks and produces solutions, hallucinations and other issues can veer us away from the solution we expected and create problems. In order to harness or tame this AI, we need to put in extra effort so it doesn't stray from our target and comes back with what we really want as an outcome. That is the struggle developers and engineers are having right now. And from a leadership point of view, throwing AI at everything doesn't solve the problem. Engineers should be given their space and time so this tool can be utilized in the right way. Just asking Claude to fix this is not going to fix the problem. AI *will* create more problems, but we’ll also be able to fix those as well. If something is created by a human and there’s a bug in development, we can easily figure it out and fix it. Now, with [thousands of lines of code written by AI](https://sedai.io/blog/would-you-ship-codebase-rewritten-by-agents), you have to pick the needle from the haystack. It’s not impossible, but engineers should be given the room to develop with AI, understand how it behaves, and slowly adapt to it. Pushing AI down the throats of engineers might overwhelm them, and with the expectation to deliver 10x with AI, people will stumble and fall. Some will read that as a 0% success rate. But the failure isn't the tool's, it's the rollout's, and that is something we need to be aware of. ## The AI FOMO Isn’t New and Neither Are Its Failures [***Nikhil Gopinath Kurup***](https://www.linkedin.com/in/nikhilgk/) *(SVP of Engineering, ML)* The thing driving all of this is FOMO, and we have trained ourselves into it over twenty years. Case in point, a few companies bet early on the internet, more jumped on Web 2.0 and social, almost everyone eventually moved to mobile, and the ones who sat out during each wave got left behind. So the reflex now is simple: Never be the one who missed “the next thing.” And we just ran this exact play with crypto. So when a bet like AI and its adoption becomes a marker of whether you're forward-thinking or not, dissent starts to look like disloyalty, and the author of this piece nails what happens next. The coordination problem is real, and executives *are* afraid to be the first one to say something sane; anyone who has been in those rooms has watched a smart person go quiet because being honest could be unsafe for their career. Where I part ways is the notion that AI has a 0% success rate and “it's all lies” conclusion, because it has the same blind spot it's diagnosing. "The projects failing now are failing for the same reason ML projects failed for a decade: nobody operationalized and nobody measured." — Nikhil Gopinath Kurup, SVP of Engineering, ML The author, as a consultant who has rejected AI implementation work, has his sample pre-filtered to the projects that were always going to fail. And having spent years building ML models, I can tell you the failure mode long predates LLMs. The model was almost never the hard part. Operationalizing it was: the data pipelines, the drift and retraining, wiring a prediction into a real system, and getting anyone to actually trust and act on the output. The model is maybe ten percent of the work. The projects failing now are failing for the same reason ML projects failed for a decade: nobody operationalized and nobody measured. Generative AI did not remove that problem, it just made it trivially easy to fake a demo that skips the expensive 90%, which is exactly why those demos hypnotize people. So has AI become too big to question? In a lot of companies right now, yes, but as long as opinion is your only currency, because you cannot argue a true believer out of faith. But crypto felt too big to question in 2021, too. The difference is that AI is actually here to stay. Web 2.0 and mobile both started as overhyped nonsense and settled into things we now use without a second thought, and AI is on the same path. The hype dies the moment somebody [measures what the technology](https://sedai.io/blog/what-is-ai-tokenomics) actually did: Did cost drop, did the SLO hold, how many actions did the system take on its own, how many incidents did it cause? You cannot argue with a number that cannot be gamed. The way you make AI questionable again is not more courage in meetings, but rather refusing to judge it on anything other than the result. ## Healthy Engineering Culture Can’t Just Be Vibe-Based [***Shankar Jothi](https://www.linkedin.com/in/shankarj/)*** *(VP of Engineering, ML)* Healthy engineering cultures should be able to question any specific use of a tool without that being read as questioning the whole strategy. Skepticism about a particular chatbot, workflow, or rollout is just good engineering. It only becomes a problem when "this specific thing isn't working" gets heard as "you're against the direction tech is moving in," because then people stop being honest about what you actually need. "If nobody's tracking whether a tool actually gets used or produces a real outcome, then the optimists and skeptics are both arguing from vibes." — Shankar Jothi, VP of Engineering, ML The healthiest teams I've seen treat AI like any other significant investment: clear success metrics, honest check-ins, and a willingness to walk away from something that isn't paying off. A lot of what gets labeled "AI failure" is really just how ordinary projects go sideways when that discipline is missing. Leaders should be outlining clear goals with clear owners and clear usage measurements. Measurement is where I think leaders should focus the most. If nobody's tracking whether a tool actually gets used or produces a real outcome, then the optimists and the skeptics are both arguing from vibes, and neither claim is verifiable. "We saw no gains" and "We implemented it badly" can look identical from the outside, which is exactly why evidence matters more than conviction here. So has AI gotten too integrated to push back on? I don't think the goal is more pushback or less. I think it's keeping space open for people to say, "This particular thing isn't working," without it becoming a referendum on anyone's judgment. Get that right and the technology mostly sorts itself out. --- *Questioning AI starts with measuring it. Sedai shows you what your agents actually cost and routes each request to the most efficient model. [**See how](https://sedai.io/platform/ai-agent-optimization).*** --- # Announcing Cloud Savings Reporting You Can Defend in Any Room Source: https://sedai.io/blog/savings-and-impact Sedai's new savings and impact dashboards bring a new level of sophistication to how cloud value is calculated and communicated — with accuracy and transparency that no other optimization platform comes close to matching. Sedai's new savings and impact dashboards bring an elevated level of sophistication to how cloud value is calculated and communicated, with accuracy and transparency that no other optimization platform comes close to matching. Cloud optimization tools all claim to save you money. Most back that up with a single number, calculated against list price, assuming 24/7 runtime, ignoring your negotiated discounts. It looks impressive on a dashboard. It just doesn't hold up in a FinOps review. Sedai has always approached optimization with nuance and precision, and our new savings and impact dashboards extend that same rigor to how we *report* the benefits we deliver, giving FinOps teams numbers they can stand behind with confidence. ## More Accurate Than the Industry Standard, by Design Most optimization tools make three simplifying assumptions when calculating savings: list price, 24/7 runtime, and static scale. These assumptions inflate the numbers and make them impossible to reconcile against an actual cloud bill. Sedai's new calculation engine rejects this oversimplification. Sedai pulls actual discount data directly from your Cost and Usage Report (CUR), precisely measures the operation hours for every resource, and tracks every scaling action. The result is a savings figure that may be more conservative than what other tools show, but which delivers a superior level of credibility and convenience for your finance team when they’re reconciling savings against their source-of-truth FinOps tools. That precision compounds over time. As Sedai continues to optimize, the numbers build — and every dollar shown can be defended, line by line, in front of the most skeptical stakeholder. ## Realized and Projected: Clarity at Every Time Horizon The new dashboards don’t just tell you what Sedai has already accomplished; it also features the savings you’re on track to deliver next. Toggle between Realized and Projected views to flip between historical savings and forward forecasts. ## Powerful Aggregation & Filters Need to compare your optimization progress across regions? Across teams? Across managed services? Our new dashboards make this simple: select the dimension you care about, and then choose a stacked total area chart, or plot them as competing series to see the relative progress. ## Cost Avoidance: The Value Other Tools Don’t Show You Direct cost savings — swapping one resource type for another, seeing the line item drop immediately — is the easy part to communicate. But a significant share of the value Sedai delivers doesn't show up that way. When Sedai right-sizes a workload, it frees up capacity on your nodes, letting you run more on your existing infrastructure without provisioning more. One large financial institution that uses Sedai reported that they were able to triple the number of workloads in their dev environment without increasing their spend, purely due to the cost avoidance actions that Sedai executed. Cost avoidance is the money you didn’t have to spend. The new dashboards give that value its own dedicated view, so the full picture is always visible alongside direct savings. ## Resource Usage: The “How” Behind the Numbers Savings figures are outputs. The resource usage reductions that drive them are the inputs, and the new dashboards make those visible in a dedicated Resource Usage screen. CPU hours, memory, and storage savings are broken down per resource, with the same Observed/Projected toggle, so any stakeholder can trace a dollar amount back to the specific infrastructure changes that generated it. This is the layer that answers the deeper FinOps question: not just how much did we save, but what exactly changed to produce that number. ## Latency Reduction: Performance, Quantified Cost is only part of what Sedai optimizes. The new dashboards add a Latency Reduction screen that puts a concrete number on the performance improvements Sedai delivers alongside cost savings, making visible a dimension of value that no other optimization platform surfaces in its reporting. ## A Reporting Experience Built for Champions The people using these dashboards aren't just curious; they're building a case to drive more adoption of Sedai within their organization to share with their executives and their finance teams. They need a Savings view that makes that job easier, not harder, and Sedai’s new Savings & Impact dashboards were built with this process in mind. ### 🔍 Granular filtering Slice savings by account, cluster, region, resource and more. Surface exactly what’s needed for any audience. ### 📈 Trend visualizations Stacked area and line charts show how savings accumulate over time, not just where they stand today. ### 🔄 Realized vs. Projected toggle Instantly switch between what's been captured and what's expected — clearly labeled, never conflated. #### ### Available now, with more to come The new savings and impact dashboards are generally available today for all Sedai customers. They represent a new foundation for how Sedai communicates value — one we'll continue to build on as the platform evolves. --- # Would You Ship a Codebase Rewritten Entirely by Agents? Source: https://sedai.io/blog/would-you-ship-codebase-rewritten-by-agents Bun ported 535,000 lines of Zig to Rust with 64 Claude agents in 11 days. Four Sedai engineering leaders on when agent-written code is safe to ship. The team behind [Bun ported 535,000 lines of Zig to Rust in 11 days](https://bun.com/blog/bun-in-rust), with one engineer, 64 Claude agents running in parallel, roughly $165,000 in API costs, and compiler errors treated as a distributed work queue.  The port is already running in Claude Code and Prisma's public beta. Zig's creator called it ["unreviewed slop."](http://www.theregister.com/devops/2026/07/14/zig-creator-calls-buns-claude-rust-rewrite-unreviewed-slop/5270743) So I asked our engineering leaders: would you ship a codebase rewritten entirely by agents? ## Shipping Codebases with Agents is the New Turing Test [***Nikhil Gopinath Kurup***](https://www.linkedin.com/in/nikhilgk/) *(SVP of Engineering, ML)* Yes, I'd ship this one. And notice what the Bun team actually said: they gave the credit to the compiler, not the model. "Compiler errors as a work queue" is basically the whole story. They didn't trust the agents at all. They built a setup where every one of the \~1,300 lines a minute got checked mechanically, and the borrow checker doesn't care how good your prose is. The way I think about it, this is really a Turing test. If you have a perfectly specified system, and a compiler is about as perfect as it gets, then the input is spec'd, the output can be verified from every angle, every claim in the spec has a test case, and there's already an implementation to diff against. Once you have all that, it stops mattering what's inside the box writing the code. Could be an army of interns, could be a few 100x engineers, could be 64 Claudes running for 11 days. If it swims like a duck and quacks like a duck, it's a duck. The author basically disappears into the verification. You can see the edges of that in the fine print, though. 19 known regressions still made it into the final compiler and look at what they were: side effects inside a `debug_assert` that disappeared in release builds, bytemuck panicking where Zig used to silently truncate, comptime format-string behavior that got lost in translation. "Ship an agent-written rewrite as far as you can verify it." — Nikhil Gopinath Kurup, SVP of Engineering, ML None of those are memory bugs. The duck test only checks for the stuff that's actually in your spec. The compiler caught its whole class and was blind to the rest. That's why they also re-ran the exact same TypeScript test suite and put two adversarial Claudes on it: one trying to ship and one trying to break. Even with all that, some behavior still slipped through. We did a smaller version of this ourselves recently. We had SDKs for our APIs in a few languages, a customer wanted a new one, and that is an almost perfect agent task. Translate from an existing SDK, re-run the same 100%-coverage test suites the other languages already had. Spec, reference, full coverage, clean duck test. The agent handled it well. But I wouldn't reach for this everywhere. That SDK job worked because the spec, the reference, and the tests all existed first, and engineers built them. Everything before that point, speccing the system, designing it, deciding what's even worth building, reviewing whether it's right, is still on the engineers here, and I'm not looking to change that.  Agents have made those people a lot faster. They're not going to replace them. Ship an agent-written rewrite as far as you can verify it. When you're writing the code and the thing that checks the code at the same time, that's still a person's job. ## Shipping Faster With AI Is in Tension With Guaranteeing Quality [Ethan Andyshak](https://www.linkedin.com/in/ethanandyshak?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAAF9tWQBm-jMFF-edozZggfxe4bu71ieQN0) *(SVP of Product)* “The agents never had to be trusted, only checked.” This is the most important detail for me. Using agents to write code is efficient, but we're not at the point yet where we can safely ship all agent code without reviewing it.  This is a challenge because reviewing code can be time-consuming: you have to read and understand what the agent did in order to figure out if it makes sense. Even when there's no AI involved, this process requires a non-trivial percentage of the time it takes to write the code. “If you try to fix AI problems with more AI, you might just be making a xerox of a xerox” — Ethan Andyshak, SVP of Product If companies try to accelerate by using agents to write large code bases quicker than they can be reviewed by humans, [humans are still a bottleneck in the workflow](https://sedai.io/blog/is-ai-slop-ruining-engineering), and you can't achieve the full acceleration that you might want.  As a result, we see new AI-driven tools coming out to take some of the burden of reviewing the code as well. At the end of the day, if you try to fix AI problems with more AI, you might just be making a xerox of a xerox — you're still losing some quality unavoidably. In the future, we hope that the original agents writing code will have closer competence to senior developers, and that the peer review agents have the same critical eye as those senior developers' colleagues. Until then, while we're in this awkward growing pains phase, the pressure to ship code faster with AI remains in tension with the pressure to use human code review to guarantee quality. ## Quality Still Takes Nuance and Iteration, Even Shipping With AI [***Hari Chandrasekhar***](https://www.linkedin.com/in/hari-chandrasekhar-8241ba7/) *(SVP of Engineering, Core)* The single biggest advantage I take from the Bun story is the validated tests. That is probably the main reason they could ship an agentic migration with confidence. The procedure they outline is a good illustration of the nuance, iteration, and meticulous thought that actually goes into building a product at the quality bar people expect. Expectations of AI systems and agents have skyrocketed, but it is still not magic. These systems think and act based on historical evidence and traces drawn from human experience and ideas. A genuinely new product or feature gets built on top of those ideas through repeated thought and improvement. Most successful products come from that additional process, and specialized products bring in niche concepts that make them stand out. The article shows how agentic systems can expedite the work, but the specialization and meticulous treatment were orchestrated by a human with the agent’s help. “Expectations of AI systems and agents have skyrocketed, but it is still not magic.” — Hari Chandrasekhar, SVP of Engineering, Core Building on Ethan’s “checked, not trusted” point: can we ship code that is entirely agent-built? Yes, if there is enough process and enough guardrails to guarantee the level of validation a product’s quality bar requires. Vibe coding is getting popular, and we have also heard reports of people taking backward steps. In areas with simple, known, already proven steps, vibe coding works, and more people without a technical background will be able to use it.  But products with niche and fresh ideas need newer thinking and rigor. Agents will make that possible far faster, but it involves a dimension of human intelligence, experience, and thought that is not yet encoded or contextualized for current AI. Over time there will be an iterative process where the systems eventually learn it. To summarize: can we ship code with agents? Yes, we are heading there. But there is a part of human intelligence, iteration, and thinking that is still required to orchestrate and set up the systems for high standards. ### Why the Bun Rewrite Actually Worked [***Suresh Mathew***](https://www.linkedin.com/in/sureshmathew/) *(CEO & Founder)* Would I ship a codebase rewritten entirely by agents? Yes, but with a big caveat. It comes down to first principles here: The fundamental issue is you’re relying on a probabilistic engine. An LLM gives you the most likely output, not a correct one. So trust is off the table from the start, meaning verification must become everything. "Take away that harness and I wouldn't ship it." — Suresh Mathew, CEO & Founder And verification is actually harder here than a human port. Humans reviewing code as they translate it is free QA. A million agent-written lines that nobody read? That checking has to be built explicitly.  Bun had years of tests and a compiler enforcing exactly what they cared about. That’s why it worked. Take away that harness and I wouldn’t ship it. *Shipping agents is the first problem. Running them is the next. Sedai optimizes the agents you've already built. [**See how](https://sedai.io/platform/ai-agent-optimization).*** *Want to know how AI agent optimization works? [**Read our guide.**](https://sedai.io/blog/what-is-ai-agent-optimization) --- # Introducing the IaC Mapping Agent Source: https://sedai.io/blog/iac-mapping-agent Sedai has always been able to find the right optimization. Now it can eliminate IaC drift at the same time. Sedai has always been able to find the right optimization. Now it can eliminate IaC drift at the same time. --- ## The drift problem no one talks about If you've worked with an optimization or autoscaling tool alongside a GitOps workflow, you've almost certainly hit this wall. The tool makes a good call — right-sizes a deployment, adjusts a memory limit, tunes replica counts. Your cluster reflects the change. Everyone's happy. Then your CD pipeline runs. Whatever Argo CD, Flux, or your own Terraform automation has on file is what gets applied. The optimization is silently overwritten. You're back where you started — or worse, now you're chasing an incident that looks like a regression but is actually just config drift. The real fix isn't to patch around your IaC. It's to make the optimization *part* of your IaC. That's what the IaC Mapping Agent does. ## What the IaC Mapping Agent does When Sedai identifies an optimization, the Mapping Agent determines the precise file, line, and field in your repository where that value lives, then Sedai raises a PR to change it. Your IaC becomes the source of truth for Sedai's recommendations, not a system it has to work around. The challenge we had to solve first: no two teams represent their infrastructure configs the same way. Some companies keep one repo per cluster; others have a repo per microservice. Helm values, Terraform variable files, Kustomize patches — all valid, all different. You can't hardcode a mapping for this. So instead of prescribing how you should structure your IaC, the agent learns how you already do it. Every optimization Sedai makes reduces to a single atomic operation: find a specific resource, locate its config file, identify the field, update the value. The IaC Mapping Agent makes that operation reliable and repeatable, regardless of how you've organized your infrastructure code. ## How it works: two-step mapping We built the agent around two sequential phases. The key was designing each phase to be reviewable and correctable by a human, because even the best automated extraction benefits from a sanity check. ### Step 1: Conventions Extraction Point the agent at your IaC repository. It scans the file tree and derives your organizational conventions — how you map clusters to folders, namespaces to files, deployments to config blocks. These conventions are surfaced as plain-language statements you can review, edit, and approve before anything else happens. Here’s an example of what that conventions extraction looks like: ### Step 2: Resource Mapping Using your approved conventions as a guide, the agent maps every resource Sedai manages to its exact file and line in the repo. For Kubernetes workloads, this means CPU requests, CPU limits, memory requests, memory limits, and replica counts, all mapped to specific paths. You approve the output; from that point on, every optimization automatically becomes a PR. Here's what an extracted resource mapping looks like in practice.  Once you approve that mapping, it's permanent. Any future optimization targeting that resource goes straight to a PR against those paths. ## Human-in-the-loop, by design We've thought carefully about where automation should stop and human judgment should begin. The answer: at every boundary between "what we inferred" and "what you intended." The extracted conventions are shown in plain language, not YAML or JSON, because they describe organizational decisions a person made, and a person should be the one to confirm them. The resource mappings are similarly surfaced for review before they're locked in. You can edit either at any point. PR auto-merge is deliberately not enabled by default. We want the engine to earn your trust before operating autonomously. Once it does, you can open that gate — but the system is designed so you never have to if you'd rather keep humans in the loop on every change. ## Before and after | Before | After | | --- | --- | | Sedai optimizes the cluster; CD pipeline overwrites it next deploy | Optimizations surface as PRs in your repo — CD pipelines apply them | | SREs maintain manual CSV or tag-based mappings for each resource | Agent infers mappings from your existing repo structure automatically | | Config drift is invisible until something breaks | Drift is impossible: IaC is always the source of truth | | IaC and live state diverge continuously | Live state and code stay in sync continuously | | Every new resource requires manual onboarding effort | New resources are mapped in minutes, not hours | ## What's supported today The initial release covers both Kubernetes and non-Kubernetes resources. We started with Kubernetes because its structure made it possible to validate high-confidence mapping quickly, and that same two-step model — conventions extraction, then resource mapping — now extends to non-Kubernetes resources as well. Supported managed fields in this release: - **Kubernetes Workloads:** CPU request, CPU limit, memory request, memory limit, and replica counts — across Deployments, StatefulSets, and DaemonSets, with full support for multi-container pods. Helm, Terraform, and Kustomize formats are all handled. - **Non-Kubernetes resources:** Support for serverless functions (Lambda), ECS services, and other non-Kubernetes compute is now available. The two-step conventions + mapping model is identical — only the convention taxonomy changes to match the resource type. - **Repository integrations:** GitHub, GitLab, and Bitbucket are all supported. For GitLab, use the project\_id as the repository path. For GitHub and Bitbucket, the standard owner/repo format applies. ## How this relates to Guardrails as Code Recently, we announced [Guardrails as Code](https://sedai.io/blog/guardrails-as-code). If you're using Guardrails as Code today, the Mapping Agent complements it — they're not alternatives. Guardrails encode the constraints on what Sedai is allowed to do (architecture families, min/max ranges, node constraints). The Mapping Agent determines where in your IaC those decisions get written back. You'd use both: guardrails to bound the solution space, the Mapping Agent to ensure every decision within that space lands in the right file. ## What's next The foundation is in place across both Kubernetes and non-Kubernetes resources. Here's where we're taking it. - **Auto-merge:** Today, every PR the agent raises requires a human to approve and merge. That's intentional — we want the engine to prove itself before you hand it the keys. But the capability is built, and for teams that reach that level of trust, auto-merge will be a configuration option. We'll be rolling that out as customers get comfortable with the system in production. - **Streamlined onboarding for large-scale environments:** We'll be honest: if you're running thousands of microservices across many repos, the initial mapping review is a real time investment for your SRE team. That's a known limitation, and we're actively exploring ways to reduce that toil — things like bulk approval flows, convention extrapolation across clusters, and smarter grouping to cut down the number of decisions a person needs to make. The agent already does the authoring; we want to make the reviewing faster too. --- ### Ready to try it in your environment? Book a Sedai demo to speak with a technical expert. --- # What is AI Agent Optimization? Source: https://sedai.io/blog/what-is-ai-agent-optimization Learn about AI agent optimization, how it works, and which teams own it in production. The complete breakdown for platform and FinOps teams. *Last updated: July 28, 2026* AI agent optimization is the practice of improving an AI agent so it completes tasks more reliably, at lower cost, and with less risk. In practice, it means managing the full runtime system: routing decisions, model selection, tool use, context management, guardrails, evaluation, and reliability infrastructure. ## **Key Takeaways** - AI agent optimization improves task success rate, cost per task, and risk controls across the full agent execution loop rather than a single model call. - Observability, governance, reliability, and optimization are the four pillars to AI agent optimization, and each addresses a different failure mode. - "FinOps for AI" is the emerging practice of applying cloud cost attribution models to the model call layer. - AI agent optimization applies to agents already running in production without a required rewrite. --- ## **What Is an AI Agent?** An AI agent is a software system that uses a large language model (LLM) to plan and execute multi-step tasks autonomously. Unlike a single model call that takes an input and returns an output, an agent operates in a loop, calling tools, retrieving information, making decisions, and taking actions across a sequence of steps to complete a goal. ## **What Is AI Agent Optimization?** [AI agent optimization](https://sedai.io/blog/ai-agent-optimization) is the ongoing practice of improving an AI agent's consistency, cost per task, and risk controls under real production conditions. In practice, this means managing routing decisions, model selection, tool use, context management, guardrails, evaluation, and reliability infrastructure across the full agent execution loop. Three metrics define a well-optimized agent:  - **Task success rate**: The rate of correct outcomes, not plausible responses - **Unit economics**: The total cost per completed task; combines model tokens, tool calls, retries, and human review - **Risk**: The rate of policy violations, unsafe actions, or irreversible mistakes per task run All three must be tracked together. For example, optimizing cost without tracking task success rate means an agent will route calls to a cheaper model regardless of whether output quality is degrading or not. What makes this distinct from prompt engineering or model fine-tuning is the unit of work: - **Prompt engineering** improves a single interaction  - **Model fine-tuning** changes the weights of a specific model By contrast, **AI agent optimization addresses the full runtime system**. So, for example, a customer support agent making eight LLM calls per ticket has eight opportunities to overpay, use the wrong model, or retry unnecessarily.  Only some of those calls need a frontier model, and without routing, all eight go to the same expensive endpoint. That inefficiency compounds across thousands of tickets a day. Addressing it at scale requires instrumentation across every agent call and continuous optimization of what that data surfaces.  Engineering teams can build these layers manually per agent, or an infrastructure layer can manage it automatically across every agent already running in production. --- ## **How AI Agent Optimization Works** AI agent optimization works through four pillars: | Pillar | What it does | Outcome | | --- | --- | --- | | Observability | Instruments every agent call, tool invocation, and decision | Surfaces what's failing | | Governance | Sets spend limits, model allowlists, and access controls | Prevents runaway spend | | Reliability | Handles retries, fallbacks, and context window limits | Maintains pipeline continuity | | Optimization | Routes calls, tunes prompts, and caches repeated inputs | Reduces cost and latency per task | ### Observability Agent observability instruments every step an agent executes, including LLM calls, tool invocations, retrieval steps, planning decisions, memory operations, and agent-to-agent handoffs.  By contrast, standard monitoring only tracks HTTP latency, error rates, and CPU utilization, which tells you whether a request completed, but not whether the agent took the right action or why a multi-turn sequence failed.  The standard for agent instrumentation is [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), which define standardized attribute names for LLM spans so traces are queryable the same way, regardless of which framework or provider produced them. A single agent conversation can generate megabytes of deeply nested spans across dozens of runs, which is why agent-specific observability tooling exists as a category separate from APM. Agent observability addresses two distinct operational gaps that standard APM cannot fill: trace level visibility and fleet-level visibility. With **trace-level visibility**, you can see what happened inside a specific agent run, and it requires a dedicated tracing provider such as LangSmith or Langfuse.  With **fleet-level visibility**, you can see what the entire agent fleet is spending and where. This means [token usage](https://sedai.io/blog/what-is-ai-tokenomics), cost, errors, and latency across every provider and model. Standard APM doesn't provide this because it tracks infrastructure metrics, not model call costs across external providers. ### Governance Governance sets the constraints an agentic fleet operates within. It controls which models and providers teams can use, and enforces that control automatically rather than relying on developer self-governance. Specifically, governance prevents model sprawl. This means when teams independently select models, credentials scatter across codebases, and there’s no single view of what the organization is spending or whether they’re using approved models. When every team makes model selections independently, cost spikes arrive without attribution and compliance gaps accumulate without visibility. To manage this, governance must use a two-tier model access control, consisting of organization-level policies and individual agents: - The **organizational policy tier** sets Allow/Deny policies for the entire fleet  - The **agent tier** overrides those policies for teams with different requirements  Every agent in the fleet inherits the org-level policy by default; agent-level configuration narrows or extends it. Alongside access control, governance at the agent layer addresses credential sprawl and missing usage attribution. Centralized credential management replaces scattered API keys with consolidated provider authentication per agent, removing the credential sprawl that makes model sprawl hard to detect. Usage attribution maps every LLM call to the team and agent that originated it, which is the prerequisite for chargeback. These controls apply automatically to every agent in the fleet, including newly deployed ones, without requiring developers to implement them per agent. ### Reliability Reliability is the infrastructure layer underneath observing and governing agents. Unlike a single API call, an agent makes dozens of LLM calls per task, which means every failure mode that exists in a standard API integration compounds across the full sequence.  LLM API calls fail in production from:  - Rate limits (429)  - Server errors (500, 502, 503, 504) - Timeouts - Partial responses  Without reliability infrastructure, any single failure cascades into full pipeline abort for every LLM call that follows in the sequence. The first line of defense is retry logic, which  handles transient failures on the same provider. | Retryable Status Codes | Non-Retryable Status Codes | | --- | --- | | 429 | 400 | | 500 | 401 | | 502 | 403 | | 503 | 404 | | 504 | - | However, client-side errors don't resolve on retry. Implementation uses exponential backoff with jitter and respects rate limit headers when available. Retrying without jitter produces thundering herd problems when multiple agents simultaneously hit the same rate limit and retry at the same interval. Typed fallback chains handle failures that retries cannot resolve. A single ranked provider list fails because different failure types need different fixes. The correct implementation uses three pools, each keyed to a specific failure type. For example: - **General API errors and timeouts** should route to an alternative provider - **Context window exceeded errors** should route to a model with a larger context window, not just a different provider - **Content policy violations** route to a different provider entirely, not a smaller model from the same provider So routing a context window error to a provider with the same window size or a policy violation to a smaller model on the same provider wastes the fallback. Each pool exists because the failure it handles has a different correct fix. But fallback chains don’t address repeated retries against a provider in a full outage.  Circuit breakers prevent these retry spirals from compounding. Without a circuit breaker, an agent retrying against a provider in a full outage burns time and compute indefinitely. A circuit breaker opens after a failure threshold is exceeded within a rolling time window, routes immediately to the fallback chain for the timeout duration, then closes after confirming recovery.  A provider outage that would abort a pipeline becomes a transparent fallback the application layer never sees. ### Optimization #### Smart Routing With the fleet observed and governed, routing optimizes which model handles each call. Most teams pick a model at build time and use it for everything. But new models release constantly, benchmarks shift, and the cost-accuracy tradeoffs change with them.  So a model that was the right choice six months ago may now cost twice as much as a newer alternative with better accuracy on the same tasks. And manual re-evaluation across an agent fleet is not feasible at scale. Production-traffic-aware routing replaces that manual process. Rather than manually defining which model handles which task type, it analyzes actual production traffic to cluster prompts into routing groups by shared behavior and optimization needs. It then explores candidate models across those groups, evaluating them on cost and latency directly from API responses, and on accuracy through a separate evaluation step. Because live traffic is unlabeled, evaluating accuracy requires a labeled dataset per routing group scored by an LLM judge, a reference model, or human review. The result is a router built from actual queries and accuracy requirements rather than generic benchmarks that may have nothing to do with the real workload. Unlike static model selection, a traffic-aware router continuously re-evaluates as new models enter the market and as the agent's own traffic evolves. Once live, the router directs each incoming prompt to the best model based on stated priorities, like minimizing cost, reducing latency, or protecting accuracy above a defined threshold. Those priorities are configured through goal and limit controls, which let you optimize for one primary objective while setting hard limits on the others. So a cost-optimized router won't exceed a latency SLA, and an accuracy-first router won't exceed budget. Then, at inference time, the router: - **Identifies** which routing group the request belongs to  - **Infers** the candidate model list based on the configured optimization objective  - **Applies** any hard limits on the other dimensions  - **Routes** to the model #### Prompt Optimization Prompt optimization treats prompts as versioned, tested artifacts rather than static text. In production agent systems, a poorly structured prompt doesn't just produce worse outputs but also more tokens, which multiplies across every call the agent makes. To optimize, manual prompt tuning is possible, but it’s based on trial and error. Engineers will often write a prompt, test it against a few examples, and adjust based on intuition.  Automated prompt optimization replaces that with systematic evaluation against a held-out test set and a defined metric, then iterates programmatically.  DSPy, an open-source framework from Omar Khattab at Databricks, compiles LLM pipelines into self-improving programs. In benchmarks, DSPy-optimized prompts outperformed expert-written few-shot prompt chains by up to 46% on complex reasoning tasks ([Khattab, 2023](http://arxiv.org/abs/2310.03714)). The implication for agent systems is that prompt quality is not fixed at deploy time. An agent running in production accumulates signals that can feed back into prompt improvement continuously, including: - What queries the agent handles - Where the agent fails - What outputs score well #### Caching Prompt caching reuses the processed computation from stable parts of a request so the provider doesn't reprocess them on every call. The stable portion of most agent requests like system prompts, tool definitions, policy documents, and RAG context, appears identically across thousands of requests per day.  Without caching, every request pays the [full input token cost](https://sedai.io/blog/tokenmaxxing-ending-era-tokenomics-has-begun) for that repeated content. With caching, those tokens bill at a significantly reduced rate. Both Anthropic and OpenAI support prompt caching natively. Whether a cache hit occurs depends on how the prompt is structured: stable content must appear first in the prompt and dynamic content last. If the ordering is wrong, the cache never hits. Get it right and the majority of input tokens in a high-volume agent system cost a fraction of what they did before. Cache hits also reduce latency. The provider skips reprocessing the cached prefix, which means time-to-first-token drops alongside cost. For customer-facing agents where latency affects user experience, caching addresses both dimensions at once. #### Context Window Management Agent costs compound with context length. In a multi-turn agent loop, every step appends to the context and input token count grows with each step. The problem is structural: an agent carries accumulated context from prior calls into each new request. The techniques for managing this fall into three categories:  - Trimming - Compression - Retrieval Trimming removes low-value content from history, like redundant tool results, verbose intermediate steps, and turns that no longer affect the current decision.  Compression summarizes prior context rather than including it verbatim, using hierarchical summarization where older content gets progressively compacted as the conversation grows.  Retrieval replaces full context inclusion with selective injection. The agent fetches only the specific prior context relevant to the current step, rather than maintaining the full history in every request. The practical question is which prior context actually affects the current decision, and which is just overhead. In most production agent systems, the answer is that most context is overhead, and trimming or compressing it has no measurable effect on output quality. ### Ready to optimize your AI Agents? Book a Sedai demo to speak with a technical expert. ## **AI Agent Optimization vs. LLM Optimization** AI agent optimization is a system-layer intervention. It decides:  - What an agent is allowed to call - What those calls cost and return  - Whether failures are handled gracefully - Whether each call is routed to the most cost-appropriate model LLM optimization is a model-layer intervention:  - Fine-tuning adjusts weights - RLHF aligns outputs - Prompt engineering structures inputs more effectively | | AI Agent Optimization | LLM Optimization | | --- | --- | --- | | Unit of work | Multi-step agent execution loop | Single model call | | What it changes | Routing, governance, observability, reliability | Fine-tuning, RLHF, prompt templates | | Where it applies | Production systems already running | Model development and evaluation | | Who owns it | Platform Engineering, FinOps | ML Engineering | Each optimization process addresses a different failure mode and are not substitutes. A poorly prompted model produces bad outputs regardless of how well the agent system is optimized. A well-prompted model running in an unobserved, unrouted, ungoverned agent system still produces unpredictable costs, governance gaps, and reliability failures in production.  Most engineering teams have more mature tooling at the model layer than at the agent system layer, and that’s where AI agent optimization comes in. ## **Why AI Agent Costs Are Hard to Attribute and Control** AI agent costs are hard to attribute and control because agent systems make dozens of LLM calls per task, and most of those calls are unrouted, unattributed, and unmonitored. The average large enterprise spends $11.6 million annually on AI models , up from $4.5 million in 2024 ([A16z](https://a16z.com/leaders-gainers-and-unexpected-winners-in-the-enterprise-ai-arms-race/#:~:text=The%20market%20is%20massive%20and%20%28still!%29%20growing%20faster%20than%20expected), 2026). And monthly token spend has increased 13x in the last 18 months ([Ramp](https://ramp.com/blog/trillion-dollar-ai-blindspot), 2026).  At the developer level, Microsoft has pulled engineers off Claude Code when usage ran up to $2,000 per engineer per month, a threshold that arrived faster than finance teams anticipated ([Yahoo Finance](https://finance.yahoo.com/sectors/technology/articles/microsoft-cut-engineers-off-ai-093500532.html), 2026). Visibility is the core problem. Engineering teams can see that AI spend is growing on their cloud invoice, but cannot see which agent drove which cost, which team is responsible, or whether the outcome justified the expense.  And this cost problem compounds because of how agents are built. Every unoptimized call type multiplies across every task, agent, and team running in production simultaneously. "FinOps for AI" has emerged to address this issue because the cost attribution model FinOps built for cloud infrastructure does not formally exist yet for the model call layer.  --- ## **Who Owns AI Agent Optimization?** ### Platform Engineers and VPs of Engineering Platform Engineers and VPs of Engineering own AI agent optimization in practice because they see AI cost spikes without attribution. They own the infrastructure agents run on, but not the model selection decisions individual teams make. So when FinOps asks what the agents are spending per task, they have no answer. ### FinOps Practitioners While FinOps owns cost attribution for cloud infrastructure, that visibility does not extend to model call attribution; they can't see what the agent fleet is spending or which team is responsible. They need the same cost ownership model for agents and LLMs that was built for cloud, meaning attributing costs:  - Per-team - Per-project - Per-workload  ### Enterprise Architects and CISOs Enterprise Architects and CISOs own the governance problem, typically escalated from legal or compliance. That means knowing which models are approved for use, who holds the credentials, and who is accountable for every LLM call.  Without centralized controls, none of those questions have a reliable answer because model selection happens team by team, credentials scatter, and there's no audit trail. ## **AI Agent Optimization Use Cases** ### Cost Attribution by Team and Agent With cost attribution, every LLM call is tagged with team, agent, and workflow identifiers. Engineering leadership can see which agents are driving token spend, at what cost-per-task, and whether that cost is producing measurable outcomes. ### Cost Anomaly Detection Cost anomaly detection catches and prevents runaway AI costs before it compounds. For example, if a retry loop misfires against an LLM provider returning errors or a single agent makes unnecessary frontier model calls, it can exhaust a daily budget in minutes.  Fleet-level monitoring catches these patterns early by surfacing call frequency cost-per-task spikes. ### Routing Low-Complexity Subtasks To Smaller Models Not every call in an agent workflow needs a frontier model. Summarization, classification, and extraction run accurately on smaller models at a fraction of the cost. Routing by task complexity reduces per-task cost without changing output quality on the full workflow. ### Enforcing Model Governance Fleet-Wide As agent fleets scale, new agents get deployed without inheriting existing governance controls. Org-level model access policies and centralized credentials are enforced automatically across every agent, including those deployed by teams who don't know the policy exists. A new agent gets the same approved model set and usage attribution as every existing agent, without any per-agent configuration from the developer. ### Cross-provider fallback during outages When a primary provider rate-limits or goes down, typed fallback routing shifts traffic to the appropriate alternative automatically. So, for example:  - Context window errors will route to a model with a larger window - Content policy violations will route to an alternative provider --- ## ### FAQ #### What is the difference between AI agent optimization and LLM optimization? LLM optimization is a model-layer intervention, which encompasses fine-tuning, RLHF, and prompt engineering. AI agent optimization is a system-layer intervention, which includes routing, governance, observability, and reliability applied to the agent's full execution loop.  LLM optimization addresses a single model call, while agent optimization addresses the multi-step system making dozens of calls per task. Both layers require investment and address different failure modes. #### What are the four components of AI agent optimization? 1. Observability instruments every step in the execution, which includes LLM calls, tool invocations, retrieval steps, latency, and cost. The standard for this instrumentation is OpenTelemetry GenAI semantic conventions, with fleet-level consolidation across every provider and project. 2. Governance enforces two-tier model access control at the org and project level, with centralized credential management and usage attribution per team. 3. Smart routing analyzes production traffic, clusters prompts into routing groups, and surfaces a Pareto-optimal model set per group, continuously updated as new models enter the market.  4. Reliability handles typed fallback chains keyed to failure type, exponential backoff with jitter, and circuit breakers that prevent retry spirals from compounding. #### How much do enterprises spend on AI models? The average large enterprise spends $11.6 million annually on AI models , up from $4.5 million in 2024 ([A16z](https://a16z.com/leaders-gainers-and-unexpected-winners-in-the-enterprise-ai-arms-race/#:~:text=The%20market%20is%20massive%20and%20%28still!%29%20growing%20faster%20than%20expected), 2026). #### What is LLM routing and why does it matter for AI agent optimization? LLM routing classifies each call in an agent workflow by task type and directs it to the model matched to that task's requirements. Without routing, every call in a multi-step agent workflow hits the most expensive model regardless of whether the task warrants it.  Summarization, classification, and entity extraction subtasks that run accurately on a smaller model consume frontier model compute unnecessarily. Routing applies the correct model to each call, reducing per-task cost without changing output quality on the full workflow. #### Can AI agent optimization work with agents I've already built? Yes. Agent optimization operates as an infrastructure layer below application code, which means it applies to agents already running in production without requiring rewrites. The instrumentation, governance, routing, and reliability controls attach to existing agent calls rather than replacing them. #### What is agentic AI governance? Agentic AI governance controls which models and providers agents in a fleet can use, and enforces that control automatically across every deployment. The core mechanism is two-tier model access control: org-level policies define the approved model set for the entire fleet, with project-level overrides for teams that need different configurations.  Centralized credential management replaces scattered API keys, and usage attribution maps every LLM call to the team and project that originated it for chargeback and accountability. #### What is the difference between application-layer and infrastructure-layer AI agent optimization? **Application-layer optimization** improves how an agent is built: routing logic, prompt structure, tool selection, guardrails, and evaluation pipelines. **Infrastructure-layer optimization** controls what the agent runs on: which models are allowed, how calls are routed across providers, what each task costs, and how failures are handled automatically. Application-layer work typically happens at build time. Infrastructure-layer optimization runs continuously in production, across every agent already deployed, without requiring a rewrite. --- *See how Sedai manages AI agent optimization across your entire fleet — no code changes required. [Book a demo to see how](https://sedai.io/demo).* --- # How Much Isolation Do AI Agents Actually Need? Source: https://sedai.io/blog/how-much-isolation-ai-agents-need AWS, Microsoft, Google, and Anthropic all isolate AI agent sessions differently. Sedai's engineering leaders weigh in on how much isolation agents really need. When a tenant-isolation check failed in Asana's MCP server, roughly 1,000 companies using Asana's AI integration could access each other's project data for five weeks straight.  Failures like that are why [AWS, Microsoft, Google, and Anthropic recently rebuilt their agent runtimes](https://thenewstack.io/agent-session-aware-runtime/) the same way: every session gets its own isolated environment, and the session, not the request, is what the platform schedules, secures, and bills for.  But none of them agree on how strong that isolation should be, because stronger isolation costs speed and money. So I asked our engineering leaders: how much isolation do AI agents actually need to keep data safe? --- ## Picking One Isolation Level Is the Wrong Question [***Benjamin Thomas***](https://www.linkedin.com/in/benjamin-thomas-97691a8/) *(Co-Founder & CTO)* I don't think you can pick one isolation level globally. Isolation strength should follow the trust level of what's running in the session. If an agent is executing arbitrary model-generated code with network access and credentials in scope, I want VM-level isolation, Firecracker-style, and I'll take the cold start.  That cost is also more manageable than the framing suggests, since you can start with the lighter VM at session initiation and escalate to heavier isolation only when the workload calls for it. Which really means the problem isn't the sandbox but the routing layer. "Isolation strength should follow the trust level of what's running in the session." — Benjamin Thomas, Co-Founder & CTO Something has to look at each incoming session and decide its trust tier, placement, and lifecycle continuously rather than as a one-time config. That kind of workload-aware decisioning is where I'd expect the real gains to show up, both on cost and on blast radius. The other wrinkle is that not all sessions are individual. Once you have shared sessions within a team, or long-running ones that outlive a single user's involvement, the question shifts from “What can escape the sandbox?” to “Who owns the session right now?” The Asana case is exactly that: isolation held but the identity binding didn't. For shared and long-lived sessions, the session must be a security principal in its own right, not just a compute unit with a timeout. And the routing layer needs to carry that identity context along with everything else. --- ## Your Sandbox Won't Save Your Work [***Aby Jacob](https://www.linkedin.com/in/aby-jacob?miniProfileUrn=urn%3Ali%3Afs_miniProfile%3AACoAAA9Ma6oBZ5hCBQxrdK-4KrTlgvFMDsGbulw)*** *(VP of Engineering)* The sandbox is a security boundary, not a durability boundary, and many agentic workloads will fail if/when developers conflate the two.  While the session is increasingly becoming the preferred unit of compute, it seems durability and recovery are intentionally outside of its scope. A sandbox was never meant to remember, just execute.  For example, when a microVM crashes halfway through an agent task, everything inside it disappears including working files, runtime environment, partially completed routines and cached data. The session ID may still exist, but it now points to an empty container. "The sandbox is a security boundary, not a durability boundary." — Aby Jacob, VP of Engineering The platform won't recover any of this, so we should design as if a crash is guaranteed: persist state outside the sandbox after every turn, make the environment reproducible from code or scripts, and make every tool call idempotent so a retry doesn't cause unintended side effects.  Resilient agents should externalize their state, treat the sandbox as ephemeral, and design every transaction to tolerate retries and partial failures.  An additional benefit of this architecture is portability. Once state lives outside the sandbox, the execution environment becomes a replaceable commodity that can be rented from any compatible provider, significantly reducing vendor lock-in. --- ## Session Isolation Costs Less Than You Think [***Shankar Jothi***](https://www.linkedin.com/in/shankarj/) *(VP of Engineering, ML)* The isolation overhead people worry about is mostly a cold-start cost rather than a steady-state one (at least for compute-bound serving paths). You pay it when a session is first provisioned, and it fades once the environment is warm and requests route back to it.  So the real variable isn't isolation itself, but how often you're establishing sessions — something lifecycle settings like idle timeouts directly influence. "The isolation overhead people worry about is mostly a cold-start cost rather than a steady-state one." — Shankar Jothi, VP of Engineering, ML It's also worth being honest that this isn't measured as cleanly as it could be. Startup latency often gets folded into a single end-to-end number alongside inference time. The more useful practice is to decompose the path into the platform-managed portion and the application-defined portion, tracking those separately. That lets you reason about regressions instead of guessing at them. More broadly, this still feels like an early and evolving space. The primitives are converging, but the operational maturity around cost, portability, and observability is still being worked out. That's where I expect most of the interesting engineering to happen over the next year. --- *Running agents in production? [**See what they're actually costing you](https://sedai.io/demo).***