The agentic DevOps stack I actually deploy in 2026

Every few weeks a client asks me what tools they should adopt for agentic DevOps. They have read the blog posts, seen the conference talks, and want a shopping list. I give them one, but not before I tell them the thing they do not want to hear: most of these tools are sandbox projects that will be abandoned in eighteen months. Adopt them for what they teach you, not for what they promise.

Here is the stack I actually deploy in production for clients in 2026. Not the theoretical stack. The one I have wired together, debugged at 2am, and would recommend to another engineer without hedging.

The runtime: kagent

kagent is a CNCF sandbox project that runs AI agents as Kubernetes resources. You declare an agent as a CRD, give it tools via MCP, and a controller reconciles it like any other Kubernetes workload. That sentence sounds simple. The implications are not.

What kagent gets right is that it treats agents the way Kubernetes treats everything else: as declarative resources that can be versioned, rolled back, audited, and observed. An agent is not a script running on someone’s laptop. It is a resource in the cluster with a controller, a service account, and RBAC. It lives inside the system it operates on.

I deploy kagent for clients who want agents that can inspect cluster state, generate Kyverno policies, triage incidents, and execute remediation under human approval. The CRD-based approach means the agent definition is in Git, reviewed in PRs, and promoted through environments like any other workload.

Where it breaks: kagent is young. The API is not stable. I have seen upgrades break agent definitions. Pin your versions and test upgrades in staging before promoting. Do not run kagent in production without a rollback strategy for the kagent controller itself.

When I would not use it: if you do not already run Kubernetes, kagent is not the reason to adopt it. If your team has never written a CRD or a controller, kagent will feel like black magic. Learn the Kubernetes extension model first.

The diagnostician: K8sGPT

K8sGPT scans your cluster, finds problems, and explains them in plain language. It reads events, logs, and resource state, then produces a natural language diagnosis with suggested remediation. For teams that are new to Kubernetes, it is a teaching tool. For teams that are experienced, it is a triage layer that catches things a human might miss.

I deploy K8sGPT as a periodic scanner, not as a real-time alerting system. It runs on a cron, scans the cluster, and posts findings to a Slack channel. Engineers read the findings when they have time. It is not pager-worthy. It is a second pair of eyes that never gets tired.

Where it breaks: K8sGPT’s diagnoses are only as good as its access to cluster state. If you run it with a restricted service account, it will miss things. If you run it with cluster-admin, you are giving an AI tool god-mode access to your production cluster. Use Kyverno or OPA Gatekeeper to restrict what K8sGPT can actually do, even if it can read everything.

When I would not use it: if your cluster is small enough that one engineer can hold the entire state in their head, K8sGPT adds noise without value. It shines at scale, when the cluster is too large for any single human to monitor manually.

The investigator: HolmesGPT

HolmesGPT is the tool I am most careful recommending, because it is the one clients get most excited about and the one most likely to be misused.

HolmesGPT investigates production incidents. It reads traces, metrics, logs, and runbooks, walks the evidence chain, and drafts a root-cause narrative. For a team that is drowning in alerts and struggling to correlate signals across Prometheus, Loki, and Jaeger, HolmesGPT can compress a two-hour investigation into ten minutes.

The danger is that teams start trusting HolmesGPT’s diagnoses without verification. I have seen cases where HolmesGPT’s narrative was confident, detailed, and wrong. The investigation was plausible but the root cause was a network policy that HolmesGPT did not have access to read.

I deploy HolmesGPT with a hard rule: its output is a hypothesis, not a conclusion. It goes into the incident channel as “HolmesGPT suggests…” not “Root cause identified.” A human engineer reviews the hypothesis, verifies it against the evidence, and either confirms or rejects it. The time savings come from having a starting point, not an ending point.

Where it breaks: HolmesGPT is only as good as the telemetry it can read. If your observability stack has gaps, and every observability stack has gaps, HolmesGPT will fill those gaps with plausible-sounding guesses. Make sure your observability is solid before adding an AI investigator on top of it.

When I would not use it: if your incident response is already fast and your team is not drowning in alerts, HolmesGPT is overhead. It is a tool for teams that have a signal-to-noise problem, not for teams that already have clean incident response.

The policy layer: Kyverno with agent awareness

Kyverno is not new. It is a CNCF incubating project that enforces policy as code on Kubernetes. What is new in 2026 is that Kyverno policies can now be generated, validated, and mutated through natural language when paired with kagent or similar agentic frameworks.

The pattern I deploy: a platform engineer describes a policy in natural language (“all pods must have resource limits, all images must come from our registry, no privileged containers”), kagent generates the Kyverno policy YAML, the policy is reviewed in a PR, and Kyverno enforces it. The agent does not enforce policy. It writes policy. Humans review it. Kyverno enforces it.

This separation matters. The agent is creative. Kyverno is deterministic. You want the creativity in the authoring step and the determinism in the enforcement step. Mixing them up is how you get an AI making security decisions in production without review.

Where it breaks: Kyverno policy generation via agents is only as good as the prompt. If you ask for “a policy that blocks bad images,” you will get a policy that blocks something, but probably not what you meant. Be specific. Review every generated policy line by line. The agent is a draftsperson, not a reviewer.

The runtime security: KubeArmor

KubeArmor provides runtime security enforcement for Kubernetes workloads. It discovers security posture, enforces zero-trust boundaries, and can intercept threats at machine speed. I deploy it for clients running agentic workloads, because agents that can act on infrastructure need runtime guardrails that work faster than a human can respond.

The threat model is simple: an agent with cluster access can do damage if it hallucinates a destructive action, if its MCP tool gateway is misconfigured, or if its prompt is manipulated. KubeArmor adds a layer of runtime enforcement that does not depend on the agent behaving correctly. Even if the agent goes rogue, KubeArmor blocks the action at the kernel level.

Where it breaks: KubeArmor’s enforcement rules need to be carefully scoped. Too broad and you block legitimate actions. Too narrow and you might as well not have it. Spend time on the rules. They are the entire value of the tool.

The protocol: MCP

MCP (Model Context Protocol) is not a tool you deploy. It is the protocol that connects agents to tools. Every agent in this stack reaches its tools through MCP servers that expose scoped, audited, revocable capabilities.

I deploy MCP servers as the boundary between agents and infrastructure. An agent that needs to run kubectl does not get shell access. It gets an MCP server that exposes a scoped subset of kubectl commands, logs every invocation, and can be revoked without touching the agent itself.

This is the part of the stack that clients understand last but need most. MCP is not a feature. It is the security boundary that makes agentic DevOps safe enough to run in production. Without it, you are giving an AI tool direct access to your cluster and hoping it behaves. With it, you have a governed, audited, revocable capability boundary.

What I do not deploy

I do not deploy autonomous agents that can execute infrastructure changes without human approval. I do not deploy agents that can modify production data. I do not deploy agents that have credentials I cannot revoke in one command. I do not deploy agents whose actions are not logged and traceable.

These are not limitations of the tools. They are principles of operating agents in production. The tools are ready. The discipline is what makes them safe.

The honest summary

This stack works. I deploy it for clients and it reduces noise, speeds up incident response, and catches issues humans miss. But every one of these tools is young, every one of them has rough edges, and every one of them will be different in twelve months.

Adopt them because they teach your team to think about agents as governed, observable, declarative workloads. That mental model will outlast every tool in this list. The specific tools will change. The principles will not.