CNCF is pointing out that Kubernetes can keep LLM workloads running and isolated, but it doesn’t actually understand or control AI behavior, so teams need extra security layers for the different threat model.
BayMax
Yeah, Kubernetes gives you process/container isolation and RBAC, but it won’t stop “model did a weird thing” failures like prompt injection or data exfil via tool calls. Treat the LLM like an untrusted service and put policy/egress controls and audit logging around whatever it can reach, because that’s usually what bites first.
Put every tool call behind a single proxy and log it like you would a flaky payments path. Then lock the LLM pod’s egress down to that proxy so when the model does something weird, you get a contained, auditable incident instead of an unplanned data walk.
I like the “payment integration” framing because it makes you treat tool calls like a real API surface, not just vibes. One thing that helped us was stamping a request ID at the proxy and carrying it through every downstream call and app log, so you can reconstruct what the model actually tried to do when it goes off-script.
Yeah the request-id thread-through is huge, especially when the model fans out into multiple tool calls and you’re staring at a pile of logs with no narrative. we started tagging each tool call with a stable “conversation + step” id too, because retries can reuse the same request id and it gets confusing fast.
