Kubernetes Controllers in Production

Kubernetes does not lack data. It lacks a reusable layer of operational meaning.

This ebook is about turning fragmented cluster state into deterministic, graph-aware context for humans, tools, and AI agents.

The central idea is simple: troubleshooting Kubernetes is not primarily a data-access problem.

The cluster already has the facts: Pods, owner references, Services, EndpointSlices, PVCs, Events, status fields. The hard part is reconstructing the situation from all of those fragments, under pressure, without losing the relationships that make the facts useful.

This book uses `kctx`, a small read-only Kubernetes context engine, as a concrete reference design for that missing layer.

It explores how to model operational context as:

• entities
• relations
• signals
• deterministic JSON
• local dependency graphs
• service traces
• namespace health snapshots
• CRD adapters
• safer inputs for AI and MCP tools

It is not a beginner Kubernetes book, and it is not a pitch for autonomous remediation.

It is for SREs, platform engineers, Kubernetes operators, infrastructure developers, and AI tooling builders who have felt the gap between raw YAML and actual operational understanding.

The thesis:

AI systems should consume structured operational context, not reconstruct it from raw YAML and logs.

If you are building platform tooling, incident workflows, internal developer platforms, Kubernetes automation, or AI-assisted operations, I think this is one of the primitives we are missing.

A controller can create a database. Who keeps the promise after that?

A response is lost and the next retry creates a second resource. An emergency dashboard change disappears when reconciliation restores declared state. Deletion stalls, a rollback cannot interpret a new checkpoint, or the only engineer who understands recovery leaves the team.

These are not just implementation problems. They expose decisions about identity, authority, evidence, and responsibility.

This book starts before scaffolding: does the work need continuous ownership, or would a Job, workflow, existing platform, or standalone service meet the need? If a controller is justified, what promise is the team prepared to automate and keep?

Through one fictional ManagedDatabase case study, the book connects design choices to their consequences under retries, stale observations, concurrent changes, partial failure, deletion, and upgrades. The same resource follows the argument from its API contract to incident response and eventual handover.

You will learn to:

• choose a control model and decide whether Kubernetes should be the public API
• define who may change desired and external state, and what status can truthfully report
• preserve operation identity across lost responses without confusing retries with new work
• design recovery around independent commits, cache lag, write conflicts, and reliable wake up paths
• retain cleanup responsibility until the evidence permits its release
• select tests and experiments that can challenge a claim, not merely repeat its assumptions
• connect observability, backpressure, compatibility, and runbooks to operational decisions
• review a reconciler through interrupted executions and transfer responsibility through demonstrated capability

Written for experienced engineers, Principal Engineers, platform architects, operators, and technical leaders, the book assumes familiarity with Kubernetes workloads and basic API concepts. It does not require Go proficiency or access to the companion repository.

Short snippets and diagrams make failure mechanisms concrete. Linked Source Notes provide primary references and bounded experimental evidence. Optional Go examples let implementers inspect those experiments; they are supporting material, not a complete production operator. A copyable Controller Design Record helps turn the reasoning into decisions, owners, evidence, and explicit conditions for proceeding.

This is not a scaffolding tutorial or a catalogue of universal best practices. It is a guide to deciding what may safely happen next, what must be learned first, and who remains responsible when automation cannot finish the job.