The Reconciler's Handbook

A controller can create a database. Who keeps the promise after that?

A response is lost and the next retry creates a second resource. An emergency dashboard change disappears when reconciliation restores declared state. Deletion stalls, a rollback cannot interpret a new checkpoint, or the only engineer who understands recovery leaves the team.

These are not just implementation problems. They expose decisions about identity, authority, evidence, and responsibility.

This book starts before scaffolding: does the work need continuous ownership, or would a Job, workflow, existing platform, or standalone service meet the need? If a controller is justified, what promise is the team prepared to automate and keep?

Through one fictional ManagedDatabase case study, the book connects design choices to their consequences under retries, stale observations, concurrent changes, partial failure, deletion, and upgrades. The same resource follows the argument from its API contract to incident response and eventual handover.

You will learn to:

• choose a control model and decide whether Kubernetes should be the public API
• define who may change desired and external state, and what status can truthfully report
• preserve operation identity across lost responses without confusing retries with new work
• design recovery around independent commits, cache lag, write conflicts, and reliable wake up paths
• retain cleanup responsibility until the evidence permits its release
• select tests and experiments that can challenge a claim, not merely repeat its assumptions
• connect observability, backpressure, compatibility, and runbooks to operational decisions
• review a reconciler through interrupted executions and transfer responsibility through demonstrated capability

Written for experienced engineers, Principal Engineers, platform architects, operators, and technical leaders, the book assumes familiarity with Kubernetes workloads and basic API concepts. It does not require Go proficiency or access to the companion repository.

Short snippets and diagrams make failure mechanisms concrete. Linked Source Notes provide primary references and bounded experimental evidence. Optional Go examples let implementers inspect those experiments; they are supporting material, not a complete production operator. A copyable Controller Design Record helps turn the reasoning into decisions, owners, evidence, and explicit conditions for proceeding.

This is not a scaffolding tutorial or a catalogue of universal best practices. It is a guide to deciding what may safely happen next, what must be learned first, and who remains responsible when automation cannot finish the job.