Running Highly Available Databases in Kubernetes — Part 2
Architecture Patterns, Operational Reality, and Responsible Design
Availability Is a System Property, Not a Feature
Before we dive into patterns and practices, one principle must be clear: high availability is not something Kubernetes or a database magically provides. It emerges from the careful orchestration of storage, networking, operational practices, and human oversight. Kubernetes is a platform; it can host highly available databases, but the guarantees come from the design and operational discipline layered on top of it.
Deploying a replicated PostgreSQL cluster, a CockroachDB cluster, or an etcd ensemble on Kubernetes does not automatically make it resilient. Kubernetes’ self-healing -restarting pods or rescheduling nodes, is only one small piece of the puzzle. The database’s own replication logic, understanding of quorum, the reliability of storage, and network stability all matter. Treat Kubernetes as critical infrastructure, not a source of HA magic.
Before You Commit: Critical Design Considerations
Running highly available databases on Kubernetes is very different from deploying web services. There are several hard questions that must be answered before any cluster is deployed.
Define the Failure Model. Many teams assume that Kubernetes will “take care” of failures, but it cannot differentiate between pod readiness and database health. A pod restart may hide deeper issues, such as replication lag or partial write failures. Your team must be able to clearly define which failures the database must survive: node crashes, pod eviction, network partitions, disk latency spikes, or zone failures.
Ownership and Responsibility. When a database runs inside Kubernetes, operational boundaries collapse. Who is responsible for failovers, replication issues, or corruption detection? Operators can automate many tasks, but they do not assume responsibility. Undefined ownership increases the likelihood of extended outages.
Storage is Foundational. Kubernetes volumes alone do not guarantee the durability or ordering semantics that databases rely on. Storage must be chosen carefully: consider replication, latency, consistency, and recovery guarantees. A database may be highly available at the application layer, but if storage introduces ordering anomalies or latency spikes, the system can silently corrupt itself. Testing restore times under realistic load is essential.
Kubernetes Itself is a Failure Domain. Control plane outages, network misconfigurations, or cluster-wide upgrades can impact databases. Your design must explicitly account for these events as first-class hazards. HA is only meaningful when the full system - including Kubernetes is treated deliberately.
Architecture Patterns for Highly Available Databases
There is no single “correct” way to run a database in Kubernetes, but some patterns are widely adopted in production. Understanding the trade-offs behind these patterns is critical.
Single Primary with Multiple Replicas
This is the classic approach for relational databases like PostgreSQL or MySQL: one writable primary and multiple read replicas. The database itself manages replication and failover logic. Kubernetes handles scheduling, volume attachment, and pod restarts, but it does not manage leadership.
Problems can arise when Kubernetes evicts or restarts pods without awareness of their role. For example, if the primary is evicted during a node upgrade, a replica may temporarily assume leadership, only to be replaced once the primary pod is rescheduled.
Mitigation involves a combination of operational controls and database-aware configurations:
-
PodDisruptionBudgets (PDBs): Prevent multiple replicas from being evicted simultaneously, protecting quorum.
-
Zone-aware scheduling: Anti-affinity rules ensure replicas are not co-located in the same failure domain.
-
Monitoring replication lag: Track how quickly replicas catch up to the primary, and route traffic to lagging replicas carefully.
Understanding these subtleties is essential to avoid unnecessary failovers or data inconsistencies.
Consensus-Based Clusters
Distributed databases like etcd, CockroachDB, and some distributed SQL systems rely on consensus protocols (Raft or Paxos). In these systems, every node participates in decision-making, and writes require quorum. Node identity and membership are critical. Automated pod rescheduling or network blips can easily break quorum, temporarily reducing availability.
To manage this, it is crucial to combine Kubernetes primitives with operational practices:
-
Topology spread constraints: Distribute replicas across nodes and zones to minimize the risk of quorum loss.
-
Anti-affinity rules: Prevent multiple nodes from landing in the same failure domain.
-
Controlled rolling updates: Apply disruptions one node at a time to avoid triggering unnecessary leader elections.
-
Database-aware health checks: Ensure readiness probes consider replication and leader status, not just pod liveness.
This pattern demonstrates that HA is a system property. Even if all pods report “Ready,” a cluster can become unavailable if quorum is lost.
Operator-Managed Databases
Operators encapsulate operational knowledge into controllers. They automate initialization, scaling, failover, upgrades, and some backup tasks.
Operators are incredibly useful; they encode repeatable, auditable workflows and reduce human error. However, they are not magic. They cannot guarantee correctness under all failure modes, nor do they replace deep understanding of database behavior. Misconfigurations outside an operator’s scope or unexpected storage anomalies can still cause outages or corruption.
Operators should be treated as automation tools, not substitutes for expertise. Their value lies in enforcing safe sequences, but responsibility for the database remains with the team.
Handling Partial Failures
High availability is not just about pods running. It is about managing all partial failures that can affect database correctness. These include network partitions, replica lag, disk anomalies, or leader unavailability.
Recovery requires more than simply restarting pods. It often involves:
- WAL replay and replica catch-up to ensure all committed transactions are durable.
- Controlled reintegration of nodes into the cluster.
- Ensuring quorum before accepting new writes to avoid split-brain scenarios.
Kubernetes cannot inherently distinguish between pod health and database health. Misinterpreting readiness probes can trigger unnecessary failovers or even corruption. Solutions include:
- Database-aware readiness and liveness probes.
- Custom controllers or operators that understand roles, quorum, and failover sequences.
- Careful orchestration of rolling updates, node drains, and maintenance operations.
Backup, Restore, and Disaster Recovery
Snapshots are convenient but are often not application-consistent. True HA requires tested restore procedures.
- Define Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO).
- Ensure restores are practically achievable under production load, not just in staging.
- Combine storage snapshots with application-consistent backups where supported.
- Automate periodic verification of backups with restore tests to catch issues before they matter.
Monitoring and Observability
Kubernetes metrics alone; pod readiness, CPU, memory, cannot ensure database correctness. Teams must track database truths:
- Replication lag and leader election events
- Write durability and checkpoint consistency
- Disk latency and I/O spikes
Alerts must reflect conditions that truly threaten correctness, not just whether pods are running. Observability is as important as deployment patterns for achieving HA.
When Not to Run Databases in Kubernetes
Sometimes, the safest choice is not to run a database in Kubernetes:
- Mission-critical workloads without deep expertise in database internals.
- Strict compliance or regulatory requirements that demand proven guarantees.
- Storage performance that is unpredictable or inconsistent.
- Unstable Kubernetes clusters with frequent control plane or networking issues.
- Teams expecting HA to “just happen” automatically.
Managed database services exist to address exactly these challenges. Using Kubernetes in these scenarios can increase operational risk.
Decision Framework
Before committing to a Kubernetes-based HA database, answer honestly:
- Can you define the failure model and know which failures matter?
- Do you understand your database internals and replication mechanics?
- Can you guarantee storage semantics, including latency, ordering, and durability?
- Can you restore data under production pressure, predictably and quickly?
- Is your team prepared to own outages end-to-end?
If the answer to any of these is “no,” pause and reconsider the approach.
Conclusion: Responsibility Over Abstraction
Running highly available databases in Kubernetes is demanding but feasible. It trades managed safety for engineered discipline, requiring a deep understanding of database internals, storage guarantees, and operational practices.
High availability is not a checkbox, feature, or magic provided by Kubernetes. It is the outcome of deliberate design, careful testing, and ongoing operational rigor. For teams willing to invest in these areas, Kubernetes can successfully host HA databases. For teams that are not, it can silently amplify risk.
The difference is judgment, not technology.
