6 min to read

Safe Pod Termination in Kubernetes - A Complete Guide

Featured image


🎯 Overview

Managing Pod termination safely is crucial in Kubernetes operations, especially during maintenance, updates, or scaling activities.

🛠️ Pod Management Commands

In Kubernetes, Pod termination can be handled in several ways depending on operational requirements such as maintenance, update, or extension operations.

Next, we will learn how to safely manage Pod termination using Kubernetes commands such as cordon, uncordon, drain, and scale.

# Cordon - Mark node as unschedulable
kubectl cordon <node-name>

# Uncordon - Mark node as schedulable
kubectl uncordon <node-name>

# Drain - Safely evict pods from node
kubectl drain <node-name> --ignore-daemonsets --delete-local-data

# Scale - Adjust replica count
kubectl scale --replicas=0 deployment <deployment-name> -n <namespace>
kubectl scale --replicas=0 statefulset <statefulset-name> -n <namespace>

📊 StatefulSet vs Stateless Applications

For tasks that require safe shutdown and re-run of Pod, it is common to use the Scale command to make Replica 0.

graph LR A[Start] --> B{Scale Up/Down?} B --> |Scale Up| C[Increase Replica Count] C --> D[New Pods Scheduled] B --> |Scale Down| E[Decrease Replica Count] E --> F[Pods Terminated] D --> G[End] F --> G[End]


StatefulSet

Scale down progressively: Reducing the number of replicas in StatefulSet triggers controlled termination of the pad, starting with the highest order. This is important for applications that need to end in order (e.g., databases).

For example, MariaDB Galera Cluster should be terminated after 2 → 1 → 0 sequentially scaling.

# StatefulSet Progressive Scaling
# Example for MariaDB Galera Cluster
kubectl scale --replicas=2 sts app-db -n <namespace>
kubectl scale --replicas=1 sts app-db -n <namespace>
kubectl scale --replicas=0 sts app-db -n <namespace>

# Stateless Direct Scaling
kubectl scale --replicas=0 deploy api -n <namespace>

Stateless

Direct scaling or deletion: Stateless applications do not remain internal, allowing you to scale back or delete pods without much concern for data consistency.

If you are using a general stateless application, you can scale directly to replicas=0 as follows.

kubectl scale --replicas=0 deploy api -n <namespace>

Additional Considerations

🔒 Pod Disruption Budget (PDB)

Pod Disruption Budget (PDB) is a Kubernetes resource that helps ensure that a specified minimum number of Pods are available during volatile disruption.

This is particularly important for maintaining the availability and resilience of applications during tasks that may affect Pod availability, such as node maintenance, cluster upgrades, or other disruptions managed by cluster administrators.

Here, volatile disturbances include all commands such as cordon, uncordon, drain, and scale. Voluntary disturbances are typically planned tasks that can be controlled and scheduled at times that minimize impact on applications.

This is different from invitational disturbances such as unpredictable and unmanageable hardware errors or unexpected node loss in PDB.

PDB Example

The PDB ensures that the Kubernetes system complies with the limitations set by the PDB, even during volitional interruptions, so that there is no significant impact on application performance and availability.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app


🔄 Node Management Workflow

Cordon, uncordon, and drain commands are used to upgrade and maintain Node and Pod.

graph LR A[Start] --> B{Node Maintenance?} B --> |Yes| C[Cordon Node] C --> D[Drain Node] D --> E[Perform Maintenance] E --> F[Uncordon Node] F --> G[End] B --> |No| G[End] 


1. Drain Process:
   - Automatically cordons node
   - Safely evicts pods
   - Respects PDB
   - Waits for pod termination

2. Maintenance Steps:
   - Cordon: Prevent new pod scheduling
   - Perform maintenance
   - Uncordon: Re-enable pod scheduling

3. Best Practices:
   - Always check PDB settings
   - Monitor pod health
   - Plan maintenance windows

⚙️ Kubernetes Node Management Commands

🔧 Command 📄 Purpose 📦 Impact on Pods 🖥️ Impact on Node
🚫 cordon Mark node as unschedulable ❌ None ⛔ Prevents new pods from being scheduled
✅ uncordon Mark node as schedulable ❌ None ✔️ Allows new pods to be scheduled
📤 drain Evacuate all pods from the node 🚪 Evicts all pods 🚫 Marks node as unschedulable
📈 scale Adjust the number of pod replicas ➕➖ Adds or removes pods ✔️ No impact on node

💡 Key Considerations

1. Pod Disruption Budgets:
   - Define minimum available pods
   - Ensure service availability
   - Handle voluntary disruptions

2. Graceful Termination:
   - Respect SIGTERM signals
   - Handle shutdown procedures
   - Monitor termination process

3. Application State:
   - Consider stateful vs stateless
   - Manage data persistence
   - Plan scaling operations


📚 Reference