RKE2 Node Certificate Renewal by Service Restart

Purpose

Renew RKE2 node certificates that are within the automatic renewal window by restarting the appropriate RKE2 service one node at a time.

For RKE2 versions before the May 2025 maintenance releases listed in the official documentation, including v1.32.4+rke2r1, warning and automatic renewal are triggered within 90 days of expiration. RKE2 renews qualifying certificates when RKE2 starts, reusing the existing keys and extending certificate validity.

References

• RKE2 Certificate Management
• RKE2 Advanced Options and Certificate Rotation
• Prerequisite: Accessing RKE2 Administrative Commands

Scope

Validated topology:

• Three RKE2 server nodes with control-plane and etcd roles
• Two RKE2 worker nodes
• RKE2 v1.32.4+rke2r1

Adapt node names before use.

Risk and change controls

Potentially disruptive: Restarting rke2-server temporarily removes one control-plane and etcd member. Restarting rke2-agent temporarily interrupts Kubernetes services on that worker.

• Perform during an approved maintenance window.
• Restart only one node at a time.
• Do not proceed until the previous node is Ready and validated.
• Confirm a recent etcd snapshot before restarting server nodes.
• Keep an administrative shell open on a healthy server node for cluster validation.

Example node order

Server nodes:

1. pkm1con01s550
2. pkm1con01s551
3. pkm2con01s550

Worker nodes:

1. pkm1con01s601
2. pkm2con01s601

Preparation from a healthy server node

Use the full-path commands from Accessing RKE2 Administrative Commands, or enter a configured root shell:

sudo -i
export PATH="/usr/local/bin:/var/lib/rancher/rke2/bin:$PATH"
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

Verify initial cluster state:

kubectl get nodes
kubectl get pods -A

Verify a recent etcd snapshot:

rke2 etcd-snapshot ls

Warnings that unrelated server configuration flags are skipped by the snapshot subcommand do not, by themselves, indicate snapshot failure. Confirm that the command lists a recent snapshot with a plausible creation time and size.

Phase 1: Server, Control-Plane, and etcd Nodes

Perform this procedure on one server node at a time.

1. Record certificate status on the target server

On the target server:

sudo /usr/local/bin/rke2 certificate check --output table

If you want a text file export of the certificates to compare before vs. after:

sudo /usr/local/bin/rke2 certificate check --output table \
  | sudo tee /root/certs-before.txt

Confirm the expiring leaf certificates show WARNING. CA certificates normally have separate, later expiration dates.

2. Restart the RKE2 server service

Potentially disruptive: This restarts the local etcd member and control-plane services.

sudo systemctl restart rke2-server

If the command returns an error, do not restart another node. Check actual service state and startup logs:

sudo systemctl status rke2-server -l --no-pager
sudo journalctl -u rke2-server.service -n 150 --no-pager

A successful recovery includes an active service and log messages indicating that etcd is ready, the API server is running, and RKE2 is up and running.

3. Validate node and cluster recovery

From a healthy server node:

kubectl get nodes
kubectl get pods -A

Do not continue until the restarted node is Ready and critical cluster components are healthy.

4. Verify certificate renewal

On the restarted server:

sudo /usr/local/bin/rke2 certificate check --output table

If you want a text file export of the certificates to compare before vs. after:

sudo /usr/local/bin/rke2 certificate check --output table \
  | sudo tee /root/certs-after.txt

Expected result:

• Previously warning leaf certificates now show OK.
• Leaf expiration dates have moved forward.
• CA expiration dates remain unchanged unless a separate CA rotation was intentionally performed.

Optional comparison:

sudo diff -u /root/certs-before.txt /root/certs-after.txt

5. Repeat for remaining servers

Repeat Steps 1 through 4 for each server, one at a time.

Phase 2: Worker Nodes

Worker maintenance decision

A normal Kubernetes drain may require --delete-emptydir-data. In this cluster, Prometheus storage was confirmed to use emptyDir, meaning eviction would delete the pod-local metrics database. Therefore, this procedure uses cordon, restart agent, validate, uncordon, without draining.

Tradeoff: workloads remain on the node and experience the worker's brief service interruption rather than being gracefully evicted. Confirm this approach is acceptable under the maintenance change.

1. Record certificate status on the worker

SSH to the target worker:

sudo /usr/local/bin/rke2 certificate check --output table

If you want a text file export of the certificates to compare before vs. after:

sudo /usr/local/bin/rke2 certificate check --output table \
  | sudo tee /root/certs-before.txt

2. Cordon the worker

From a server node:

kubectl cordon pkm1con01s601
kubectl get nodes

Cordon prevents new pods from being scheduled to the worker. It does not evict existing pods.

3. Restart the RKE2 agent

On the target worker:

Potentially disruptive: Kubernetes node services briefly stop. Existing workloads on the worker may experience interruption.

sudo systemctl restart rke2-agent

If the command reports a failure:

sudo systemctl status rke2-agent -l --no-pager
sudo journalctl -u rke2-agent.service -n 150 --no-pager

Do not proceed to another worker until service and node health are restored.

4. Validate worker recovery

From a server node:

kubectl get nodes
kubectl get pods -A -o wide

Wait until the worker is Ready. Review affected workloads for unexpected failures or restart loops.

5. Verify certificate renewal

On the restarted worker:

sudo /usr/local/bin/rke2 certificate check --output table

If you want a text file export of the certificates to compare before vs. after:

sudo /usr/local/bin/rke2 certificate check --output table \
  | sudo tee /root/certs-after.txt

Expected result:

• Previously warning leaf certificates now show OK.
• Expiration dates have moved forward.

Optional comparison:

sudo diff -u /root/certs-before.txt /root/certs-after.txt

6. Uncordon the worker

From a server node:

kubectl uncordon pkm1con01s601
kubectl get nodes

7. Repeat for the remaining worker

Repeat Steps 1 through 6 for each remaining worker, one at a time.

Alternative: Drain the Worker

Use this only after reviewing local ephemeral storage, PodDisruptionBudgets, replica placement, and storage attachment behavior.

Dry-run assessment:

kubectl drain pkm1con01s601 \
  --ignore-daemonsets \
  --dry-run=client

• --ignore-daemonsets allows drain to continue without deleting DaemonSet-managed pods.
• --delete-emptydir-data authorizes deletion of data in pod-local emptyDir volumes when affected pods are evicted.

Destructive to ephemeral pod data: Do not add --delete-emptydir-data until each reported pod has been reviewed and the loss of its emptyDir contents is acceptable.

If formally approved:

kubectl drain pkm1con01s601 \
  --ignore-daemonsets \
  --delete-emptydir-data

After maintenance and validation:

kubectl uncordon pkm1con01s601

Final Validation

From a server node:

kubectl get nodes
kubectl get pods -A

On every node:

sudo /usr/local/bin/rke2 certificate check --output table

Success criteria

• [ ] Every node was handled one at a time.
• [ ] All five nodes report Ready.
• [ ] All target leaf certificates report OK with extended expiration dates.
• [ ] CA certificate dates were not unexpectedly changed.
• [ ] No critical pods remain in Pending, CrashLoopBackOff, or repeated restart states.
• [ ] Both workers are uncordoned.
• [ ] Maintenance evidence and before/after certificate output are retained.

Stop and escalate conditions

Stop the procedure and investigate before proceeding if any of the following occurs:

• A server or worker does not return to Ready.
• RKE2 remains inactive or failed.
• etcd does not become ready on a restarted server.
• Certificate status remains WARNING after a confirmed successful restart.
• Critical workloads fail to recover.
• More than one etcd/control-plane node is unavailable.