Skip to content

Rocky Linux Kubernetes Cloud-Init VMs

Setting Up a k3s Kubernetes Cluster on Rocky Linux in Proxmox

Overview

This guide walks through creating a k3s Kubernetes cluster using Rocky Linux cloud images in Proxmox with cloud-init for automated VM provisioning.

Cluster Configuration:

  • Rocky1 (192.168.2.41) - Control Plane
  • Rocky2 (192.168.2.42) - Worker Node
  • Rocky3 (192.168.2.43) - Worker Node

Prerequisites

  • Proxmox VE host
  • Rocky Linux cloud image (downloaded from cloud-images.rockylinux.org)
  • Network with static IP availability
  • DNS servers configured (192.168.2.7 and 1.1.1.1 in this example)

Step 1: Create Cloud-Init Snippets

Do this FIRST - these snippets are created once on your Proxmox host and can be reused for all future cloud-init deployments.

SSH Password Authentication Snippet

On your Proxmox host, create a snippet to enable SSH password authentication:

# Create snippets directory if it doesn't exist
mkdir -p /var/lib/vz/snippets

# Create cloud-init snippet for SSH password auth
cat <<'EOF' > /var/lib/vz/snippets/enable-ssh-password.yml
#cloud-config
ssh_pwauth: true
chpasswd:
  expire: false

runcmd:
  - sed -i 's/^PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
  - sed -i 's/^#PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
  - echo "PasswordAuthentication yes" > /etc/ssh/sshd_config.d/50-cloud-init.conf
  - systemctl restart sshd
EOF

This snippet only needs to be created once. It will be applied to VMs during the cloning process.

Step 2: Download Rocky Linux Cloud Image

On your Proxmox host:

# SSH to Proxmox host
ssh root@proxmox-host

# Download Rocky Linux 10 cloud image (adjust version as needed)
cd /var/lib/vz/template/iso
wget https://download.rockylinux.org/pub/rocky/10/images/x86_64/Rocky-10-GenericCloud-Base.latest.x86_64.qcow2

# Rename for clarity
mv Rocky-10-GenericCloud-Base.latest.x86_64.qcow2 rocky-10-cloud.qcow2

Step 3: Create Base Cloud Image Template VM

Follow the steps in Cloud-Init VM Template Creation

Step 4: Boot Template and Install k3s Prerequisites

Now we can boot the template VM once to install prerequisites before converting to template.

Start the Template VM

# Start the VM
qm start 900

Wait for cloud-init to complete (watch the console or wait 1-2 minutes), then SSH into the VM.

Check the console or your DHCP server to find the assigned IP address, then:

# SSH into the template VM (replace IP with actual DHCP-assigned IP)
ssh jeff@<TEMPLATE_IP>

Install Prerequisites

Once SSH'd into the template VM:

# Install kernel modules extra package (required for k3s)
sudo dnf install -y kernel-modules-extra

# Reboot to load new kernel modules
sudo reboot

After reboot, SSH back in and continue:

# Configure SELinux to permissive
sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

# Load required kernel modules
cat <<EOF | sudo tee /etc/modules-load.d/k3s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# Verify modules loaded
lsmod | grep br_netfilter
lsmod | grep overlay

# Configure sysctl parameters for k3s
cat <<EOF | sudo tee /etc/sysctl.d/k3s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# Apply sysctl settings
sudo sysctl --system

# Verify sysctl settings (all should return = 1)
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Configure NetworkManager DNS Management

Prevent NetworkManager from overriding DNS settings:

# Disable NetworkManager DNS management
sudo bash -c 'cat >> /etc/NetworkManager/NetworkManager.conf <<EOF

[main]
dns=none
EOF'

# Restart NetworkManager
sudo systemctl restart NetworkManager

Step 5: Clean and Convert to Template

Clean Machine-Specific Data

Still in the template VM (900), clean up before converting to template:

# Stop cloud-init
sudo systemctl stop cloud-init

# Clean cloud-init state (will regenerate on clones)
sudo cloud-init clean --logs --seed

# Remove machine-id (will regenerate on first boot of clones)
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id

# Remove SSH host keys (will regenerate on first boot of clones)
sudo rm -f /etc/ssh/ssh_host_*

# Clean shell history
history -c
cat /dev/null > ~/.bash_history

# Clean logs
sudo truncate -s 0 /var/log/wtmp
sudo truncate -s 0 /var/log/lastlog

# Shutdown the VM
sudo shutdown -h now

Convert to Template

On your Proxmox host:

# Convert VM to template
qm template 900

Or via Proxmox GUI: Right-click VM 900 → Convert to template

Important: Once converted to a template, you cannot modify it. If you need to make changes, you must convert it back to a VM, make changes, clean, and re-convert to template.

Step 6: Clone Template for Cluster Nodes

Clone via Proxmox GUI

For each node (Rocky1, Rocky2, Rocky3):

  1. Right-click template 900 → Clone
  2. Target Node: (select your Proxmox node)
  3. VM ID: 241 (for Rocky1), 242 (for Rocky2), 243 (for Rocky3)
  4. Name: Rocky1, Rocky2, Rocky3
  5. Mode: Full Clone (recommended)
  6. Click Clone

Repeat this process three times to create Rocky1, Rocky2, and Rocky3.

Configure Cloud-Init for Each Cloned VM

Important: Do this BEFORE starting the VMs.

For each cloned VM, configure cloud-init via Proxmox GUI:

Rocky1 (VM 241):

  1. Select VM 241 → Cloud-Init tab
  2. Configure:
  3. User: jeff
  4. Password: (your password)
  5. DNS domain: local
  6. DNS servers: 192.168.2.7 1.1.1.1
  7. IP Config (net0):
    • IPv4: Static
    • IPv4/CIDR: 192.168.2.41/24
    • Gateway: 192.168.2.1
  8. Click Regenerate Image

Rocky2 (VM 242):

  • Same configuration as Rocky1
  • IPv4/CIDR: 192.168.2.42/24
  • Click Regenerate Image

Rocky3 (VM 243):

  • Same configuration as Rocky1
  • IPv4/CIDR: 192.168.2.43/24
  • Click Regenerate Image

Apply Cloud-Init Snippet to Clones (If Not Already Inherited)

The clones should inherit the cloud-init snippet from the template, but verify or re-apply:

# On Proxmox host, verify or apply snippet to clones
qm set 241 --cicustom "user=local:snippets/enable-ssh-password.yml"
qm set 242 --cicustom "user=local:snippets/enable-ssh-password.yml"
qm set 243 --cicustom "user=local:snippets/enable-ssh-password.yml"

# Regenerate cloud-init images
qm cloudinit update 241
qm cloudinit update 242
qm cloudinit update 243

Step 7: Start and Verify VMs

Start the VMs

# Start all three VMs (on Proxmox host)
qm start 241
qm start 242
qm start 243

Or via GUI: Right-click each VM → Start

Wait for cloud-init to complete on first boot (1-2 minutes).

Verify VMs

SSH into each VM and verify configuration:

# SSH to Rocky1
ssh jeff@192.168.2.41

# Verify hostname
hostnamectl
# Should show: Static hostname: Rocky1

# Verify IP configuration
ip addr show | grep "inet 192.168.2"
# Should show: inet 192.168.2.41/24

# Verify DNS
cat /etc/resolv.conf
# Should show nameservers 192.168.2.7 and 1.1.1.1

# Test DNS resolution
ping -c 3 google.com

# Verify cloud-init completed successfully
cloud-init status
# Should show: status: done

# Verify k3s prerequisites - kernel modules
lsmod | grep br_netfilter
lsmod | grep overlay
# Both should return results

# Verify k3s prerequisites - sysctl settings
sysctl net.ipv4.ip_forward net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables
# All three should return = 1

# Verify SELinux is permissive
getenforce
# Should show: Permissive

Repeat verification for Rocky2 (192.168.2.42) and Rocky3 (192.168.2.43).

Troubleshooting

DNS Issues

If DNS is not resolving properly (showing Tailscale DNS or incorrect nameservers in /etc/resolv.conf):

# Check current DNS configuration
cat /etc/resolv.conf

# Verify cloud-init DNS settings were applied
cloud-init query ds

# If incorrect, manually configure NetworkManager to not manage DNS
sudo bash -c 'cat >> /etc/NetworkManager/NetworkManager.conf <<EOF

[main]
dns=none
EOF'

# Restart NetworkManager
sudo systemctl restart NetworkManager

# Manually set DNS
sudo bash -c 'cat > /etc/resolv.conf <<EOF
nameserver 192.168.2.7
nameserver 1.1.1.1
EOF'

# Test DNS
ping -c 3 google.com

Root Cause: NetworkManager may override cloud-init DNS settings. The NetworkManager DNS fix should already be in the template, but if you're still experiencing issues, apply the manual fix above.

SSH Password Authentication Not Working

If SSH password authentication is disabled after cloning:

Verify cloud-init snippet was applied:

# On Proxmox host, check if snippet is configured
qm config 241 | grep cicustom

# Should show: cicustom: user=local:snippets/enable-ssh-password.yml

If snippet is not applied:

# Apply snippet to VM
qm set 241 --cicustom "user=local:snippets/enable-ssh-password.yml"

# Regenerate cloud-init image
qm cloudinit update 241

# Reboot VM
qm reboot 241

Temporary fix via console:

  1. Access VM via Proxmox console
  2. Login with username/password (console login always works)
  3. Run:
echo "PasswordAuthentication yes" | sudo tee /etc/ssh/sshd_config.d/50-cloud-init.conf
sudo systemctl restart sshd

Kernel Modules Not Loading

If br_netfilter or overlay modules fail to load:

# Verify kernel-modules-extra is installed
rpm -qa | grep kernel-modules-extra

# If not installed
sudo dnf install -y kernel-modules-extra

# Reboot
sudo reboot

# After reboot, try loading modules again
sudo modprobe overlay
sudo modprobe br_netfilter

# Verify
lsmod | grep overlay
lsmod | grep br_netfilter

Cloud-Init Not Running or Incomplete

If cloud-init didn't complete or apply settings:

# Check cloud-init status
cloud-init status

# Check cloud-init logs for errors
sudo cat /var/log/cloud-init.log
sudo cat /var/log/cloud-init-output.log

# Manually re-run cloud-init (only if safe to do so)
sudo cloud-init clean
sudo cloud-init init

Hostname Not Set Correctly

If hostname shows "localhost" or generic name instead of Rocky1/Rocky2/Rocky3:

Via Proxmox GUI:

  1. Verify Cloud-Init tab shows correct hostname for IP configuration
  2. Click Regenerate Image
  3. Reboot VM

Via CLI:

# On Proxmox host, check current cloud-init config
qm cloudinit dump 241 user

# Manually set hostname if needed
sudo hostnamectl set-hostname Rocky1

Summary

You now have:

  • ✅ Reusable Rocky Linux cloud image template (VM 900) with k3s prerequisites
  • ✅ Cloud-init snippets for automated SSH configuration (created once, reusable)
  • ✅ Three Rocky Linux VMs ready for k3s installation:
  • Rocky1: 192.168.2.41 (control plane)
  • Rocky2: 192.168.2.42 (worker)
  • Rocky3: 192.168.2.43 (worker)
  • ✅ All k3s prerequisites installed and configured
  • ✅ Proper DNS and networking configuration
  • ✅ SSH password authentication enabled

Next Steps

With all three VMs configured and verified, you're ready to install k3s:

  1. Install k3s on Rocky1 (control plane node)
  2. Join Rocky2 and Rocky3 as worker nodes
  3. Verify cluster functionality
  4. Deploy workloads to your cluster

Reusing This Template

For future k3s clusters:

  1. Cloud-init snippet already exists - no need to recreate
  2. Template is ready - just clone VM 900
  3. Configure cloud-init for each clone with unique IP/hostname
  4. Start VMs - everything auto-configures on first boot
  5. Install k3s - skip all prerequisite steps

The template approach saves significant time on future deployments!

Additional Resources

  • Rocky Linux Cloud Images: https://rockylinux.org/cloud-images/
  • Proxmox Cloud-Init Documentation: https://pve.proxmox.com/wiki/Cloud-Init_Support
  • k3s Documentation: https://docs.k3s.io/
  • k3s System Requirements: https://docs.k3s.io/installation/requirements