General MR

Implementation of a generic UserVMs Ansible role for automated VM creation in Proxmox via Crossplane integration. This enables the BigBang Edge Controller to provision user-defined VMs directly through Kubernetes resources, with all configuration driven by EdgeInstall CRD specifications.

Changes Made

New Files Created

containers/bigbangedge/config/rbac/role.yaml - Added RBAC permissions for Crossplane Proxmox resources
containers/bigbangedge/roles/proxmox/tasks/create_provider_config.yml - Proxmox ProviderConfig and credentials creation
containers/bigbangedge/roles/uservms/tasks/main.yml - UserVMs role entry point
containers/bigbangedge/roles/uservms/tasks/deploy_vms.yml - Core VM provisioning logic
containers/bigbangedge/roles/uservms/defaults/main.yml - Default configuration variables (fallback values)

Modified Files

containers/bigbangedge/playbooks/proxmox/proxmox_pb.yml - Added ProviderConfig creation task
containers/bigbangedge/roles/rke2/tasks/deploy_cluster.yml - Removed duplicate Proxmox ProviderConfig/secret creation

Key Features Implemented

VM Management

Multi-VM Support: Creates user-defined VMs with configurable count
Generic Design: Role can provision any type of VMs based on EdgeInstall specifications
Template-Based Cloning: Supports cloning from VM templates
Flexible Targeting: Distributes VMs across specified Proxmox nodes with round-robin assignment
Lifecycle Management: Proper startup ordering and boot configuration
PXE Boot Support: VMs configured with boot: "order=net0;scsi0" for network-based OS installation

Storage Configuration

Multi-Disk Support: Configurable number of SCSI disks per VM (default: 3 disks - 20GB boot, 10GB data, 10GB data)
Ceph Integration: Uses existing vm-storage pool for distributed storage
Virtio-SCSI: Optimized disk controller for performance

Network Architecture

Multi-Bridge Support: Primary (vmbr0), orchestrator (vmbr1), management networks
VLAN Segmentation: Configurable VLAN tags (default: workload 112, default 111, management 110)
MAC Address Management: Deterministic MAC assignment from EdgeInstall role_vars
3 NICs per VM: Separate interfaces for workload, admin, and management traffic

EdgeInstall Integration

All configuration from EdgeInstall role_vars: VM specs, disk config, network bridges, VLAN tags, MAC addresses
Fallback defaults: defaults/main.yml provides sensible defaults when EdgeInstall doesn't specify values
Variable precedence: EdgeInstall spec.role_vars.uservms overrides defaults via Ansible variable precedence

Configuration Structure

The role expects configuration via EdgeInstall CRD spec.role_vars.uservms. All VM configuration data (VM specs, disk config, network bridges, VLAN tags, MAC addresses) must be provided through role_vars:

apiVersion: edgecompute.platformone.io/v1alpha1
kind: EdgeInstall
metadata:
  name: my-deployment
spec:
  role_vars:
    uservms:
      vm_specs:
        ram: 16384
        cores: 8
        sockets: 1
        agent: true
        qemu_os: "l26"
        scsihw: "virtio-scsi-pci"
        onboot: true

      disk_config:
        - name: "scsi0"
          size: "20"
          storage: "vm-storage"
        - name: "scsi1"
          size: "10"
          storage: "vm-storage"
        - name: "scsi2"
          size: "10"
          storage: "vm-storage"

      network_bridges:
        primary: "vmbr0"
        orchestrator: "vmbr1"
        default_equivalent: "vmbr0"

      vlan_tags:
        workload: 112
        default: 111
        management: 110

      user_vms:
        count: 6
        prefix: "uservm"
        vmid_start: 2001
        startup_order: "order=4"
        target_nodes: ["mitr01dhv01", "mitr01dhv02", "mitr01dhv03", "mitr01dhv04"]
        mac_addresses:
          primary: ["02:00:00:00:00:01", "02:00:00:00:00:02", ...]
          admin: ["02:00:00:00:01:01", "02:00:00:00:01:02", ...]
          mgmt: ["02:00:00:00:02:01", "02:00:00:00:02:02", ...]

Design Decisions

ProviderConfig Separation

Moved Proxmox ProviderConfig creation from RKE2 role to Proxmox role
Proper separation of concerns (RKE2 manages Kubernetes, Proxmox manages hypervisor)
Fixed JSON credentials formatting using to_json filter to ensure proper double quotes

PXE Boot Configuration

Boot order: order=net0;scsi0 (network first, disk second)
Enables automated OS installation via network
Aligns with existing infrastructure VM patterns

Generic VM Design

No application-specific logic (previously contained HPCM-specific scheduler/compute split)
Single user_vms configuration block for all VMs
Round-robin node assignment algorithm for load distribution

Integration Points

Crossplane Integration

Uses qemu.proxmox.crossplane.io/v1alpha1 API
Requires valid ProviderConfig named "default"
Creates Qemu resources that Crossplane reconciles to Proxmox VMs

Storage Integration

Integrates with existing Ceph distributed storage
Uses vm-storage pool from pve_ceph_pools
No new storage configuration required

Network Integration

Uses existing Proxmox bridge configuration
VLAN tagging for network segmentation
MAC address determinism for predictable networking

Testing

Testing via Crossplane Qemu Resources

The UserVMs role creates Crossplane Qemu resources automatically when invoked via EdgeInstall. For direct testing, you can create a Qemu resource manually:

apiVersion: qemu.proxmox.crossplane.io/v1alpha1
kind: Qemu
metadata:
  name: test-uservm-01
spec:
  forProvider:
    vmid: 4000
    agent: true
    bootdisk: "scsi0"
    boot: "order=net0;scsi0"
    qemuOs: "l26"
    cores: 8
    memory: 16384
    name: "test-uservm-01"
    scsihw: "virtio-scsi-pci"
    sockets: 1
    targetNode: "mitr01dhv01"
    onboot: true
    startup: "order=4"
    disks:
      - scsi:
        - scsi0:
          - disk:
            - size: "20"
              storage: "vm-storage"
          scsi1:
          - disk:
            - size: "10"
              storage: "vm-storage"
          scsi2:
          - disk:
            - size: "10"
              storage: "vm-storage"
    network:
    - bridge: "vmbr0"
      model: "virtio"
      tag: 112
      id: 0
      macaddr: "02:00:00:00:00:01"
    - bridge: "vmbr1"
      model: "virtio"
      tag: 111
      id: 1
      macaddr: "02:00:00:00:01:01"
    - bridge: "vmbr0"
      model: "virtio"
      tag: 110
      id: 2
      macaddr: "02:00:00:00:02:01"

Apply with: kubectl apply -f test-uservm.yaml

This creates a VM directly via Crossplane, bypassing the Ansible operator. The UserVMs role generates these Qemu resources automatically when triggered by an EdgeInstall CRD

Relevant logs/screenshots

Linked Issue

issue

Upgrade Notices

(Include any relevant notes about upgrades here or write "N/A" if there are none)

Edited Oct 02, 2025 by John Patterson

Admin message

LEVIATHAN-255: Generic UserVMs Ansible Role with Proxmox Integration