Data Importer tolerations and nodeSelector
We had to use metal nodes to utilize KVM(non-emulated) kubevirt nodes as a separate kubernetes worker node class. These metal nodes are labeled and tainted to only allow kubevirt nodes. The ebs-csi storage class driver usually has multiple AZs defined. This caused a problem when the containerized-data-importer is on a non-metal node AZ when pulling down the image and converting to a containerDisk. When a KubeVirt VirtualMachine is spun up and pointing to the dataVolume on an AZ not represented by a metal node, scheduling fails. CDI doesn't seem to support nodeSelectors and tolerations(from what we could gather from the docs) for the importer, so we had to hack around it using mutations.
---
apiVersion: mutations.gatekeeper.sh/v1beta1
kind: Assign
metadata:
name: data-importer-tolerations
spec:
applyTo:
- groups: [""]
kinds: ["Pod"]
versions: ["v1"]
match:
scope: Namespaced
kinds:
- apiGroups: ["*"]
kinds: ["Pod"]
labelSelector:
matchLabels:
app: containerized-data-importer
location: "spec.tolerations"
parameters:
assign:
value:
- key: "metal"
operator: "Equal"
value: "True"
effect: "NoSchedule"
---
apiVersion: mutations.gatekeeper.sh/v1beta1
kind: Assign
metadata:
name: data-importer-nodeselector
spec:
applyTo:
- groups: [""]
kinds: ["Pod"]
versions: ["v1"]
match:
scope: Namespaced
kinds:
- apiGroups: ["*"]
kinds: ["Pod"]
labelSelector:
matchLabels:
app: containerized-data-importer
location: "spec.nodeSelector"
parameters:
assign:
value:
node.kubernetes.io/pool: "metal"
It would be nice if upstream CDI supported setting node selectors and tolerations on the importer and other pods, then we could just add in the configs from the helm chart.