Skip to main content

Azure Kubernetes Service–Volume node affinity conflict

When trying to deploy a pod on our AKS cluster, it hanged in the pending state. I looked at the logs and noticed the following warning:

FailedScheduling – 1 node(s) had volume node affinity conflict

The pod I tried to deploy had a persistent volume claim and I was certain that the persistent volume was succesfully deployed and available.

What was going wrong?

It turned out that my AKS cluster was deployed in 3 availability zones but I had only 2 nodes running:

AKS cluster is gedeployed in 3 zones maar er zijn maar 2 nodes:

$ kubectl describe nodes | grep -e "Name:" -e "failure-domain.beta.kubernetes.io/zone"

Name:               aks-agentpool-38609413-vmss000003

                    failure-domain.beta.kubernetes.io/zone=westeurope-1

Name:               aks-agentpool-38609413-vmss000004

                    failure-domain.beta.kubernetes.io/zone=westeurope-2

Here we can see that our nodes are living in zones westeurope-1 and westeurope-2.

If we now take a look at our persistent volume, we can see that it is deployed in zone westeurope-3:

$  kubectl describe pv pvc-ecc

                   failure-domain.beta.kubernetes.io/zone=westeurope-3

Annotations:       pv.kubernetes.io/bound-by-controller: yes

                   pv.kubernetes.io/provisioned-by: kubernetes.io/azure-disk

                   volumehelper.VolumeDynamicallyCreatedByKey: azure-disk-dynamic-provisioner

Finalizers:        [kubernetes.io/pv-protection]

StorageClass:      default

Status:            Bound

Claim:             appservice-ns/appservice-ext-k8se-build-service

Reclaim Policy:    Delete

Access Modes:      RWO

VolumeMode:        Filesystem

Capacity:          100Gi

Node Affinity:

  Required Terms:

    Term 0:        failure-domain.beta.kubernetes.io/region in [westeurope]

                   failure-domain.beta.kubernetes.io/zone in [westeurope-3]

Message:

That explains why the deployment failed as the pod cannot connect to the volume on another zone.

As a quick solution I introduced a 3th node in the cluster so that at least one node can connect to the persistent volume.

Labels:

Popular posts from this blog

Podman– Command execution failed with exit code 125

After updating WSL on one of the developer machines, Podman failed to work. When we took a look through Podman Desktop, we noticed that Podman had stopped running and returned the following error message: Error: Command execution failed with exit code 125 Here are the steps we tried to fix the issue: We started by running podman info to get some extra details on what could be wrong: >podman info OS: windows/amd64 provider: wsl version: 5.3.1 Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:2655: connectex: No connection could be made because the target machine actively refused it. That makes sense as the podman VM was not running. Let’s check the VM: >podman machine list NAME         ...
Read more

Azure DevOps/ GitHub emoji

I’m really bad at remembering emoji’s. So here is cheat sheet with all emoji’s that can be used in tools that support the github emoji markdown markup: All credits go to rcaviers who created this list.
Read more

Cleaner switch expressions with pattern matching in C#

Ever find yourself mapping multiple string values to the same result? Being a C# developer for a long time, I sometimes forget that the C# has evolved so I still dare to chain case labels or reach for a dictionary. Of course with pattern matching this is no longer necessary. With pattern matching, you can express things inline, declaratively, and with zero repetition. A small example I was working on a small script that should invoke different actions depending on the environment. As our developers were using different variations for the same environment e.g.  "tst" alongside "test" , "prd" alongside "prod" .  We asked to streamline this a long time ago, but as these things happen, we still see variations in the wild. This brought me to the following code that is a perfect example for pattern matching: The or keyword here is a logical pattern combinator , not a boolean operator. It matches if either of the specified pattern...
Read more