The art of simplicity

Reducing the carbon footprint and CO2 emission of our (cloud) workloads, is a responsibility of all of us. If you are running a Kubernetes cluster, have a look at Kube-Green . kube-green is a simple Kubernetes operator that automatically shuts down (some of) your pods when you don't need them. A single pod produces about 11 Kg CO2eq per year( here the calculation). Reason enough to give it a try! Installing kube-green in your cluster The easiest way to install the operator in your cluster is through kubectl. We first need to install a cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.5/cert-manager.yaml Remark: Wait a minute before you continue as it can take some time before the cert-manager is up & running inside your cluster. Now we can install the kube-green operator: kubectl apply -f https://github.com/kube-green/kube-green/releases/latest/download/kube-green.yaml Now in the namespace where we want t...

Running a local Kubernetes cluster is easy thanks to Podman Desktop and Kind . In this post I show you how to setup a kind environment from Podman Desktop. But first an introduction… What is Podman Desktop? Podman Desktop is an open source graphical tool enabling you to seamlessly work with containers and Kubernetes from your local environment. It offers similar functionality as Docker Desktop. Installing Podman Desktop(on Windows) is easy: Download the Windows installer . Start the Podman Desktop installer, open the downloaded file. Podman Desktop uses WSL2 to run a a Linux distribution in a virtual machine. It can be that a restart is required as the installer will try to enable/install WSL2 during the process. More information: Windows | Podman Desktop (podman-desktop.io) What is Kind? Kind is an open source project that allows to run Kubernetes clusters in a container engine (could be Docker or Podman or others). This is usually quite he...

In a previous post I showed how you can use ConfigMaps in Kubernetes . This allows you to update for example our appsettings.json without the need to upload a new container image and update our pods. Sounds great! However there is one challenge with this approach. In Kubernetes a config map is mounted as a symlink, and .NET is not able to pick up changes applied to the original file by just knowing about the symlink. I mentioned some workarounds but an out-of-the-box solution was not available at that time. DOTNET_USE_POLLING_FILE_WATCHER Now a solution to the problem above is available through the DOTNET_USE_POLLING_FILE_WATCHER environment variable. By setting this value to “true” or “1”, we can instruct .NET to not use the default FileSystemWatcher but using a polling mechanism instead. More information: dotnet watch

When trying to deploy a pod on our AKS cluster, it hanged in the pending state. I looked at the logs and noticed the following warning: FailedScheduling – 1 node(s) had volume node affinity conflict The pod I tried to deploy had a persistent volume claim and I was certain that the persistent volume was succesfully deployed and available. What was going wrong? It turned out that my AKS cluster was deployed in 3 availability zones but I had only 2 nodes running: AKS cluster is gedeployed in 3 zones maar er zijn maar 2 nodes: $ kubectl describe nodes | grep -e "Name:" -e "failure-domain.beta.kubernetes.io/zone" Name:               aks-agentpool-38609413-vmss000003                     failure-domain.beta.kubernetes.io/zone= westeurope-1 Name:      ...

ASP.NET Core offers Health Checks Middleware for reporting the health of your application. Health checks are exposed as HTTP endpoints and can be probed by container orchestrators and load balancers to check an app's status. For Kubernetes I typically create 2 type of health checks: Readiness indicates if the app is running normally but isn't ready to receive requests. Liveness indicates if an app has crashed and must be restarted. This is a general guideline we use for all containerized applications that we have running in Kubernetes. Last week I was reviewing one of those applications and I noticed the following code: This code is perfectly fine, but I couldn’t remember exactly what the role of the Predicate was in the example above. Time to dig in somewhat deeper… By default, the Health Checks Middleware runs all registered health checks. To run a subset of health checks, you can provide a function that returns a boolean to the Predicate option. So...

For our database migrations we are usingKubernetes Jobs and init containers as discussed here . However when we tried to deploy the job container, it failed with the following error: Error from server (Forbidden): jobs.batch "example-migration" is forbidden: User "system:serviceaccount:example-ns:default" cannot get resource "jobs" in API group "batch" in the namespace "example-ns": Azure does not have opinion for this user. To read and list jobs, the deployment is using the default service account in the “example-ns” namespace. This default service account does not have the necessary api rights in the kubernetes cluster. To fix it we created a new service account, role and role binding: After doing that, we had to update our deployment to use this service account:

In the previous post in this series I talked about why Azure Arc is also interesting for developers. Today we finally move on to the more practical part and try to get it up and running on our local machine. Let’s first focus on what you need to have up and running on your local machine first: Make sure your kubeconfig file is configured properly and you are working against your k8s cluster context . Install or update Azure CLI to version 2.25.0 and above . Install and Set Up kubectl Install Helm 3 . If you are on a Windows environment, a recommended and easy way is to use the Helm 3 Chocolatey package . As we want to run Azure Arc on our local machine, we also need to have a local AKS cluster up and running. You can use Minikube , MicroK8S , KIND (Kubernetes in Docker), or any other flavor you like that can be installed locally. I tested both in MiniKube and KIND. Now we can move on to the Azure side. Let’s see what we need the...

Before I dive into the details on how to get Azure Arc up and running on your laptop, it would be a good idea to start with a short introduction. Therefore we first have to dive in how Azure works. The hearth of the Azure ecosystem is the Azure control plane. This control plane manages all the resources you can find in Azure. It helps you to inventorize, organize and govern all resources and multiple tools exist that can help you to interact with it (think ARM templates , Bicep , Terraform , Pulumi , …) You probably know this control plane better as the Azure Resource manager. It controls and manage all the Azure resources which can be as big as a Kubernetes cluster and as small as a static ip address. These resources run inside an Azure region, one of the datacenters that Microsoft has all around the world. So where does Azure Arc fits into this picture? If we bring Azure Arc into the picture, we can bring resources that are not running on Azure to the Azure control plane ...

As mentioned yesterday I promised to write a series of follow up posts about my ‘Running Azure on your laptop’ session. I’ll use this post as a placeholder to point to the different parts. Microsoft is more and more embracing a hybrid cloud approach. As part of this evolution, an increasing amount of ‘Azure only’ services become available outside Azure. This idea is not new, people who work long enough in the Microsoft ecosystem maybe remember Azure Pack ,  which was a way to install Azure software on your own hardware.It gave you the Azure portal and some of it’s services. I never tried it myself and I don’t know any customer who used it in the wild. A couple of years later, Microsoft announced the Azure Pack’s successor, Azure Stack. This was a hardware appliance, that you could install in your own datacenter. Over time, the name evolved to Azure Stack Portfolio as multiple flavors of Azure Stack became available. Azure Stack is still available today and keeps evolving. At...

Last weekend I gave a presentation at VisugXL about Azure Arc . I’ll write a few follow-up posts explaining the steps I took to get it all up and running(and where I got into trouble). If you can’t wait until then, here is already the presentation:

One of the ways you can save some money using Azure is by using spot node pools for your Azure Kubernetes Service cluster. What’s a spot node pool? Using a spot node pool allows you to take advantage of unused Azure capacity at a significant cost savings. At any point in time when Azure needs the capacity back, the Azure infrastructure will evict spot nodes. Therefore, Spot nodes are great for workloads that can handle interruptions like batch processing jobs, dev/test environments, large compute workloads, and more. Remark: A spot node pool can't be the cluster's default node pool. A spot node pool can only be used for a secondary pool. Pricing for a spot node pool Pricing for spot instances is variable , based on region and SKU. For more information, see pricing for Linux and Windows . You do have the option to set a max price. In case the price is exceeded the spot node is evicted from your cluster. Schedule a deployment to use the spot node pool A spot node ...

What is Dapr ? Dapr helps developers build event-driven, resilient distributed applications. Whether on-premises, in the cloud, or on an edge device, Dapr helps you tackle the challenges that come with building microservices and keeps your code platform agnostic. The Dapr ecosystem keeps growing and now they’ll have their first virtual conference ‘DaprCon’ next week. DaprCon will include a variety of content including a keynote, technical sessions, panel discussions and real-world experiences of adopters building with Dapr DaprCon is a fully virtual event that will be streamed on YouTube and attendance is free! To watch the live events just follow these two links: DaprCon Day 1 DaprCon Day 2 More information: https://blog.dapr.io/posts/2021/10/05/join-us-for-daprcon-october-19th-20th-2021/

Quick tip if you are hosting your ASP.NET Core application in Kubernetes: have a look at the Microsoft Application Insights for Kubernetes nuget package . Through this package you can enrich your Application Insights telemetry data with Kubernetes specific information: After adding the nuget package you can register the enricher by adding the following code: More information: https://github.com/microsoft/ApplicationInsights-Kubernetes

In case your deployments on Kubernetes fails, the following diagram can help: (It is created by the people from learnk8s who provide Kubernetes training) A PDF version of this diagram can be found here: https://learnk8s.io/a/a-visual-guide-on-troubleshooting-kubernetes-deployments/troubleshooting-kubernetes.v2.pdf

Small update about the post from yesterday, if your Kubernetes cluster is not exposed publicly, you can also choose to just disable the security check by adding the ‘--no-auth’ argument. Here is the updated yaml file:

Last week I blogged about how you can run dotnet monitor as a sidecar in your Kubernetes cluster . Although the yaml file I shared worked on my local cluster (inside Minikube), it didn’t work when I tried to deploy it to AKS. Nothing happened when I tried to connect to the specified URL’s. To fix this I had to take multiple steps: First I had to explicitly set the ‘—urls’ argument inside the manifest: Now I was able to connect to the url but it still failed. When I took a look at the logs I noticed the following message: {"Timestamp":"2021-07-27T18:48:29.6522095Z","EventId":7,"LogLevel":"Information","Category":"Microsoft.Diagnostics.Tools.Monitor.ApiKeyAuthenticationHandler","Message":"MonitorApiKey was not authenticated. Failure message: API key authentication not configured.","State":{"Message":"MonitorApiKey was not authenticated. Failure message: A...

Last week, we got into problems when booting up our AKS cluster(we’ll shut the development cluster down every night to safe costs). Instead of green lights, our Neo4J database refused to run. In the logs, we noticed the following error message: ERROR Invalid memory configuration - exceeds physical memory. Let me share what caused this error. Maybe you’ve read my article about resource limits in Kubernetes . There I talked about the fact that you can set resource limits at the container level. What I didn’t mention in the article is that you can also configure default limits at the namespace level through limit ranges . From the documentation : A LimitRange provides constraints that can: Enforce minimum and maximum compute resources usage per Pod or Container in a namespace. Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace. Enforce a ratio between request and limit for a resource in a namespace. Set defaul...

When invoking ‘kubectl’, it failed with the following error message: PS /home/bart> kubectl apply -f ./example.yaml E0720 07:58:14.668222     182 azure.go:154] Failed to acquire a token: unexpected error when refreshing token: refreshing token: adal: Refresh request failed. Status Code = '400'. Response body: {"error":"invalid_grant","error_description":"AADSTS700082: The refresh token has expired due to inactivity. The token was issued on 2021-03-31T13:22:18.9100852Z and was inactive for 90.00:00:00.\r\nTrace ID: 68f8e37d-4d18-4e7d-a3e6-b11291831a02\r\nCorrelation ID: 65ee9420-d6f9-4a7c-8214-a82756c7ecc8\r\nTimestamp: 2021-07-20 07:58:14Z","error_codes":[700082],"timestamp":"2021-07-20 07:58:14Z","trace_id":"68f8e37d-4d18-4e7d-a3e6-b11291831a02","correlation_id":"65ee9420-d6f9-4a7c-8214-a82756c7ecc8","error_uri":" https://login.mic...

In Kubernetes it is a best practice to configure resource limits for your containers. VSCode will even warn you if it couldn’t detect resource limits in your manifest files: Setting resource limits prevents a container consuming too much resources and impacting other workloads. By setting limits, pods will be terminated by Kubernetes when their limits are exceeded. This helps in keeping the cluster healthy and stable. The most common resources to specify are CPU and memory, but others exists. Here is a short example on how to configure this at the container level: In the example above, the CPU usage is limited to 250m or  250 milliCPU (1/4th of a vCPU/Core) and memory usage is limited to 512Mi or 512MiB. Next to resource limits, it is also possible to configure resource requests . Setting a resource request indicates the amount of that resource that you expect the container will use. Kubernetes will use this information when determining which node to schedule the pod on...

Yesterday I blogged about ‘dotnet monitor’ and how it can help you to collect diagnostic artifacts at runtime in a uniform way. Let’s have a look today on how to use ‘dotnet monitor’ inside a Kubernetes cluster. When running in a cluster, it is recommend to run the dotnet-monitor container as a sidecar alongside your application container in the same pod. Here is an example manifest on how to set this up: Most important to notice in the manifest is that you need to share a volume between the application container and the sidecar. Let’s deploy this manifest: $ kubectl apply –f ./dotnetmonitor.yaml Once your pods are up and running, we need to use port forwarding to be able to access the diagnostics endpoint from our local machine. To do this, we first need to find the name of the pod : $ kubectl get pod -l app=dotnet-monitor-example NAME READY STATUS RESTARTS AGE dotnet-monitor-example-78997f8fdf-nrhp7 2/2 Running 0 ...