Friday, March 26, 2021

Kubernetes–Working with multiple namespaces

Kubernetes has the concept of namespaces to logically seperate different parts of your cluster.

To simplify working with different namespaces, it is recommend to create a separate context:

kubectl config set-context <contexname> --namespace=<namespace> --cluster=<clustername> --user=<clusteruser>

After a new context is created, you can switch to this new context using:

kubectl config use-context <contextname>

All commands you execute afterwards will apply to this context.

You can verify your current context by doing the following:

kubectl config current-context

To view all the contexts:

kubectl config view

Thursday, March 25, 2021

Kubernetes - Reload your ASP.NET Core configuration when a configmap changes

Yesterday I blogged about configmaps and how changing the configmap is not picked up by ASP.NET Core. In this post I want to dig deeper in why this is the case and suggest a possible solution.

The CreateDefaultBuilder() method in ASP.NET Core will load your appsettings.json file and monitor it for changes (see the reloadOnChange: true):

Behind the scenes the reloadOnChange setting will monitor the config file for changes based on the last modified date. Seems logical…

Let’s continue to find out why this doesn’t work with Kubernetes. In Kubernetes a config map is mounted as a symlink:

kubectl exec -it <pod-name> -- bash
root@example-deployment-839f6c6546-c783b:/appg# ls -la
rwxrwxrwx 3 root root 4096 Mar 25  10:01 .
drwxr-xr-x 1 root root 4096 Mar 25 10:47 ..
drwxr-xr-x 2 root root 4096 Mar 25 10:01 ..2021_03_25_10_01_16.386067924
lrwxrwxrwx 1 root root 31 Mar 25 10:01 ..data -> ..2021_03_25_10_01_16.386067924
lrwxrwxrwx 1 root root 53 Mar 25 10:47 appsettings.json -> ..data/appsettings.json

Unfortunately when you update the config map, the last modified date doesn’t change, although the file reference itself is updated. This is a known issue as discussed here.

One suggested workaround for this problem is to use a a configuration provider that understand symlinks(notice that this solution only works on linux):

Another possible solution is mentioned by Francisco Beltrao and uses the content itself to detect changes instead of the last modified date:

The code required for this solution can be found in this GitHub repo: https://github.com/fbeltrao/ConfigMapFileProvider

Wednesday, March 24, 2021

Kubernetes–Trigger a new deployment after a configmap changes

We are using configmaps as a way to store configuration data for our pods. At startup these configmaps are mounted as configuration files inside the container.

Unfortunately an update of a configmap will not be picked up by ASP.NET Core.

A way to solve this is by explicitly triggering a new deployment afterwards:

kubectl rollout restart deploy/example-api-deployment

I created another(better?) approach but that is for another blog post…

Tuesday, March 23, 2021

ElasticSearch on AKS–Debug a failed rollout

Installing ElasticSearch in a Kubernetes cluster is easy thanks to the available operator. Unfortunately that doesn’t guarantee a successfull rollout.

Let’s see how we can find out why a deployment failed.

An ElasticSearch cluster is rolled out as a statefulset. So let’s first have a look a the available statefulsets in our AKS cluster:

PS /home/bart> kubectl get statefulset

No resources found in default namespace.

NAME                READY   AGE

search-es-default   0/1     13h

OK, we see that the statefulset is not ready. Let’s continue our investigation and check the related pods:

PS /home/bart> kubectl get pods -l elasticsearch.k8s.elastic.co/statefulset-name=search-es-default

NAME                  READY   STATUS    RESTARTS   AGE

search-es-default-0   0/1     Pending   0          13h

OK, our pod is in a pending state and not ready. Let’s drill down in more detail about this pending pod:

PS /home/bart> kubectl describe pod search-es-default-0

Name:           search-es-default-0

Namespace:      default

Priority:       0

Node:           <none>

Labels:         common.k8s.elastic.co/type=elasticsearch

                controller-revision-hash=search-es-default-67676d5954

                elasticsearch.k8s.elastic.co/cluster-name=search

                elasticsearch.k8s.elastic.co/config-hash=1368074956

                elasticsearch.k8s.elastic.co/http-scheme=https

                elasticsearch.k8s.elastic.co/node-data=true

                elasticsearch.k8s.elastic.co/node-ingest=true

                elasticsearch.k8s.elastic.co/node-master=true

                elasticsearch.k8s.elastic.co/node-ml=true

                elasticsearch.k8s.elastic.co/node-remote_cluster_client=true

                elasticsearch.k8s.elastic.co/node-transform=true

                elasticsearch.k8s.elastic.co/node-voting_only=false

                elasticsearch.k8s.elastic.co/statefulset-name=search-es-default

                elasticsearch.k8s.elastic.co/version=7.11.2

                statefulset.kubernetes.io/pod-name=search-es-default-0

Annotations:    co.elastic.logs/module: elasticsearch

Status:         Pending

IP:

IPs:            <none>

Controlled By:  StatefulSet/search-es-default

Init Containers:

  elastic-internal-init-filesystem:

    Image:      docker.elastic.co/elasticsearch/elasticsearch:7.11.2

    Port:       <none>

    Host Port:  <none>

    Command:

      bash

      -c

      /mnt/elastic-internal/scripts/prepare-fs.sh

    Limits:

      cpu:     100m

      memory:  50Mi

    Requests:

      cpu:     100m

      memory:  50Mi

    Environment:

      POD_IP:                  (v1:status.podIP)

      POD_NAME:               search-es-default-0 (v1:metadata.name)

      NODE_NAME:               (v1:spec.nodeName)

      NAMESPACE:              default (v1:metadata.namespace)

      HEADLESS_SERVICE_NAME:  search-es-default

    Mounts:

      /mnt/elastic-internal/downward-api from downward-api (ro)

      /mnt/elastic-internal/elasticsearch-bin-local from elastic-internal-elasticsearch-bin-local (rw)

      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)

      /mnt/elastic-internal/elasticsearch-config-local from elastic-internal-elasticsearch-config-local (rw)

      /mnt/elastic-internal/elasticsearch-plugins-local from elastic-internal-elasticsearch-plugins-local (rw)

      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)

      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)

      /mnt/elastic-internal/transport-certificates from elastic-internal-transport-certificates (ro)

      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)

      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)

      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)

      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)

      /usr/share/elasticsearch/data from elasticsearch-data (rw)

      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)

Containers:

  elasticsearch:

    Image:       docker.elastic.co/elasticsearch/elasticsearch:7.11.2

    Ports:       9200/TCP, 9300/TCP

    Host Ports:  0/TCP, 0/TCP

    Limits:

      memory:  2Gi

    Requests:

      memory:   2Gi

    Readiness:  exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3

    Environment:

      POD_IP:                     (v1:status.podIP)

      POD_NAME:                  search-es-default-0 (v1:metadata.name)

      NODE_NAME:                  (v1:spec.nodeName)

      NAMESPACE:                 default (v1:metadata.namespace)

      PROBE_PASSWORD_PATH:       /mnt/elastic-internal/probe-user/elastic-internal-probe

      PROBE_USERNAME:            elastic-internal-probe

      READINESS_PROBE_PROTOCOL:  https

      HEADLESS_SERVICE_NAME:     search-es-default

      NSS_SDB_USE_CACHE:         no

    Mounts:

      /mnt/elastic-internal/downward-api from downward-api (ro)

      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)

      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)

      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)

      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)

      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)

      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)

      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)

      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)

      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)

      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)

      /usr/share/elasticsearch/data from elasticsearch-data (rw)

      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)

      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)

Conditions:

  Type           Status

  PodScheduled   False

Volumes:

  elasticsearch-data:

    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName:  elasticsearch-data-search-es-default-0

    ReadOnly:   false

  downward-api:

    Type:  DownwardAPI (a volume populated by information about the pod)

    Items:

      metadata.labels -> labels

  elastic-internal-elasticsearch-bin-local:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

    SizeLimit:  <unset>

  elastic-internal-elasticsearch-config:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  search-es-default-es-config

    Optional:    false

  elastic-internal-elasticsearch-config-local:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

    SizeLimit:  <unset>

  elastic-internal-elasticsearch-plugins-local:

    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

    SizeLimit:  <unset>

  elastic-internal-http-certificates:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  search-es-http-certs-internal

    Optional:    false

  elastic-internal-probe-user:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  search-es-internal-users

    Optional:    false

  elastic-internal-remote-certificate-authorities:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  search-es-remote-ca

    Optional:    false

  elastic-internal-scripts:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      search-es-scripts

    Optional:  false

  elastic-internal-transport-certificates:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  search-es-default-es-transport-certs

    Optional:    false

  elastic-internal-unicast-hosts:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      search-es-unicast-hosts

    Optional:  false

  elastic-internal-xpack-file-realm:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  search-es-xpack-file-realm

    Optional:    false

  elasticsearch-logs:

    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:

    SizeLimit:   <unset>

QoS Class:       Burstable

Node-Selectors:  <none>

Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists

                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

  Type     Reason            Age   From               Message

  ----     ------            ----  ----               -------

  Warning  FailedScheduling  13h   default-scheduler  0/3 nodes are available: 1 Insufficient memory, 2 node(s) had volume node affinity conflict.

Wow! That is a lot of information. Luckily the important info can be found at the end. There are no nodes with sufficient memory available in the node pool. Therefore the Kubernetes scheduler cannot schedule a deployment of the pods.  Higher in the information above you can see that at least 2GB of memory is expected.

Time to extend your node pool…

Monday, March 22, 2021

Await using in C#8

For a long time, the ‘using’ statement in C# introduced some convenience when Disposing classes implementing the IDisposable interface.

The following statement:

is syntactic sugar for:

In C# 8.0 a new System.IAsyncDisposable interface was introduced. This interface enables asynchronous cleanup operations.

To use it you need to use the ‘await using’ statement:

that is syntactic sugar for:

Friday, March 19, 2021

NPM–401 error when restoring packages from an Azure DevOps Artifacts NPM registry

I had to make some changes to an Angular application, so I cloned the app and ran ‘npm install’. This failed with the following error message:

npm WARN tar ENOENT: no such file or directory, open 'D:\Projects\CMS\Bridges.CMS.SPA\node_modules\.staging\dayjs-f8637d82\plugin\weekOfYear.d.ts'

npm WARN tar ENOENT: no such file or directory, open 'D:\Projects\CMS\Bridges.CMS.SPA\node_modules\.staging\dayjs-f8637d82\plugin\weekYear.d.ts'

npm ERR! code E401

npm ERR! Unable to authenticate, need: Bearer authorization_uri=https://login.windows.net/<guid>, Basic realm="https://pkgsprodsu3weu.app.pkgs.visualstudio.com/", TFS-Federated   

Some packages should be downloaded from our own NPM registry hosted in Azure DevOps Artifacts. As I couldn’t authenticate succesfully this resulted in a 401 error.

To solve this I first had to install the vsts-npm-auth package:

PS D:\Projects\cms\ui\CMS\Bridges.CMS.SPA> npm install -g vsts-npm-auth

C:\Users\bart\AppData\Roaming\npm\vsts-npm-auth -> C:\Users\bart\AppData\Roaming\npm\node_modules\vsts-npm-auth\bin\vsts-npm-auth.exe

+ vsts-npm-auth@0.41.0

added 1 package from 1 contributor in 1.947s

Then I could login and create the required .npmrc file by executing the following command:

PS D:\Projects\cms\ui\CMS\Bridges.CMS.SPA> vsts-npm-auth -config .npmrc

vsts-npm-auth v0.41.0.0

-----------------------

Creating npmrc file. Path: C:\Users\bart\.npmrc

Getting new credentials for source:https://pkgs.dev.azure.com/<my-organization>/<my-project>/_packaging/bridges/npm/registry/, scope:vso.packaging_write vso.drop_write

More information: https://docs.microsoft.com/en-us/azure/devops/artifacts/npm/npmrc?view=azure-devops

Thursday, March 18, 2021

ASP.NET Core - ASPNETCORE_FORWARDEDHEADERS_ENABLED environment variable

Although you can directly use Kestrel to act as an “Edge” server, a typical approach is to host it behind a reverse proxy(NGINX, Azure Application Gateway, …). If you are running your ASP.NET Core application inside Kubernetes, this is typically handled through an ingress. This ingress defines the hostname and paths your application should be exposed at. An ingress controller will map this configuration to an implementation. This can be a direct mapping to some infrastructure configuration such as a load balancer(e.g. Azure Application Gateway) or a reverse proxy running inside the cluster.

When your app is running behind a reverse proxy, it is important to configure your application to use the “forwarded headers” added by the reverse proxy. This can be done through the Forwarded Headers middleware.

Starting from .NET Core 3.0 you can take a different approach through the usage of the ASPNETCORE_FORWARDEDHEADERS_ENABLED environment variable. . Turning on the Forwarded Headers Middleware is as simple as setting this environment variable to ‘true’.

Important to notice is that you should only take this approach when you’re:
A. Positive you're behind a trusted proxy
B. AND the deployment environment is unpredictable such that you can't specify KnownProxies or KnownNetworks. A cloud provider is such an environment.

Otherwise you should configure ForwardedHeadersOptions as shown above and set KnownProxies/Networks.

Wednesday, March 17, 2021

Azure DevOps Pipelines–Failing Powershell tasks

Recently some of our Azure DevOps pipelines started to fail. We noticed that this only happened on pipelines where we had tasks that used Powershell behind the scenes.

The error message we got looked like this:

Cannot process an element with node type "Text"

A search on the Internet brought us to a solution. Disabling the “Run PowerShell in Parallel” setting solved the problem.

Anyone with a better solution?

Tuesday, March 16, 2021

Kubernetes–Override an appsettings.json file in an ASP.NET Core app through ConfigMaps

Kubernetes has the concept of a ConfigMap. A ConfigMap gives you an easy way to override files in a Pod what makes it an easy way to replace config files in a container.

Let’s first create a ConfigMap that uses an AppSettings.json:

Next we need to create a deployment where we load this ConfigMap and mount it on the correct path:

Let’s walk to the important parts of the Deployment.yaml step by step:

  • First we need to load the ConfigMap as a volume. We used the name of our configmap ‘example-configmap’ and specified a name to point to this volume ‘appsettings-volume’:

volumes:

- name: appsettings-volume

configMap:

name: example-configmap

  • Now we need to mount this volume on the correct path. Notice that we mount the appsettings.json on the ‘app’ path. This is because the root of our .NET Core app is inside the app folder. We  also specified the subPath to ensure that only the specified file is overridden, and any other existing content in the mount directory is retained:

volumeMounts:

          - name: appsettings-volume

mountPath: /app/appsettings.json

subPath: appsettings.json

Monday, March 15, 2021

Kubernetes–Open a bash prompt on a Pod

When using Kubernetes, sooner or later you’ll have a Pod that doesn’t work as expected. Last friday I blogged about what to do when your Pod is in a crash loop.

Today I’ll share a trick to open a shell on a Pod in trouble.

To open a Bash shell on a failing pod you can execute the following command:

kubectl exec –it <pod name> – bash

This will open a Bash shell on the specified pod:

PS /home/bart> kubectl exec -it example-pod-86f495c7f7-7s87g -- bash
root@example-pod-86f495c7f7-7s87g:/app#

Friday, March 12, 2021

Debug a Pod in a CrashLoopBackoff

When using Kubernetes, sooner or later you’ll encounter a failed deployment where your pods are in CrashLoopBackoff. CrashLoopBackOff tells that a pod crashes right after the start. Kubernetes tries to start pod again, but the pod crashes again and this goes in a loop.

It’s hard to know what is going on as the pod is restarting before you even have a chance to take a look at its logs.

So how to figure out why your pod is failing?

The trick is to call the logs command but include the ‘-p’ parameter to get log messages from previous instantiations as well:

kubectl logs [podname] -p

Thursday, March 11, 2021

AKS with Azure Application Gateway–Reroute to root path

I’m currently working on a setup where we combine AKS(Azure Kubernetes Service) with Azure Application Gateway for ingress. As always it’s quite an adventure especially in a fast moving ecosystem like Kubernetes.

In our setup we have one public IP address and domain. Our different applications are hosted as subsites.

An example:

To do this we configure our ingress object like this:

2 things important to notice here:

  • We added an ‘kubernetes.io/ingress.class’ annotation to specify that we want to use ‘azure/application-gateway’.
  • We use the path to redirect the correct prefix to the correct service. e.g. traffic coming from app1 will be directed to service app1-api-service.

The problem with the setup above is that although our requests arrive perfectly to the correct service and pods, they arrive at the following relative URI /app1. Our pods expect to get the traffic at the root URI and the ASP.NET core routing doesn’t know how to handle the prefix. We could fix this at the Pod level but a better solution is to change this in the ingress and direct the traffic to the root of our pod.

To achieve this we have to add an extra annotation in our ingress object:

appgw.ingress.kubernetes.io/backend-path-prefix: "/"

The final ingress object looks like this:

Wednesday, March 10, 2021

.NET Core–Scan your dependencies for vulnerabilities

Did you know that similar to ‘NPM audit’ you can scan your .NET Core dependencies for vulnerabilities?

Starting from the .NET 5.0.2 SDK, you can scan and list any known vulnerabilities in your dependencies with the dotnet list package --vulnerable command:

By default you only see vulnerabilities within your top-level packages. If you are interested in seeing vulnerabilities within your transitive packages, you can use the --include-transitive parameter:

Safe coding!

Tuesday, March 9, 2021

ASP.NET Core–Prevent redirection on API requests

In one of my ASP.NET Core applications I have Cookie authentication enabled.

When a user tries to access a controller and is not authorized, they are redirect to Account/Login?ReturnUrl=[...]. This is great for MVC endpoints but doesn’t make much sense for API requests.

There are multiple ways to avoid this from happening. The way I solved it is by checking the incoming request in the CookieAuthenticationEvents. When the request starts with ‘/API’, we return a 401 status code instead of redirecting the user:

Monday, March 8, 2021

EF Core–AsSplitQuery()

Entity Framework makes it really easy to fetch related data. For example let’s get all the Customers with their ordered products.

This involves 3 entities: Customer –> Order –> Product.

To fetch this in one go in EF Core, you can use the following LINQ query:

On the database this will end up in a query like this:

The problem is that this results in a cartesian product as the customer data is duplicated for every ordered product in the result set.

A solution exists in EF Core through the AsSplitQuery method that allows Entity Framework to split each related entity into a query on each table:

More information: https://docs.microsoft.com/en-us/ef/core/querying/single-split-queries

Friday, March 5, 2021

Microsoft Ignite 2021 – Book of news

I’m currently bingwatching all sessions from Microsoft Ignite 2021. In case you still want to have some free time the upcoming weeks, you can get a nice summary of the most important announcements through the Microsoft Ignite Book of News:

 

Thursday, March 4, 2021

Editing files in Azure Cloud Shell

Small trick I wasn’t aware of that it was possible in Azure Cloud Shell.

You can directly edit files inside the Cloud shell. Therefore enter ‘code <filename>’:

Wednesday, March 3, 2021

.NET Core HttpClient–Testing redirections

When testing a specific API I had to check if the user was redirected to the correct location. However although I thought that I had written my api correctly, the response code didn’t match.

Here is the related test code:

Do you spot my mistake? Let’s have a look at the documentation:

The default HttpClient will automatically handle redirects what makes it impossible to check for the 302 status code

To change this behavior, you need to create your own HttpHandler and use it when building up your HttpClient instance:

Tuesday, March 2, 2021

NHibernate –Dependency Injection enabled IInterceptor

Yesterday I showed the AuditInterceptor we are using in one of my applications. Maybe you noticed that we were using dependency injection:

Important to notice is that we are injecting a scoped dependency that contains the current user id(through the IUserFactory).

The trick to get this working is to set the interceptor when the session is created.

Therefore we register our IInterceptor implementation in the IoC container(Autofac in our situation):

Then when we construct a new session instance we’ll get the IInterceptor from the container and link it to the session:

Monday, March 1, 2021

NHibernate–Interceptor doesn’t work for inherited objects

I got some trouble with NHibernate. I created the following interceptor to update audit fields before saving changes to the database:

This interceptor does it’s job for simple objects but seems not to work for inherited objects:

To get it working I had to change the interceptor to always return true in the OnSave and OnFlushDirty methods: