Friday, July 24, 2020

Kubernetes–Troubleshooting ImagePullBackOff errors

After deploying a new pod, I couldn’t access it. Time to take a look what was going on:

C:\Users\BaWu\Desktop>kubectl get pods --all-namespaces

NAMESPACE              NAME                                                              READY   STATUS             RESTARTS   AGE

kube-system            addon-http-application-routing-default-http-backend-7fc6fc27bj2   1/1     Running            0          93m

kube-system            addon-http-application-routing-external-dns-6c6465cf6f-hqn2w      1/1     Running            0          93m

kube-system            addon-http-application-routing-nginx-ingress-controller-668m4rb   1/1     Running            0          93m

kube-system            azure-cni-networkmonitor-cn57j                                    1/1     Running            0          10d

kube-system            azure-ip-masq-agent-4sjmw                                         1/1     Running            0          10d

kube-system            coredns-544d979687-5c7rt                                          1/1     Running            0          10d

kube-system            coredns-544d979687-rbbh9                                          1/1     Running            0          10d

kube-system            coredns-autoscaler-78959b4578-jdr24                               1/1     Running            0          10d

kube-system            dashboard-metrics-scraper-5f44bbb8b5-dfw47                        1/1     Running            0          10d

kube-system            kube-proxy-8d5sr                                                  1/1     Running            0          10d

kube-system            kubernetes-dashboard-785654f667-2gcbn                             1/1     Running            0          10d

kube-system            metrics-server-85c57978c6-bzsc2                                   1/1     Running            0          10d

kube-system            omsagent-rs-5f579fcfd-9pqpf                                       0/1     ImagePullBackOff   0          2d10h

kube-system            omsagent-rs-6b6cdf78fc-26mpb                                      1/1     Running            1531       10d

kube-system            omsagent-wfgtp                                                    0/1     ImagePullBackOff   0          2d10h

kube-system            tunnelfront-f7bd7ccb-t7g95                                        2/2     Running            1          6d20h

kubernetes-dashboard   dashboard-metrics-scraper-c79c65bb7-w9thj                         0/1     ImagePullBackOff   0          27m

kubernetes-dashboard   kubernetes-dashboard-56484d4c5-c4cwv                              0/1     ImagePullBackOff   0          27m

The deployment turned out to be in the imagepullbackoff  state. There can be various reasons on why this is the case. Let’s figure out what could cause this by calling describe. This gave us a lot of extra information:

C:\Users\BaWu\Desktop>kubectl describe pod kubernetes-dashboard-56484d4c5-c4cwv --namespace=kubernetes-dashboard

Name:         kubernetes-dashboard-56484d4c5-c4cwv

Namespace:    kubernetes-dashboard

Priority:     0

Node:         aks-agentpool-27676582-vmss000000/

Start Time:   Fri, 24 Jul 2020 12:53:52 +0200

Labels:       k8s-app=kubernetes-dashboard


Annotations:  <none>

Status:       Pending




Controlled By:  ReplicaSet/kubernetes-dashboard-56484d4c5



    Container ID:

    Image:         kubernetesui/dashboard:v2.0.0

    Image ID:

    Port:          8443/TCP

    Host Port:     0/TCP




    State:          Waiting

      Reason:       ImagePullBackOff

    Ready:          False

    Restart Count:  0

    Liveness:       http-get https://:8443/ delay=30s timeout=30s period=10s #success=1 #failure=3

    Environment:    <none>


      /certs from kubernetes-dashboard-certs (rw)

      /tmp from tmp-volume (rw)

      /var/run/secrets/ from kubernetes-dashboard-token-bxq7s (ro)


  Type              Status

  Initialized       True

  Ready             False

  ContainersReady   False

  PodScheduled      True



    Type:        Secret (a volume populated by a Secret)

    SecretName:  kubernetes-dashboard-certs

    Optional:    false


    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)


    SizeLimit:  <unset>


    Type:        Secret (a volume populated by a Secret)

    SecretName:  kubernetes-dashboard-token-bxq7s

    Optional:    false

QoS Class:       BestEffort



        for 300s

        for 300s


  Type     Reason     Age                  From                                        Message

  ----     ------     ----                 ----                                        -------

  Normal   Scheduled  30m                  default-scheduler                           Successfully assigned kubernetes-dashboard/kubernetes-dashboard-56484d4c5-c4cwv to aks-agentpool-27676582-vmss000000

  Normal   Pulling    28m (x4 over 30m)    kubelet, aks-agentpool-27676582-vmss000000  Pulling image "kubernetesui/dashboard:v2.0.0"

  Warning  Failed     28m (x4 over 30m)    kubelet, aks-agentpool-27676582-vmss000000  Failed to pull image "kubernetesui/dashboard:v2.0.0": rpc error: code = Unknown desc = Error response from daemon: Get net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

  Warning  Failed     28m (x4 over 30m)    kubelet, aks-agentpool-27676582-vmss000000  Error: ErrImagePull

  Normal   BackOff    27m (x6 over 30m)    kubelet, aks-agentpool-27676582-vmss000000  Back-off pulling image "kubernetesui/dashboard:v2.0.0"

  Warning  Failed     26s (x117 over 30m)  kubelet, aks-agentpool-27676582-vmss000000  Error: ImagePullBackOff

This indicates that the Kubernetes cluster cannot talk to This makes sense as I only configured a trust with ACR.

Thursday, July 23, 2020

Kubernetes–the server could not find the requested resource

When trying to deploy the Kubernetes dashboard on an AKS cluster it failed with the following error message:

Error from server (NotFound): the server could not find the requested resource

Here was the full command I tried to execute:

C:\Users\BaWu>kubectl apply -f

Error from server (NotFound): the server could not find the requested resource

The problem was related to the fact that kubectl supports one version (older or newer) of kube-apiserver.

I checked the installed version using:

C:\Users\BaWu>kubectl version

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"windows/amd64"}

Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"89d8075525967c7a619641fabcb267358d28bf08", GitTreeState:"clean", BuildDate:"2020-06-23T02:52:37Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

I had version 1.9 installed where the api was on version 1.16. To solve it I had to download and install a newer version of kubectl. Instructions to do this can be found here:

Wednesday, July 22, 2020

Using gRPC for your internal microservices

gRPC is a really great fit as a communication protocol between your (internal) microservices. It runs on top of HTTP/2 and gives you great performance.

One caveat is that the HTTP/2 prerequisite requires by default that all communication is happening securely. So you have to setup TLS and create certificates for all your services.

But what should you do when you are using Kubernetes and use TLS termination at the ingress controller level?

A new feature announced in .NET Core 3.0 brings rescue. You can turn on unencrypted connections for HTTP/2 by setting the DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP2UNENCRYPTEDSUPPORT environment variable to 1 or by enabling it in the app context:

Tuesday, July 21, 2020

Razor Class Libraries–Static Content 404 issue –Part 2

I’ll continue my journey using Razor Class Libraries in ASP.NET Core.

Here are my previous posts:

After a first colleague returned to his desk with a solution for the problem I discussed yesterday, a second colleague arrived and explained he had a similar problem.

This time I could pinpoint the issue to the following PackageReference that was (still) used in a referenced project:

<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.2.0" />

Static files worked differently in .NET Core 2.2 Razor Class Libraries. The inclusion of the Microsoft.AspNetCore.Mvc v2.2.0 library broke the behaviour in ASP.NET Core 3.x applications. This reference is no longer needed as it is now a part of the Microsoft.AspNetCore.App framework reference.

Monday, July 20, 2020

Razor Class Libraries–Static Content 404 issue –Part 1

I’ll continue my journey using Razor Class Libraries in ASP.NET Core.

Here are my previous posts:

Today I want to share an issue a colleague got when he tried to use a Razor Class Library I created.

When he tried to load a static asset from the RCL, the asset was not found and a 404 error was returned to the browser.

It took me a while to pinpoint the issue but in the end it turned out that the following setting in his ASP.NET Core project caused the problem:


After commenting out this line in the csproj file, the static assets were loaded correctly

I have no clue why this solved the problem as I don’t see any relation between these features…

Friday, July 17, 2020

Razor Class libraries–Clean up your content path

I’ll continue my journey using Razor Class Libraries in ASP.NET Core.

Here are my previous posts:

Today I want to write a small addition to my post from yesterday. As I mentioned yesterday to reference some content inside your razor views you need to use a path similar to _content/{LIBRARY NAME}.

This path doesn’t look so nice. Luckily you change it to a different path name by editing the RCL project properties and adding a StaticWebAssetBasePath.

Now you can access your files using /myownpath/example.css.

Thursday, July 16, 2020

Razor Class Libraries–Static Content

I’ll continue my journey using Razor Class Libraries in ASP.NET Core.

Here are my previous posts:

Today I want to cover how you can use static content inside your Razor Class library.

To include static content(images, fonts, stylesheets,…) in your RCL you need to create a wwwroot folder in the class library and include any required files there:

When packing an RCL, all content in the wwwroot folder is automatically included in the package.

As this content becomes part of the DLL you cannot just reference them from the root path(“~”. Instead the path is constructed using ‘_content/{LIBRARY NAME}/’.

For example to reference an example.css file that you stored inside a RCL named ‘Example.RCL’, the correct way to include this css file in your application becomes:

Wednesday, July 15, 2020

Kubernetes- The Virtual Kubelet

If you are looking at running Kubernetes in a cloud, sooner or later you’ll hear about ‘Virtual Kubelet’. But what is it?

Let’s first go to and look at the definition there:

Virtual Kubelet is an open-source Kubernetes kubelet implementation that masquerades as a kubelet.

Mmm. That didn’t help a lot. But wait there is a second sentence, maybe that will explain everything:

This allows Kubernetes nodes to be backed by Virtual Kubelet providers such as serverless cloud container platforms.

Nope. Doesn’t ring a bell. Let’s try to explain this in our own words:

A Kubernetes cluster is divided into two components:

  • Control plane nodes provide the core Kubernetes services and orchestration of application workloads.
  • Nodes run your application workloads.

Kubernetes expects that a node is a virtual (or for the vintage fans a physical) machine. On every node a kind of agent is running that maanages the containers that were created by Kubernetes and runs them on the node it manages. This agent is called a ‘kubelet’.

Okay, we are one step closer. We know what a kubelet is. But what is than a ‘virtual kubelet’?

Let’s introduce the cloud into the picture. On most cloud platforms you can not only run virtual machines but typically there are other (higher level) managed services; like for example serverless with Azure functions or Azure Container instances. Virtual Kubelet is an open-source implementation of Kubernetes kubelet with the purpose of connecting Kubernetes to other APIs. It registers itself as a node and allows us to deploy on top of other cloud native services not limited to virtual machines.

Aha, finally we get the picture…

Tuesday, July 14, 2020

.NET Conf “Focus on Microservices”

Every year I shout out on my blog that a new edition of .NET Conf is coming(this year November 10-12 for the .NET 5 launch).

What I was not aware of is that the organizers of .NET Conf started a series of smaller events focused on specific things you can do with .NET. There have been 2 editions so far; one focusing on Blazor and the other one on Xamarin.

The next one is about Microservices (who could have guessed that?) on July 30, 2020.

.NET Conf: Focus on Microservices is a free, livestream event that features speakers from the community and .NET teams that are working on designing and building microservice-based applications, tools and frameworks. Learn from the experts their best practices, practical advice, as well as patterns, tools, tips and tricks for successfully designing, building, deploying and running cloud native applications at scale with .NET.

Check out the agenda and the amazing list of speakers.

Monday, July 13, 2020

Razor Class Libraries–Views not found

Last week I talked about Razor Class Libraries as a nice and easy way to package and share UI components for your ASP.NET Core MVC/Razor pages application.

Inside my Razor Class Library I had some shared ViewComponents that I placed in a Shared folder. Unfortunately when trying to use these ViewComponents inside my ASP.NET Core Web application, the ViewComponents were not found?!

It costed me some headaches before I discovered what I was doing wrong. It is really important that your shared View(Components) reside under a Views folder as this is where Razor will search it views by convention. You can override the convention if you want to but it is probably easier to just move everything to a Views folder like I did:

Friday, July 10, 2020

MassTransit–Youtube videos

Great tip of you want to learn everything about the ins and outs of MassTransit. Chris Patterson took the time to create a great (free) video set available here:

Thursday, July 9, 2020

ASP.NET Core–Razor Class Libraries

A little known feature in ASP.NET Core (MVC/Razor pages) is the support for Razor Class libraries(RCL).  Through the RCL you can package and reuse UI components like View Components or Razor Components but also share full feature areas containing Razor view, controllers, models and so on…

Let’s get started

  • Open Visual Studio and choose Create a new project.
  • Select Razor Class Library and click Next.

  • Give the library a name and click on Create.
    • Remark 1: To avoid a file name collision with the generated view library, ensure the library name doesn't end in .Views.
    • Remark 2: Select Support pages and views if you need to support views. Otherwise only Razor components are supported.

Now you can start adding your View Components and Razor components to your RCL. When building your RCL 2 DLL’s are created:

  • One DLL containing all your logic
  • One DLL containing your Razor views (ends with .Views)

You can now either reference this RCL directly or include it in your project through NuGet.

What’s nice is that you still can override a view by providing the same Razor markup file (.cshtml) in your web app. The file in your web app will always take precedence.

Wednesday, July 8, 2020

MassTransit–Debugging your configuration

When configuring MassTransit there are a lot of moving parts. You can have multiple endpoints, middlewares, … that all impact your application.

To understand how your bus instance is wired together you can use the GetProbeResult method:

Before you can use the code above, you’ll need this small extension method:

More info:

Tuesday, July 7, 2020

Entity Framework Core - Soft delete

The easiest way to implement soft delete in EF Core is through query filters.

This filter can be specified when creating our EF Core model:

Now every time when query for the ‘Role’ object an extra ‘WHERE’ clause is included that filters with IsDeleted=false.

But hold your horses we are not there yet, we also have to override our SaveChanges()and SaveChangesAsyc() methods on the DbContext otherwise the ‘Role’ entity will still be removed from the database when we call Remove().

Remark: As a possible improvement you could generate a base class or interface for all ‘soft deletable’ entities.

Monday, July 6, 2020

Quality is a team issue

I forgot where I found the following quote but I copied and printed it out as a reminder for myself:

Quality is a team issue. The most diligent developer placed on a team that just doesn’t care will find it difficult to maintain the enthusiasm needed to fix niggling problems. The problem is further exacerbated if the team actively discourages the developer from spending time on these fixes.

It remains one of the biggest lessons I learned during my software career and it manifested itself in 2 ways:

  • Supermotivated developers eager to learn new things ending in a burn/born out after six months on a project. I saw really talented and motivated people (and especially these people) getting completely fed up by an organization not willing to move. I even saw people leave the IT industry because of this.
  • A negative trend in general software quality when one or more of the team members didn’t put the quality bar at the same level as the rest of the team. How much we tried to convince the developer to raise the bar, it unfortunately always ended to a lowering of the quality standard of the whole team instead. (See the Broken Window story)

Luckily I’ve also seen the other way around; when everyone aims for the same quality level (no matter if you are a UX’r, developer, analyst, tester, architect, …) than miracles happen. No problem becomes too complex to tackle and speed of delivery increases without sacrificing on quality.

But it all starts with one common value; that quality is a team issue…

Friday, July 3, 2020

Entity Framework - Mapping issue

I had the following entities in my domain:

And this is how it was mapped through Entity Framework:

Unfortunately Entity Framework got confused when I tried to create a new Role and RoleAssignment. It generated the wrong query for the RoleAssignment and tried to set the “Id” field to the Role id.

To fix it I had to explicitly tell Entity Framework to use the Role_Id column to map the foreign key relationship:

Strange thing is that this code was working in a previous release of the application.

Thursday, July 2, 2020

Razor compilation

In .NET Core (3.x) Razor compilation happens by default at build time. This means that every time you make a change to a Razor file, you have to rebuild your application before a change becomes visible inside your frontend. This is different compared to .NET where Razor compilation happened at runtime.

If you have to work a lot inside Razor, build time compilation can be annoying and slow down your development process. Luckily it is possible to enable runtime compilation for your ASP.NET Core Web application.

One option you have is to enable runtime compilation at project creation:

  • Create a new ASP.NET Core Web application in Visual Studio
  • Check the Enable Razor runtime compilation check box when creating your project.

If you have an existing project, you have to take a different approach:

Wednesday, July 1, 2020

.NET Core–Backgroundservice lifetime

.NET Core 3 introduces a new worker template that uses the concept of a BackgroundService. The BackgroundService is a base class for implementing a long running IHostedService.

One important thing to be aware of is that the BackgroundService has a different lifetime than the host. This means that although a BackgroundService exits, it doesn’t mean that the host application will exit as well.

One way to guarantee that the host application stops when the BackgroundService exits, is by injecting the IHostApplicationLifetime in your service and call StopApplication():