The art of simplicity

Last week I introduced RabbitMQ streams and how you could produce and consume streams through the RabbitMQ.Stream.Client in .NET. Yesterday I showed how you can improve and simplify producing messages by using a Reliable Producer. Today I want to introduce its counterpart on the consumer side; the Reliable Consumer. Introducing Reliable Consumers Reliable Consumers builts on top of Consumer and adds the following features: Auto-Reconnect in case of disconnection Auto restart consuming from the last offset Handle the metadata Update Auto-Reconnect The Reliable Consumer will try to restore the TCP connection when the consumer is disconnected for some reason. Auto restart consuming from the last offset The Reliable Consumer will restart consuming from the last offset stored. So you don’t have to store and query the last offset yourself. Handle the metadata update If the streams  topology changes (ex:Stream deleted or add/remove follower), the client re...

Yesterday I talked about RabbitMQ Streams, a new persistent and replicated data structure in RabbitMQ 3.9 which models an append-only log with non-destructive consumer semantics. I demonstrated how you could build a small example application in C# to test this stream. I first tried this code against a local cluster I had running in a docker container (check my repo if you want a Dockerfile where this plugin is already enabled: https://github.com/wullemsb/docker-rabbitmq ). At first this failed with the following error message: System.Net.Sockets.SocketException: No such host is known In my configuration you can see that I’m pointing to the local Loopback address which should be localhost: Let’s open the debugger and see what is going on… When I looked at the connection settings I noticed the following: You can see that the advertised host is not ‘localhost’ but a random string. This is the random name assigned to the node in my cluster. To get rid of this pr...

RabbitMQ has been the message broker of my choice for a long time. It has served me well over the years and I still like to use it today. Recently, I was able to add an extra reason to the list why I like RabbitMQ when I noticed that a new feature was added in RabbitMQ 3.9; Streams . RabbitMQ Streams From the documentation : Streams are a new persistent and replicated data structure in RabbitMQ 3.9 which models an append-only log with non-destructive consumer semantics. With streams you get Kafka like functionality in RabbitMQ without all the complexity that comes with maintaining and managing your Kafka cluster. It has been created with the following use cases in mind: Large amount of subscribers; in traditional queuing we use a dedicated queue for each consumers. This becomes ineffective whehn we have large number of consumers. Time-travelling; Streams will allow consumers to attach at any point in the log and read from there. Performance: Streams have been ...

If you are interested in message or event based architectures, bookmark the link to the following Github repo: https://github.com/clemensv/messaging This repository is created by Clemens Vaster, the product architect of the messaging and eventing services in the Microsoft Azure cloud. It refers to a lot of talks and articles that help you to get started with message and event based systems and links to the product pages of multiple cloud providers and open source products that exist in this space.

The default serialization format for Kafka is Avro. I mentioned how to use this in C# yesterday. Today I got into trouble when trying to send a specific message. I changed the example a little bit to explicitly point out the issue. You see in the code above that I set the value for ‘Browser’ to ‘null’. When trying to send this message it failed with the following error: Local: Value serialization error Let’s have a look at the related avro schema: The problem is that in the schema is specified that the Browser field should a value of type string. ‘Null’ is not a valid value for string. This explains why it fails. To solve this I have two options; 1) Either change the code to send an empty string instead of null: 2) Either update the schema to allow null values for the browser field: More about Avro: https://en.wikipedia.org/wiki/Apache_Avro

To help you with using Avro as the serialization format for your Kafka messages, a .NET core global tool avrogen is available. First install the tool using dotnet tool install: Next step is to specify your message schema. Therefore you need to create an .avsc file and add your message specification: Now it’s time to generate the necessary code: This will generate the following: The generated type can than be used by your Producer and Consumer logic. More information: https://www.confluent.io/blog/avro-kafka-data/

My first attempt to send a typed message through Kafka resulted in the following error message: Value cannot be null. (Parameter 'Value serializer not specified and there is no default serializer defined for type PageViewEvent) Here is the code I was using: As the error message mentions, you need to explicitly specify what serializer should be used for your message object. Therefore you need to use the SchemaRegistryClient and specify a serializer(I’m using Avro in the sample below):

By default when you use the Confluent Kafka .NET client , a topic is created automatically for you when you publish your first message. However this will create a topic using the default settings. Typically you want to have more control when creating a topic. This is possible through the usage of the AdminClient: