The art of simplicity

Last week, one of the server admins contacted me that the disk of our ElasticSearch instance was filling up quite fast. He asked me if this was normal? :-) Short answer, no. Let's investigate the issue together... I started by opening up Kibana and opened Stack Management through the menu on the left. On the Stack Management page, I clicked on Index Management . Here I can see the list of all indices. I sorted on Storage size and noticed that MetricBeats was causing the issue. That is strange as I know for sure that I had configured a lifecycle policy that would cleanup old data. But then I noticed that the health of one of the indices is ‘yellow’. Let’s see what causing this by executing the following request: GET /.ds-metricbeat-8.2.0-2022.12.03-000011/_ilm/explain?human This returns the following result: Notice line 24. The problem was that the index was configured to have at least one replica. As we are running a single node cluster in development, the...

At one of my clients we are using MetricBeats to monitor our RabbitMQ cluster . Of course this can result in a lot of data over time. Recently I was called by one of the system administrators asking why the disk was filling up on the servers hosting ElasticSearch. Let’s find out together… Index lifecycle policies To keep the amount of data on your ElasticSearch cluster under control, you can configure a lifecycle policy . A lifecycle policy moves data through multiple phases; A hot phase: used for your most recent most frequently searched data. It provides the highest indexing AND search performance but comes with the highest cost A warm phase: optimized for search performance over index performance. In this phase it is expected that the data doesn’t change that often anymore. A cold phase: optimized for cost saving over search performance. In this phase the data is read-only. A delete phase: deletes the data you longer need Let’s see how to configure a lifecy...

After upgrading our ElasticSearch cluster, I restarted our batch processing service. This service gets a data offload every night from an external party and we'll use this data to build up our search index. The BulkAllObservable helper As a lot of documents should be indexed, I use the BulkAllObservable helper to optimize the process. This helper is part of the ElasticSearch.NET client and takes care of all the complex retry, backoff or chunking logic. It gives you full control on how the underlying BulkApi should be called. This code worked perfectly before. Unfortunately after upgrading to Elastic 8.3, that was no longer the case. Enable compatibility mode I saw the following error message in the logs: BulkAll halted after receiving failures that can not be retried from _bulk   at Nest.BulkAllObservable`1.HandleDroppedDocuments(List`1 droppedDocuments, BulkResponse response)  at Nest.BulkAllObservable`1.<BulkAsync>d__20.MoveNext() --- End of s...

One of the reasons we are using ElasticSearch is to bring all the metrics from multiple systems together in one place. One of the systems we want to monitor is our RabbitMQ cluster. Let's walk through the steps to get our RabbitMQ metrics send to ElasticSearch and available in Kibana. The RabbitMQ Metrics Integration Open up Kibana and click on the hamburger icon in the left corner. This will open up a side menu with a big ‘Add integrations’ button at the bottom. Hit this button to go to the Integrations overview page. Here you can see the list of all available integrations. Let’s search for “rabbitmq”. We get 2 results back. Click on the ‘RabbitMQ Metrics’ integration to go to the installation instructions. Install MetricBeat on Windows The RabbitMQ metrics integration is part of MetricBeat . MetricBeat is a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on those ser...

Upgrading ElasticSearch can be quite challenging(at least for me it was). I talked about some other issues I had while upgrading before . Let's add a last one to the list... When talking about the bootstrap check failures in a previous post , I shared my final solution. But if you were paying attention as a reader you maybe noticed that I didn’t follow the suggestions mentioned inside the bootstrap check error: bootstrap check failure [1] of [2]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured I didn’t set the discovery.seed_hosts , discovery.seed_providers or cluster.initial_master_nodes setting as mentioned, instead I changed the discovery.type to single-node . In fact my first attempt looked like this: However this resulted in the following error message: [2022-05-09T11:33:18,943][WARN ][stderr      ...

Upgrading ElasticSearch can be quite challenging(at least for me it was). I talked about some other issues I had while upgrading before. Let's add another one to the list... One of the ways how ElasticSearch helps you to do the right things is to do some bootstrap checks . These bootstrap checks inspect a variety of Elasticsearch and system settings and compare them to values that are safe for the operation of Elasticsearch. If Elasticsearch is in development mode, any bootstrap checks that fail appear as warnings in the Elasticsearch log. If Elasticsearch is in production mode, any bootstrap checks that fail will cause Elasticsearch to refuse to start. In this case the bootstrap checks returned the following errors: [2022-05-09T09:53:42,647][ERROR][o.e.b.Bootstrap          ] [SERVER1] node validation exception [2] bootstrap checks failed. You must address the points described in the following [2] lines before starting Elasticsearch...

Upgrading ElasticSearch can be quite challenging(at least for me it was). I talked about some other issues I had while upgrading before. Let's add another one to the list... After installing ElasticSearch 8.1 I couldn’t get it started. Here is the error I got instead: 12:31:34 » Exception in thread "main" java.lang.RuntimeException: starting java failed with [1] 12:31:34 » output: 12:31:34 » 12:31:34 » error: 12:31:34 » Unrecognized VM option 'UseConcMarkSweepGC' 12:31:34 » Error: Could not create the Java Virtual Machine. 12:31:34 » Error: A fatal exception has occurred. Program will exit. 12:31:34 » at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:118) 12:31:34 » at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:86) 12:31:34 » at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59) 12:31:34 » at org.elasticsearch.tools.launchers.JvmOptionsParser.main...

After installing a new ElasticSearch cluster I updated the elasticsearch.yml to expose it outside the local Virtual Machine. This is how the updated elasticsearch.yml looked like: There is definitely something wrong with this configuration as ElasticSearch refused to start. A look at the log file showed the following error information: [2022-05-09T09:45:02,104][ERROR][o.e.b.Bootstrap] [SERVER1] Exception org.elasticsearch.transport.BindTransportException: Failed to bind to 192.168.0.1:[9300-9399] at org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:489) ~[elasticsearch-8.2.0.jar:8.2.0] at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:450) ~[elasticsearch-8.2.0.jar:8.2.0] at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:147) ~[?:?] at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:96) ~[?:?] at org.elastic...

Yesterday I shared how I got into trouble when trying to uninstall an ElasticSearch instance. I explained how we could investigate the issue and find the root cause by checking the log files. Today I want to continue by explaining what I found in the log files and how I solved the problem. While browsing through the log file, I noticed the following error message: Calling custom action Elastic.Installer.Msi!Elastic.Installer.Msi.Elasticsearch.CustomActions.Immediate.ElasticsearchExistingDirectoriesAction.ElasticsearchExistingDirectories System.Exception: Can not execute Elastic.InstallerHosts.Elasticsearch.Tasks.Install.ExistingDirectoriesTask the model that it was passed has the following errors BADELASTICSEARCHYAMLFILE: The elasticsearch.yml file we found in ES_PATH_CONF appears to be invalid  and prevented seeding current values. It seems that ElasticSearch checks the elasticsearch.yml configuration file during the uninstall and doesn’t like what it finds there(whic...

Elastic has their own Application Performance Monitoring solution as part of their Elastic Observability product. An important part of the solution are ‘agents’. Agent are responsible for instrumenting your application, collecting all the metrics and sending it to the APM server. Specifically for .NET the APM agent is released as a serie of NuGet packages . With the release of OpenTelemetry for .NET I was wondering if we could replace this Elastic APM specific solution with standard OpenTelemetry. In a first incarnation APM server didn’t support the OpenTelemetry standard and you had to use a seperate collector that converts the OpenTelemetry data to the Elastic APM format: Since version 7.13 of Elastic APM this is no longer necessary.The OpenTelemetry Collector exporter for Elastic was deprecated and replaced by the native support of the OpenTelemetry Line Protocol in Elastic Observability (OTLP). Now the only thing you need to do is to add some specific attributes ...

Installing ElasticSearch in a Kubernetes cluster is easy thanks to the available operator . Unfortunately that doesn’t guarantee a successfull rollout. Let’s see how we can find out why a deployment failed. An ElasticSearch cluster is rolled out as a statefulset. So let’s first have a look a the available statefulsets in our AKS cluster: PS /home/bart> kubectl get statefulset No resources found in default namespace. NAME                READY   AGE search-es-default   0/1     13h OK, we see that the statefulset is not ready. Let’s continue our investigation and check the related pods: PS /home/bart> kubectl get pods -l elasticsearch.k8s.elastic.co/statefulset-name=search-es-default NAME                  READY   STATUS    RESTART...

I’m a big fan of ElasticSearch. One of the products in the Elastic suite is App Search . I tried to get some documents through the App Search API using the following code: Unfortunately this didn’t work but resulted in the following error message: JSON must be an array or hash The reason I got this exception is because the API is expecting a JSON array or object for the id parameter. Unstructured JSON data is not permitted. You have to encapsulate your object in an array [] or a hash {} (what the error message is referring to). If you are using query parameters, ensure that ids is followed with %5D%5B - this is an array, but escaped: ?ids%5D%5B=:

After upgrading your Azure DevOps Server instance, you should also upgrade your ElasticSearch instance. If the instance is running on the same server as Azure DevOps, it can be automatically updated using the Configure Search feature in the Azure DevOps Administration console. In case you are running ElasticSearch on a different machine (what I would recommend) you need to copy the Search service package to the target server. You can find the link in the wizard that is used to Configure Search: On the remote server extract the installer files and execute the following command: \Configure-TFSSearch.ps1 --operation update The installer should be able to find the existing ElasticSearch instance and update it for you. More information: https://docs.microsoft.com/en-us/azure/devops/project/search/administration

When trying to update an ElasticSearch cluster through the Windows Installer(MSI) it always seemed to fail. In the error logs I found the following message: System.IO.IOException: Source and destination path must have identical roots. Move will not work across volumes.    at System.IO.Directory.InternalMove(String sourceDirName, String destDirName, Boolean checkHost)    at System.IO.Abstractions.DirectoryWrapper.Move(String sourceDirName, String destDirName)    at Elastic.InstallerHosts.Elasticsearch.Tasks.Install.CreateDirectoriesTask.CreateConfigDirectory(FileSystemAccessRule rule)    at Elastic.InstallerHosts.Elasticsearch.Tasks.Install.CreateDirectoriesTask.ExecuteTask()    at Elastic.InstallerHosts.SessionExtensions.Handle(Session session, Func`1 action) There is a problem with the installer when you are using different volumes for your ElasticSearch application and your ElasticSearch data(which I think...

While preparing for an ElasticSearch upgrade I checked the Upgrade assistant . One (critical) issue was mentioned: Discovery configuration is required in production mode Let’s have a look what the documentation has to mention: Production deployments of Elasticsearch now require at least one of the following settings to be specified in the elasticsearch.yml configuration file: discovery.seed_hosts discovery.seed_providers cluster.initial_master_nodes discovery.zen.ping.unicast.hosts discovery.zen.hosts_provider The first three settings in this list are only available in versions 7.0 and above. If you are preparing to upgrade from an earlier version, you must set discovery.zen.ping.unicast.hosts or discovery.zen.hosts_provider . In our case we don’t want to form a multiple-node cluster. So we ignore the documentation above and instead change the discovery.type to single-node. For more information about when you might use this se...

After upgrading ElasticSearch.NET to the latest version, my application failed with the following error message: Could not load type 'Elasticsearch.Net.IInternalSerializerWithFormatter' from assembly 'Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d'. A look at my packages.config (yes it is still an old(er) ASP.NET application), showed the following: <package id="CommonServiceLocator" version="2.0.1" targetFramework="net461" requireReinstallation="true" /> <package id="Elasticsearch.Net" version="7.7.1" targetFramework="net472" /> <package id="Iesi.Collections" version="4.0.1.4000" targetFramework="net461" /> <package id="LazyCache" version="0.7.1.44" targetFramework="net461" /> <package id="Microsoft.CSharp" version="4.6.0" tar...

By default Kibana is configured to only be accessible from ‘localhost’. If you want to expose it outside your server, you’ll have to update the configuration. Go to the config folder and open the kibana.yml file. Find the following section: # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. # server.host: "localhost" Uncomment the ‘server.host’ section and specify a non-loopback address. An example: server.host: "0.0.0.0" Don’t forget to also open up the necessary ports in your firewall(by default port 5601)

For Kibana(part of the ELK) stack no out-of-the-box script exists to install and run it as a windows service. (This in contrast to ElasticSearch which has a batch file that allows you to install it as a windows service). As a workaround you can use NSSM ; the Non-Sucking Service Manager. With NSSM you can take any executable and run it as a windows service. Here are the steps to use it: Download NSSM . Extract the zip and put the nssm.exe executable on a location of your choice. Run nssm install <servicename> ; .e.g. nssm install kibana This will open up a configuration window where you can specify the executable you want to run and configure some other service related settings Click on Install service.

As mentioned in some of the previous posts I am migrating an ‘old’ 5.3 instance of ElasticSearch to 7.6.1. In my first (too optimistic) attempt I directly migrated to ElasticSearch 7.6.1. When starting the ElasticSearch cluster this resulted in the following error message: [myindex/gTTtwHT3ShiAz-eR94eKmw]] was created with version [5.3.0] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.6.1.                at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkSupportedVersion(MetaDataIndexUpgradeService.java:113)                at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:87)                at org.elas...

While migration from ElasticSearch 5 to 7.6.1 I had to take an extra step and first migrate to ElasticSearch 6.8. When I tried to run the 6.8 version, it failed with the following error message: 2020-03-30 10:18:32 Commons Daemon procrun stderr initialized OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Exception in thread "main" java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.jdk.internal.vm.annotation")                at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)                at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)      ...