After upgrading our ElasticSearch cluster, I restarted our batch processing service. This service gets a data offload every night from an external party and we'll use this data to build up our search index.
The BulkAllObservable helper
As a lot of documents should be indexed, I use the BulkAllObservable helper to optimize the process. This helper is part of the ElasticSearch.NET client and takes care of all the complex retry, backoff or chunking logic. It gives you full control on how the underlying BulkApi should be called.
This code worked perfectly before. Unfortunately after upgrading to Elastic 8.3, that was no longer the case.
Enable compatibility mode
I saw the following error message in the logs:
BulkAll halted after receiving failures that can not be retried from _bulk at Nest.BulkAllObservable`1.HandleDroppedDocuments(List`1 droppedDocuments, BulkResponse response) at Nest.BulkAllObservable`1.<BulkAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Nest.Extensions.<ProcessAsync>d__30`2.MoveNext()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Nest.Extensions.<ForEachAsync>d__29`2.MoveNext()
I enabled DebugMode:
And saw the following lower level error message:
Invalid NEST response built from a successful (200) low level call on POST
This brought me to the following Github issue: https://github.com/elastic/elasticsearch-net/issues/6154
To fix the issue, I had to update the latest 7.x ElasticSearch.NET client (The 8.x client was still in beta when I wrote this post) and enable the compability mode by calling .EnableApiVersioningHeader();