Friday, August 31, 2018

Angular - Karma–Chromeless tests fail with ERROR - Disconnected, because no message in 10000 ms.

After switching from PhantomJs to Chrome in headless mode for our Angular tests, our tests started to behave flaky with sometimes success and sometimes failure. When the tests failed, we saw the following error message in the logs:

[32m28 08 2018 10:41:44.851:INFO [HeadlessChrome 68.0.3440 (Windows 7.0.0)]: [39mConnected on socket W4CI4bKonF845CKZAAAA with id 22875346

2018-08-28T08:41:54.8669569Z [33m28 08 2018 10:41:54.860:WARN [HeadlessChrome 68.0.3440 (Windows 7.0.0)]: [39mDisconnected (1 times), because no message in 10000 ms.

2018-08-28T08:41:54.8864889Z HeadlessChrome 68.0.3440 (Windows 7.0.0) ERROR

2018-08-28T08:41:54.8864889Z   Disconnected, because no message in 10000 ms.

2018-08-28T08:41:54.8874655Z HeadlessChrome 68.0.3440 (Windows 7.0.0) ERROR

2018-08-28T08:41:54.8874655Z   Disconnected, because no message in 10000 ms.

2018-08-28T08:41:54.8874655Z

To fix it, we changed the default Karma settings to a more fault tolerant configuration.Therefore we opened up the karma.conf.js file and added the following 2 settings:

  • browserDisconnectTolerance: 2, // maximum number of tries a browser will attempt in the case of a disconnection
  • browserNoActivityTimeout: 50000, // How long will Karma wait for a message from a browser before disconnecting from it (in ms).

Here is our complete karma.conf.js file:

Thursday, August 30, 2018

FluentValidation MustAsync causes API timeout

One of the developers in my team complained that one of the ASP.NET Web API requests was really slow and asked me to investigate the performance issue.

When I opened the solution I saw the following code:

Do you notice what’s wrong?

The problem is that inside the validator an async call is done where if you look at the controller the validator is invoked in a synchronous way. This causes the API to hang as ASP.NET keeps waiting for a result that is never returned.

The solution was simple; just switch to the ValidateAsync method and await the result in the controller:

Wednesday, August 29, 2018

Running chrome in headless mode for Angular E2E tests

If you used the Angular CLI to create your Angular project, there is not that much work left to do to start using Chrome in headless mode to run your end-to-end tests.

Let’s walk through the steps:

  • Open the protractor.conf.js file in the e2e folder. Change the capabilities entry in the config to include a chromeOptions object like this:
capabilities: {   chromeOptions: {     args: [ "--headless" ]   },   'browserName': 'chrome'
},

By using the args property, we pas a --headless option. This configures Chrome to run in headless mode.

That is all we need to do . Just kick off your E2E tests using:

npm run e2e

Tuesday, August 28, 2018

ASP.NET Core–Swagger error when using a virtual directory

After changing our ASP.NET Core application to no longer run under the root site(e.g. https://localhost/) but under a virtual directory instead (e.g. https://localhost/mysampleapp/) , loading the API definitions in Swagger no longer worked and failed with the following error message:

Fetch error undefined swagger/v1/swagger.json

clip_image002

To fix the problem, I had to do 2 things:

First I updated the Swagger json generation to set the SwaggerDoc to a specific path. Although the documentation mentions the first parameter of SwaggerDoc as the name, it becomes part of the URI of your swagger.json file.

In this case I set the name to ‘v1’:

As a second step, I had to update the Swagger UI configuration to generate the swagger endpoint on a relative path. Important here is that the name of the swagger doc matches part of the URI before the swagger.json; in our case ../swagger/v1/swagger.json

Monday, August 27, 2018

Ionic Menu control won’t refresh

Last week we lost a lot of time investigating an issue with ‘ion-menu’, one of the built-in controls of Ionic. The problem was that once the menu was opened it would no longer refresh it’s data.

image

What we do is showing some user data on the top of the menu, this data is returned with an http call from the server and updated through an observable subscription. However pushing a new value on the observable had no effect on the view although the component data was in fact updated.

As a workaround we used the built-in event bus to publish an event once the user was loaded. We handle this event on the viewcomponent and use it to explictly trigger a change detection:

More information about the ChangeDetectorRef can be found here: https://alligator.io/angular/change-detection-strategy/

Friday, August 24, 2018

Discover the world of .NET with .NET Conf

The next edition of .NET Conf is getting closer. From September 12 until September 14, you can learn about the latest and greatest in the .NET ecosystem.

image

From the .NET Conf site:

.NET Conf is a FREE, 3 day virtual developer event co-organized by the .NET community and Microsoft.

Over the course of the three days you have a wide selection of live sessions streaming here that feature speakers from the community and .NET product teams. It's a chance to learn, ask questions live, and get inspired for your next software project.

You will learn to build for web, mobile, desktop, games, services, libraries and more for a variety of platforms and devices all with .NET. We have sessions for everyone, no matter if you are just beginning or are a seasoned engineer. We’ll have presentations on .NET Core and ASP.NET Core, C#, F#, Azure, Visual Studio, Xamarin, and much more.

Thursday, August 23, 2018

Angular Console–Angular CLI made easy

The guys from Nrwl are introducing the Angular Console; a more approachable way to work with all the features that the Angular CLI already provides.

The Angular Console is available at https://angularconsole.com/ and offers the following set of features:

  • Trivial Code Generation: Leverage the CLI's built in generators for Components, Providers and more, all without having to remember a single command line flag.
  • Run Custom NPM Scripts: As well as the standard `ng` commands, all of your custom npm scripts are automatically detected and allow you to run them with a single click.
  • Discover and Install Extensions: Search through a comprehensive list of official and community schematics to extend the power of the Angular CLI even further, and install them directly in the UI.
  • Build CLI Commands Visually: Using a visual form-driven UI, you get all of the flags and arguments for your chosen command available inline.
  • Integrated Terminal Output: As you compose and execute your commands in a visual way, you will also see your changes reflected in the inline terminal output in real-time.
  • Import Existing Projects: Create new Angular CLI projects from scratch, or import existing ones.

Getting started

  • After installing the Angular Console, you get a new menu item and desktop icon. Running the application will bring you to the Recently Opened Workspaces screen where you can see all projects you were working on before.

image

  • Let’s create a new Angular workspace by clicking on Create. A new wizard is started and you first have to specify 3 things
    • A working folder
    • A workspace name
    • The schematics you want to use

image

  • Click Create to continue. You see the command that gets generated and all output is written to a console:

image

  • After the installation is done, you can immediately trigger the different ng commands like build, serve, …

image

  • Let’s try for example to create a new component by clicking on Generate Component. Now you get a nice UI that allows you to specify all fields. Fill in all required fields and click Generate.

image

  • You also have the option to Generate code based on the available schematics:

image

  • And last but not least you can also directly run all tasks that are found inside your package.json:

image

Enough reasons to give the Angular Console a try…

Wednesday, August 22, 2018

Randomize the order of a list in SQL Server

For a project I had a specific requirement where I needed to show data in a random order. Instead of doing this code, I found an easy solution to do this using raw SQL.

The trick is to add an ORDER BY NewId() to your SQL statement. This is guaranteed to produce a different order every time.

Let’s show a quick example using the good old Northwind database:

When we execute this query, we'll see a different order everytime:

image

Next run:

image

Tuesday, August 21, 2018

ASP.NET–Optimize build performance for solution

After updating to Visual Studio 2018 Update 15.8 I noticed a new menu option in Visual Studio under Build –> ASP.NET Compilation –> Optimize Build Performance for Solution:

image

What does it do?

ASP.NET projects carry a copy of the compiler to be able to compile views at runtime. However, on a developer machine when the copy of the compiler doesn’t match Visual Studio’s copy, your build performance is impacted on the order of 1-3 seconds per incremental build. Enabling this feature will update your project’s copy of the compiler to match Visual Studio’s which should speed up your incremental builds.

When you select the menu option, you’ll get a screen that allows you to select all projects where to apply this change:

image

After you click OK, Visual Studio will replace the Microsoft.Net.Compilers package with a newer Microsoft.CodeDom.Providers.DotNetCompilerPlatform package:

image

Remark: This is applicable to ASP.NET Framework projects only, it does not apply to ASP.NET Core.

Monday, August 20, 2018

Running 1.000.000 containers with Azure Service Fabric

I tried Azure Service Fabric before but didn’t like it so much at that time(about 2 years ago) due to 2 reasons:

  • Really flaky SDK and Visual Studio integration. We had a lot of problems to keep everything up and running when SDK updates where released.
  • Lack of ecosystem around it, especially the dashboard and monitoring options were too limited compared to other solutions

However after watching the following video, I think it’s time I reevaluate my opinion and give Service Fabric a second try:

Friday, August 17, 2018

TFS 2018–Installation failure

Didn’t had my best day yesterday as I was trying to rollout Team Foundation Server 2018 at a customer.

When we tried to start the installation, it failed while installing the .NET Framework 4.7 with a not-so-clear error message “Cannot find object or property”.

clip_image002

Inside the setup log, we found a little more information:

[1930:1564][2018-08-16T09:54:05]i319: Applied execute package: netfxfullredist, result: 0x80092004, restart: None

[1930:1564][2018-08-16T09:54:05]e000: Error 0x80092004: Failed to execute EXE package.

[1AC0:00A8][2018-08-16T09:54:05]i351: Removing cached package: netfxfullredist, from path: C:\ProgramData\Package Cache\76054141A492BA307595250BDA05AD4E0694CDC3\

[1AC0:00A8][2018-08-16T09:54:05]i372: Session end, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{2b3bf919-c7d8-41a9-bff7-734a9750b00c}, resume: None, restart: None, disable resume: No

[1AC0:00A8][2018-08-16T09:54:05]i330: Removed bundle dependency provider: {2b3bf919-c7d8-41a9-bff7-734a9750b00c}

[1AC0:00A8][2018-08-16T09:54:05]i352: Removing cached bundle: {2b3bf919-c7d8-41a9-bff7-734a9750b00c}, from path: C:\ProgramData\Package Cache\{2b3bf919-c7d8-41a9-bff7-734a9750b00c}\

[1AC0:00A8][2018-08-16T09:54:05]i371: Updating session, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{2b3bf919-c7d8-41a9-bff7-734a9750b00c}, resume: None, restart initiated: No, disable resume: No

[1930:1564][2018-08-16T09:54:05]i000: MUX:  Apply Complete: Disk Space Used in bytes for Installation:  MaxAppDrive: 0  MaxSysDrive: 2208534528  AppDrive: 0  SysDrive: 1619668992

[1930:1564][2018-08-16T09:54:05]i000: MUX:  Free Disk Space after install:  SystemDrive C:\ 32985960448 bytes  AppDrive C:\ 32985960448 bytes

[1930:1564][2018-08-16T09:54:05]i000: MUX:  Go to Finished page.

[1930:1564][2018-08-16T09:54:05]i000: MUX:  Watson Bucketting Parameters

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P1 - tfs_trial

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P2 - 16.131.27701.1

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P3 - 16.131.27701

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P4 - Install

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P5 - netfxfullredist

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P6 - Install

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P7 - 0x80092004

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P8 -

[1930:1564][2018-08-16T09:54:05]i000: MUX:  P9 -

[1930:1564][2018-08-16T09:54:05]i399: Apply complete, result: 0x80092004, restart: None, ba requested restart:  No

It is clearly a problem with the installation of the .NET Framework and searching for the error code 0x80092004 gave some results.

Unfortunately every solution we found and tried didn’t solve the problem. Sad smile

To be continued…

Thursday, August 16, 2018

TFS Work Hub no longer works after installing Security Update for SQL Server 2016 SP2 CU(KB4293807)

UPDATE: This is indeed an issue confirmed by Microsoft: https://blogs.msdn.microsoft.com/sqlreleaseservices/issue-with-security-update-for-the-remote-code-execution-vulnerability-in-sql-server-2016-sp2-cu-august-14-2018/. Thanks Michael for sending me the link.

After installing the security update below on our TFS database tier, we could no longer connect to the Work hub in TFS(other features still seemed to work).
image
The error we got on the work page was:
TF246017 Team Foundation Server Could Not Connect to the Database.Verify that the server that is hosting database is operational,and that network problem are not blocking communication with server.
In the windows event logs, we found the following information:
A user request from the session with SPID 61 generated a fatal exception. SQL Server is terminating this session. Contact Product Support Services with the dump produced in the log directory.
I couldn’t find a (better) solution, so we decided to rollback this security update.

Tuesday, August 14, 2018

MassTransit–Tests started to timeout after upgrading to MassTransit 5

On one of my projects, we are using a rather older version of MassTransit(version 3.5). As we needed some of the new features, we thought it was time for an upgrade.

Afbeeldingsresultaat voor masstransit

After updating all our NuGet packages, everything seemed to compile. Unfortunately when we tried to run our unit tests, every test that was using the built-in TestFramework timed out.

Here is the code we were using:

Problem is that the following line keeps waiting for an incoming message:

var orderMessage = await _observer.PostConsumed;

This code worked perfectly before in MassTransit 3.5 but not anymore in MassTransit 5.0.

So what’s changed to cause this behavior?

Starting from MassTransit 4, the in-memory message fabric used by the InMemoryTestFixture behaves the same as RabbitMQ. Previously, every message was published to every endpoint, regardless of whether or not there was a binding for that message type. Now, if there is no binding, the message isn't delivered.

To fix it, we’ll have to add at least one consumer that listens to the specified message type. Here is the altered code:

We override the ConfigureInMemoryReceiveEndpoint method and register a no-op Consumer.

Monday, August 13, 2018

The developer’s guide to Microsoft Azure–August edition

Writing a book about Microsoft Azure would be quite a painful experience for any editor, as the book will be outdated before it reaches the book store.

Anyway, Microsoft announced an updated version of the Developer’s Guide to Microsoft Azure eBook. This updated version includes all the news from Microsoft Build, along with new services and features announced since then.

image

The eBook (PDF available for download) covers the following chapters:

  • Chapter 1: The Developer’s Guide to Azure
  • Chapter 2: Getting started with Azure
  • Chapter 3: Securing your application
  • Chapter 4: Adding intelligence to your application
  • Chapter 5: Working with and understanding IoT
  • Chapter 6: Where and how to deploy your Azure services
  • Chapter 7: Microsoft Azure in Action
  • Chapter 8: Summary and where to go next

Remark: EPUB and Mobi format is also available.

Friday, August 10, 2018

Unable to start process dotnet.exe.

After switching from IIS Express to IIS, I got into trouble when I tried to start the ASP.NET Core web application through Visual Studio.

When hitting F5, instead of building and launching my application, I got the following error message:

Unable to start process C:\Program Files\dotnet\dotnet.exe. The web server request failed with authentication failure 401, Unauthorized.

clip_image002

Luckily sometimes error messages can be meaningful and enabling anonymous authentication in the launchSettings.json indeed solved the problem:

{

  "iisSettings": {

    "windowsAuthentication": false,

    "anonymousAuthentication": true,

    "iisExpress": {

      "applicationUrl": "http://localhost:53125",

      "sslPort": 0

}

  },

  "profiles": {

    "IIS Express": {

      "commandName": "IISExpress",

      "launchBrowser": true,

      "environmentVariables": {

        "ASPNETCORE_ENVIRONMENT": "local"

      }

    }

  }

}

Thursday, August 9, 2018

Fixing DTC issues

As mentioned in a previous post, we were encountering issues when using the Distributed Transaction Coordinator.

By using the DTCTester tool we were able to prove that the problem was indeed related to the DTC settings and not a bug in our code(as if that could happen Winking smile).

Executed: dtctester
DSN:  test
User Name: username
Password: password
tablename= #dtc26446
Creating Temp Table for Testing: #dtc26446
Warning: No Columns in Result Set From Executing: 'create table #dtc26446 (ival int)'
Initializing DTC
DtcGetTransactionManager failed: 8004d01b
DtcGetTransactionManager Call Error: Indicates unable to connect to the transaction manager, or the transaction manager is unavailable.

We were able to solve the issue by adding an exception for msdtc.exe in our firewall:

  • Click on the Windows icon and start typing ‘firewall’. Choose ‘Allow an app through Windows Firewall’.

image

  • The Allowed Apps window is loaded. Click on Change Settings. The Allow another app… button becomes enabled.

image

  • Click on the Allow another app… button and choose Browse… to select an application.

image

  • Search for MSDTC.exe and click on Open.

image

  • Click on Add to close the wizard.

image

  • Let’s try the DTCTester again to see if it works now:

Executed: dtctester
DSN:  test
User Name: username
Password: password
tablename= #dtc27076
Creating Temp Table for Testing: #dtc27076
Warning: No Columns in Result Set From Executing: 'create table #dtc27076 (ival int)'
Initializing DTC
Beginning DTC Transaction
Enlisting Connection in Transaction
Executing SQL Statement in DTC Transaction
Inserting into Temp...insert into #dtc27076 values (1)
Warning: No Columns in Result Set From Executing: 'insert into #dtc27076 values (1) '
Verifying Insert into Temp...select * from #dtc27076 (should be 1): 1
Press enter to commit transaction.

Commiting DTC Transaction
Releasing DTC Interface Pointers
Successfully Released pTransaction Pointer.
Disconnecting from Database and Cleaning up Handles

  • Fixed!

Wednesday, August 8, 2018

Testing Distributed Transaction Coordinator(DTC) settings using DTCTester

After a datacenter move, we had some network changes that caused distributed transactions involving the DTC to fail.

To help me find the root cause of the issue I used the DTC tester tool created by Microsoft. Let me explain how to use it:

Installation

C:\Users\BaWu\Desktop>dtctester
Too few parameters
Usage: dtctester <dsn name> <user name> <password>

  • The tool requires 3 parameters to be used;
    • A data source name(DSN)
    • A user name
    • A password

Creating a DSN

  • Next step is to create a User DSN for the database we want to connect to.
  • Click on your Windows start icon and start typing ‘data sources’

image

  • Select the ODBC Data Sources (32-bit) option. The ODBC Data Sources Administrator tool is loaded.

image

  • On the User DSN tab click Add… This will open the Create New Datasource Wizard.

image

  • Select the SQL Server driver from the list and click Finish. The current wizard is closed and a new Create new Data Source for SQL Server wizard is loaded.

image

  • Type in a Name for the DSN, an (optional) description and most important your SQL Server name / Instance name. Click Next > to continue.

image

  • On the next screen, select the preferred authentication method. In my case I’m using SQL Server authentication so I select the second option. In that case you have to specify a Login ID and Password. Click Next >.

image     

  • Click Next > and Finish to complete the wizard.

Usage

  • Now we can invoke the dtctester tool by providing the DSN name we just created and a SQL server username and password:

C:\Users\BaWu\Desktop>dtctester test username password

  • Here is the result we got back:

Executed: dtctester
DSN:  test
User Name: username
Password: password
tablename= #dtc26446
Creating Temp Table for Testing: #dtc26446
Warning: No Columns in Result Set From Executing: 'create table #dtc26446 (ival int)'
Initializing DTC
DtcGetTransactionManager failed: 8004d01b
DtcGetTransactionManager Call Error: Indicates unable to connect to the transaction manager, or the transaction manager is unavailable.

Tuesday, August 7, 2018

Joining multiple strings together using SQL

In a project we had some rather complex database structure and decided to use raw SQL to query it instead of trying to tweak Entity Framework to remain performant while querying this database.

One of the things we had to do was join multiple fields together into one string field separated by spaces. We originally had the following SQL to achieve this(of course this is a simplified version Smile):

SELECT COALESCE(mad.Street, '')+' '+COALESCE(mad.Number, '')+' '+COALESCE(mad.Addition, '')+' '+COALESCE(mad.PostalCode, '')+' '+COALESCE(mad.City, '') AS Address FROM Suppliers

This rather ugly SQL did its job but was not fun to look at or maintain.

Starting from SQL Server 2017 you have a better alternative available: CONCAT_WS.

From the documentation:

This function returns a string resulting from the concatenation, or joining, of two or more string values in an end-to-end manner. It separates those concatenated string values with the delimiter specified in the first function argument. (CONCAT_WS indicates concatenate with separator.)

Exactly what we need and it allows us to simplify the query above to:

SELECT CONTACT_WS(' ', mad.Street, mad.Number, mad.Addition, mad.PostalCode, mad.City) AS Address FROM Suppliers

Now we still have to convince our DBA to upgrade to SQL Server 2017 Winking smile

Monday, August 6, 2018

TFS 2018–TFVC_RepositoryCodeIndexing Job Failing

Back from Holliday with a sequel for my last post. To quickly summarize our problem, we noticed very long wait times before jobs executed by the TFSJobAgent where executed. Things like triggering builds or releases, sending notifications and emails took hours before they got executed.

Will diving into the TFS administration pages, we identified the TFVC_RepositoryCodeIndexing jobs used for search as the root cause of the issue.

Let’s drill deeper into the problem by looking at a specific job log:

Events (46099) completed with status FailedAndRetry. Event 46099 completed with message 'Patch-PeriodicCatchUp: Code patch operation for Repository Id 96f8a036-54bc-47dc-9c37-08fc23d4a07b failed with error Error code -----> MaxAcceptableFractionOfFailedItemsInCrawlerThresholdBreached Exception message -----> Could not download content of lots of files, hence failing this Job. Failed Items Count: 38, Total Items: 38. Exception type -----> Microsoft.VisualStudio.Services.Search.Common.CrawlerException Exception source -----> Microsoft.VisualStudio.Services.Search.Crawler Stack trace, at Microsoft.VisualStudio.Services.Search.Crawler.Definitions.AbstractCrawler.ThrowIfFailedCrawlItemsBreachThreshold(Int32 failedRecordsCount, Int32 totalItemsToBeCrawled) at Microsoft.VisualStudio.Services.Search.Crawler.Extensions.Code.Tfvc.TfvcPatchHttpCrawler.TfvcCodeTraverser(TfvcPatchIndexCrawlSpec crawlSpec, TfvcHttpClientWrapper tfvcHttpWrapper, CrawlSinkParams crawlSinkParams) at Microsoft.VisualStudio.Services.Search.Crawler.Definitions.AbstractCrawler.Run(IndexingExecutionContext indexingExecutionContext) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.VisualStudio.Services.Search.Crawler.Definitions.AbstractCrawler.Run(IndexingExecutionContext indexingExecutionContext) at Microsoft.VisualStudio.Services.Search.Server.Jobs.Operations.SingleStageCPF.ExecutePipeline(PipelineContext pipelineContext, StringBuilder resultMessage, BookkeepOperation bookkeepOp) at Microsoft.VisualStudio.Services.Search.Server.Jobs.Operations.SingleStageCPF.Run(PipelineContext pipelineContext, StringBuilder resultMessage) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.VisualStudio.Services.Search.Server.Jobs.Operations.SingleStageCPF.Run(PipelineContext pipelineContext, StringBuilder resultMessage) at Microsoft.VisualStudio.Services.Search.Server.Jobs.Operations.Code.RepositoryCodeIndexingOperation.ExecuteCrawlerParserAndFeeder(PipelineContext pipelineContext, StringBuilder resultMessage) at Microsoft.VisualStudio.Services.Search.Server.Jobs.Operations.Code.RepositoryCodePatchOperation.RunOperation(IndexingExecutionContext indexingExecutionContext) [Crawler phase Total items crawled = 0. Time Taken = 15962320ms for Crawl spec type 'Microsoft.VisualStudio.Services.Search.Crawler.CrawlSpecs.TfvcPatchIndexCrawlSpec', Crawler extension 'Microsoft.VisualStudio.Services.Search.Crawler.Extensions.Code.Tfvc.TfvcPatchHttpCrawler'. ]'. OnPremiseTelemetryProperties :{"TFS.Search.Indexing.ParentCorrelationId":"5e7dfff9-485c-42ad-9101-8a258f530276","TFS.Search.Indexing.CorrelationId":"c911ee0a-2001-43fb-987d-96bb4bb8d826","TFS.Search.Indexing.CollectionName":"DefaultCollection","TFS.Search.Indexing.CollectionId":"ce06fba0-ec2f-466a-8569-1e9a13789614","TFS.Search.Indexing.ProjectName":"Ner","TFS.Search.Indexing.ProjectId":"96f8a036-54bc-47dc-9c37-08fc23d4a07b","TFS.Search.Indexing.ProjectIndexingUnitId":"144","TFS.Search.Indexing.RepositoryName":"$/Ner","TFS.Search.Indexing.RepositoryId":"96f8a036-54bc-47dc-9c37-08fc23d4a07b","TFS.Search.Indexing.RepositoryIndexingUnitId":"145","TFS.Search.Indexing.CrawlerServiceFailure":"1","TFS.Search.Indexing.CrawlerServiceFailureReason":"Microsoft.VisualStudio.Services.Search.Common.CrawlerException","TFS.Search.Indexing.CrawlingTime":"15962320","TFS.Search.Indexing.OperationStatus":"Failed","TFS.Search.Indexing.EntityType":"Code","TFS.Search.Indexing.IndexingUnitType":"TFVC_Repository","TFS.Search.Indexing.ChangeType":"Patch","TFS.Search.Indexing.JobExecutionTime":"15962332"}

We were able to see the specific items that failed to index by executing the following query on the collection databases:

select [IndexingUnitId] from [<collection_db>].[Search].[tbl_IndexingUnit] where TFSEntityId like ‘96f8a036-54bc-47dc-9c37-08fc23d4a07b’

select * from [<collection_db>].[Search].[tbl_ItemLevelFailures] where IndexingUnitId = <IndexingUnitId fetched above>

Unfortunately this didn’t bring us any closer to a solution.

In the end we choose the most drastic approach, removed the Search extensions from all collections and removed ElasticSearch itself(https://bartwullems.blogspot.com/2017/11/tfs-2018-remove-elasticsearch.html).

After everything was removed, we reinstalled ElasticSearch, added the Search extensions again and this time everything seemed to work…