With the release of Azure DevOps Server 2022, I thought it would be a good time to finally make the switch to HTTPS for our internal Azure DevOps server. With the idea to minimize downtime, I decided to first introduce the HTTPS endpoint before upgrading the Azure DevOps server. But o boy, what I thought would be an easy task turned out to be quite a journey.
In part 1 I explained the steps that should be done at the server side, in part 2 I continued with an explanation of the changes done on the client side. I thought that the job was done but then I got a phonecall about the build server being offline.
So in this third and final part we have a look at the errors we got on the build server and how we fixed them.
Update build agents
When I logged in on the build server, I noticed that all agents were offline. In the logs I found the following error message:
[2022-12-20 09:48:01Z INFO RSAEncryptedFileKeyManager] Loading RSA key parameters from file D:\b\3\.credentials_rsaparams
[2022-12-20 09:48:01Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2022-12-20 09:48:01Z WARN VisualStudioServices] Authentication failed with status code 401.
Date: Tue, 20 Dec 2022 09:48:00 GMT
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
WWW-Authenticate: Bearer, Basic realm=https://tfs.servername.be/tfs, Negotiate, NTLM
X-TFS-ProcessId: 18369b0e-76da-49a2-82fc-8872e485de03
ActivityId: 3333de4a-972f-4d46-84b1-8003815cd0a0
X-TFS-Session: e82dc2ff-9872-4ec9-93a8-df2d4088a1f8
X-VSS-E2EID: 4278ba7f-f6dc-46a9-9e57-17d7fc8b091b
X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
X-TFS-ServiceError: TF400813%3A%20Resource%20not%20available%20for%20anonymous%20access.%20Client%20authentication%20required.
X-Powered-By: ASP.NET
Lfs-Authenticate: NTLM
X-Content-Type-Options: nosniff
[2022-12-20 09:48:01Z ERR VisualStudioServices] POST request to http://tfs.servername.be:8080/tfs/_apis/oauth2/token failed. HTTP Status: BadRequest
[2022-12-20 09:48:01Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2022-12-20 09:48:01Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2022-12-20 09:48:01Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2022-12-20 09:48:01Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2022-12-20 09:48:01Z ERR MessageListener] Catch exception during create session.
[2022-12-20 09:48:01Z ERR MessageListener] Microsoft.VisualStudio.Services.OAuth.VssOAuthTokenRequestException: The audience of the token is invalid.
at Microsoft.VisualStudio.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
at Microsoft.VisualStudio.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
at Microsoft.VisualStudio.Services.Agent.AgentServer.EstablishVssConnection(Uri serverUrl, VssCredentials credentials, TimeSpan timeout)
at Microsoft.VisualStudio.Services.Agent.AgentServer.ConnectAsync(Uri serverUrl, VssCredentials credentials)
at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.CreateSessionAsync(CancellationToken token)
[2022-12-20 09:48:01Z INFO MessageListener] Retriable exception: The audience of the token is invalid.
[2022-12-20 09:48:01Z INFO MessageListener] Sleeping for 30 seconds before retrying.
[2022-12-20 09:48:31Z INFO MessageListener] Attempt to create session.
[2022-12-20 09:48:31Z INFO MessageListener] Connecting to the Agent Server...
[2022-12-20 09:48:31Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2022-12-20 09:48:31Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2022-12-20 09:48:31Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2022-12-20 09:48:31Z WARN VisualStudioServices] Authentication failed with status code 401.
The build server was still using the old http based URL but the OAuth token was generated for the HTTPS endpoint resulting in an audience error:
The audience of the token is invalid.
One option to fix this is by removing and reinstalling the agent with the updated URL. But it also possible to directly update the configuration files. Let’s take that approach:
- Go to the folder where you have your build agent installed. There you should find a .agent and a .credentials file.
- Open the .agent file and update the serverUrl parameter with the new URL:
- Now open the .credentials file and update both the authorizationServerUrl parameter and the oauthEndpointUrl parameter with the new URL:
- Repeat these steps for all your agents on the build server.
- After you have done these changes, you should restart the windows services that host your agent.
Now the error above should be gone and your build agents should be online again.
Update GIT
I thought the job was done, but then I got a second phonecall, although the agents were available again all GIT based builds failed with the following error message:
Task : Get sources
Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version : 1.0.0
Author : Microsoft
Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Syncing repository: SOFACore (Git)
Prepending Path environment variable with directory containing 'git.exe'.
git version
git version 2.26.2.windows.1
git init "D:\b\2\_work\120\s"
Initialized empty Git repository in D:/b/2/_work/120/s/.git/
git remote add origin https://tfs.servername.be/tfs/DefaultCollection/Framework%20en%20Tooling/_git/SOFACore
git config gc.auto 0
git config --get-all http.https://tfs.servername.be/tfs/DefaultCollection/Framework%20en%20Tooling/_git/SOFACore.extraheader
git config --get-all http.proxy
git -c http.extraheader="AUTHORIZATION: bearer ***" fetch --force --tags --prune --progress --no-recurse-submodules origin
fatal: unable to access 'https://tfs.servername.be/tfs/DefaultCollection/Framework%20en%20Tooling/_git/SOFACore/': SSL certificate problem: unable to get local issuer certificate
##[warning]Git fetch failed with exit code 128, back off 8.539 seconds before retry.
git -c http.extraheader="AUTHORIZATION: bearer ***" fetch --force --tags --prune --progress --no-recurse-submodules origin
fatal: unable to access 'https://tfs.servername.be/tfs/DefaultCollection/Framework%20en%20Tooling/_git/SOFACore/': SSL certificate problem: unable to get local issuer certificate
##[warning]Git fetch failed with exit code 128, back off 3.294 seconds before retry.
git -c http.extraheader="AUTHORIZATION: bearer ***" fetch --force --tags --prune --progress --no-recurse-submodules origin
fatal: unable to access 'https://tfs.servername.be/tfs/DefaultCollection/Framework%20en%20Tooling/_git/SOFACore/': SSL certificate problem: unable to get local issuer certificate
##[error]Git fetch failed with exit code: 128
Finishing: Checkout SOFACore@master
This is the same problem I had on the client machines where Git is using OpenSSL to find and validate the issuer certificate instead of the windows certificate store. This can be fixed in the same way by switching to schannel.
Therefore I introduced a .gitconfig file in the c:\users\<buildserviceaccount> folder with the following content:
That solved the error message above.
Update node.js
I was ready to celebrate my success when I got a third phonecall. Oh no! Some of the other build tasks failed with the following error message:
Err: unable to verify the first certificate
This is a node.js related error. A lot of the build tasks in Azure DevOps are created using node.js. Node.js doesn’t use the windows certificate store either so it has no clue how it could validate the SSL connection.
To fix it, I had to take the following actions:
-
Export the root CA's certificate as a "Base-64 encoded X.506 (.CER)" file. Store it on a location on your build server.
-
Create a .credentials file in the build agent folder(same location as where you can find the .agent and .credentials files).
-
Add the following configuration data:
-
Notice that we point the caCert parameter to the location of the exported root CA file.
- Repeat these steps for all your agents on the build server.
- After you have done these changes, you should restart the windows services that host your agent.
It is also possible to do a clean reinstall of your build agent. In that case add the --sslcacert parameter when executing the .config cmd:
.\config.cmd --sslcacert <caCertsPath>
Finally my phone remained silent. Time to grab a coffee!