The art of simplicity

As part of our build process we push some of our artifacts as NuGet packages to Team Foundation Server. After adding a new build agent, executing the push failed with the following log output: 2019-09-02T13:30:07.1191540Z ##[section]Starting: dotnet push 2019-09-02T13:30:07.1191540Z ============================================================================== 2019-09-02T13:30:07.1191540Z Task         : .NET Core 2019-09-02T13:30:07.1191540Z Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command. For package commands, supports NuGet.org and authenticated feeds like Package Management and MyGet. 2019-09-02T13:30:07.1191540Z Version      : 2.131.0 2019-09-02T13:30:07.1191540Z Author       : Microsoft Corporation 2019-09-02T13:30:07.1191540Z Help         : [More Information]( ...

Yesterday I blogged about a problem that I had with the XAML Build controllers. I reconfigured the /queue what seemed to solve the problem. Unfortunately the problem reappeared after a few hours. Instead of getting a 404 error, this time the details exposed a 500 error code. In the logs I noticed the following error message: WebHost failed to process a request. Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/17729775 Exception: System.ServiceModel.ServiceActivationException: The service '/queue/SampleCollection/Services/v4.0/MessageQueueService2.svc' cannot be activated due to an exception during compilation.  The exception message is: Memory gates checking failed because the free memory (419909632 bytes) is less than 5% of total memory.  As a result, the service will not be available for incoming requests.  To resolve this, either reduce the load on the machine or adjust the value of minFreeMemoryPercentageToActivateServi...

When I started a XAML Build Controller on the build server, I noticed the following behavior: The controller and agents start up and change to the Ready state while the icons went green A few seconds later the icons turn red again, although the controller and agents remain in the same state. On the Build Controller, a details link appear that shows the following error message: TF400324: Team Foundation services are not available from server <Server/CollectionName>. Page not found So what was going on? Everything was caused by a change in IIS I did. Before the TFS server was hosted on https://mytfs.company.be/tfs but they wanted to get rid of the /tfs part. So I removed the tfs application from IIS and moved all services to the root of the site. This worked perfectly for TFS itself but caused issues for the build controllers. Why? Because there are in fact 2 applications that should be hosted: the root TFS webservices now hosted at https://m...

After migrating to Azure DevOps Server 2019, I got stuck when trying to reconfigure a XAML build controller. Instead of replacing an existing build controller I got an error explaining that the build controller I was using had still a build in progress. Time to dig into the Azure DevOps database to found out what build I should cancel… The ‘go to’ table is tbl_BuildQueue. Every build in the queue has a status. Here are all the statuses for tbl_BuildQueue 0 – None 1 – In progress 2 – Queued 4- Postponed 8 – Completed 16 – Cancelled Create an update query that set all ‘In progress’ builds to Canceled: UPDATE tbl_BuildQueue SET [Status]=16 where [Status]=1 If this doesn’t help you can also check the tbl_Build table. Here are all the statuses for tbl_Build: 1 – In progress 2 – Succeeded 4- Partially Succeeded 8 – Failed 16 – Stopped Let’s update the builds here as well: UPDATE tbl_Build SET [BuildStatus]=16 where [...

After renaming our collection, we started to get build errors on our build server for the builds that were using TFS Version control. The error we got was the following: The path ‘c:\build\appname\sources ’is already mapped to workspace ‘BuildServer_123’. When I looked at the details of the workspace mapping using tf workspaces /owner:* I noticed that it was still referring to the old collection instead of the new name. That explained why TFS tried to create the workspace again, resulting in the error message above. To fix the issue I had to clear the local TFS cache on the build server. This can be done by deleting the content of the following folder: C:\Users\{BuildServiceAccount}\AppData\Local\Microsoft\Team Foundation\{VersionNumber}\Cache {BuildServiceAccount} refers to the account that is used to run the build agent and {VersionNumber} refers to the used Azure DevOps version.

For most of my projects we just take a picture from our Retrospective result and that’s it. There is no digital trace which makes it hard to find the things decided. Recently I started using Team Retrospectives, an extension for Azure DevOps. So far I liked the experience, but let’s wait for conclusions in a few sprints… From the extension documentation : Team Retrospectives, a Microsoft Garage project, is an extension for Azure DevOps that provides a first-class experience for retrospectives and general feedback board scenarios. Collect feedback on your project milestones, organize and prioritize, and create and track actionable tasks to help your team improve over time. Optimized for both desktop and mobile. Celebrate the good, improve the bad, and increase productivity with Team Retrospectives.  

Last week a colleague asked me to explain the concept of Pull Requests using our Azure DevOps Server instance. I followed the steps as described here to active a branch policy on their master branch. Then I did a local change, committed it and tried to push it to the master branch, but instead of getting an error stating that I couldn’t push to this branch, it succeeded! I tried it a few times and every time I was able to push no matter what policies I activated. Was it a bug? Or did I do something wrong? Turned out it was neither. Let’s find out what happened: Go to the Code –> Branches section inside Azure DevOps(TFS). Click on the 3 dots … and choose Branch Security. On the Security screen, I clicked on my user account and had a look at the permissions on the right. I had the Exempt from policy enforcement permission enabled. This overrules are policies and explained why I was able to push my changes. Some extra notes from the documentati...

For a specific build task I’m using inside Azure DevOps it was required to allow the build to access the OAuth token to interact with TFS/Azure DevOps. This option is disabled by default and it took me some time to figure out where I could enable it. Here are the (easy once you know it) steps: Go to Pipelines –> Build . Select the build/release pipeline that you want to configure from the list and click on Edit in the right corner On the Edit Build Definition screen, click on the ‘Run on Agent’ section (in my case Agent Job 1) In the Properties , you find the ‘Allow scripts to access the OAuth token’ in the Additional options section

I thought that the first week of the new year would be a good time do some cleaning and decided to remove some old branches from Team Foundation Server Version Control. However deleting one of the branches failed with the following error message: Another user had one of the files locked inside his (private) workspace which prevented me from deleting this branch. As I couldn’t contact this person, I had to find another solution. Remove the lock Removing a lock can be done using the tf undo command: Open a Visual Studio command prompt Enter the following command tf undo /workspace:FRIEND-PC;personname  /collection:http://servername/DefaultCollection $/xxxx/ProjectFolder/ProjectName/filepath/filename.css Here is an example: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise>tf undo /workspace:"Framework en Tooling 169;Bart Wullems" /collection:http://tfs.server.be:8080/tfs/defaultcollection "$/Framework en Tooling/Ma...

Got a call for help from a colleague who couldn’t get an Acceptance release out of the door. Problem was that the automated build responsible for packaging and deploying the Acceptance version failed. He asked me to take a look… Inside the build logs we noticed the following error message: 2018-11-13T06:35:22.4133574Z (CoreCompile target) -> 2018-11-13T06:35:22.4133574Z   CSC : error CS0006: Metadata file 'D:\b\4\agent\_work\_temp\.sonarqube\resources\1\Google.Protobuf.dll' could not be found [D:\b\4\agent\_work\40\s\MTIL.Domain\MTIL.Domain.csproj] 2018-11-13T06:35:22.4133574Z   CSC : error CS0006: Metadata file 'D:\b\4\agent\_work\_temp\.sonarqube\resources\1\SonarAnalyzer.CSharp.dll' could not be found [D:\b\4\agent\_work\40\s\MTIL.Domain\MTIL.Domain.csproj] 2018-11-13T06:35:22.4133574Z   CSC : error CS0006: Metadata file 'D:\b\4\agent\_work\_temp\.sonarqube\resources\1\SonarAnalyzer.dll' could not be found [D:\b\4\agent\_work\40\s\M...

With the release of Team Foundation Server Update 3, security was enhanced by enabling basic authentication between TFS and the Search service. Before there was no security enabled out-of-the-box. This means that when you try to upgrade to Update 3, you are required to provide a new user/password combination: More information: https://blogs.msdn.microsoft.com/devops/2018/09/13/search-msrc-fix-for-tfs-2017-update-3/

2 days ago Microsoft announced Azure DevOps, the new incarnation of VSTS and TFS. Instead of one big mothership, we’ll get 5 smaller vessels that integrate perfectly with each other(of course!) but also with other tools and products. Here are the 5 building blocks: Azure Pipelines CI/CD that works with any language, platform, and cloud. Connect to GitHub or any Git repository and deploy continuously. Learn More > Azure Boards Powerful work tracking with Kanban boards, backlogs, team dashboards, and custom reporting. Learn more > Azure Artifacts Maven, npm, and NuGet package feeds from public and private sources. Learn more > Azure Repos Unlimited cloud-hosted private Git repos for your project. Collaborative pull requests, advanced file management, and more. Learn more > Azure Test Plans All in one planned and exploratory testing solution. Learn more > More information about the Azure DevOps launch can be found on Channel 9; https://channel9.msd...

Didn’t had my best day yesterday as I was trying to rollout Team Foundation Server 2018 at a customer. When we tried to start the installation, it failed while installing the .NET Framework 4.7 with a not-so-clear error message “Cannot find object or property”. Inside the setup log, we found a little more information: [1930:1564][2018-08-16T09:54:05]i319: Applied execute package: netfxfullredist, result: 0x80092004, restart: None [1930:1564][2018-08-16T09:54:05]e000: Error 0x80092004: Failed to execute EXE package. [1AC0:00A8][2018-08-16T09:54:05]i351: Removing cached package: netfxfullredist, from path: C:\ProgramData\Package Cache\76054141A492BA307595250BDA05AD4E0694CDC3\ [1AC0:00A8][2018-08-16T09:54:05]i372: Session end, registration key: SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{2b3bf919-c7d8-41a9-bff7-734a9750b00c}, resume: None, restart: None, disable resume: No [1AC0:00A8][2018-08-16T09:54:05]i330: Removed bundle dependency provider: {2b3b...

UPDATE: This is indeed an issue confirmed by Microsoft:  https://blogs.msdn.microsoft.com/sqlreleaseservices/issue-with-security-update-for-the-remote-code-execution-vulnerability-in-sql-server-2016-sp2-cu-august-14-2018/ . Thanks Michael for sending me the link. After installing the security update below on our TFS database tier, we could no longer connect to the Work hub in TFS(other features still seemed to work). The error we got on the work page was: TF246017 Team Foundation Server Could Not Connect to the Database.Verify that the server that is hosting database is operational,and that network problem are not blocking communication with server. In the windows event logs, we found the following information: A user request from the session with SPID 61 generated a fatal exception. SQL Server is terminating this session. Contact Product Support Services with the dump produced in the log directory. I couldn’t find a (better) solution, so we decided to rollback this secu...

Back from Holliday with a sequel for my last post. To quickly summarize our problem, we noticed very long wait times before jobs executed by the TFSJobAgent where executed. Things like triggering builds or releases, sending notifications and emails took hours before they got executed. Will diving into the TFS administration pages , we identified the TFVC_RepositoryCodeIndexing jobs used for search as the root cause of the issue. Let’s drill deeper into the problem by looking at a specific job log: Events (46099) completed with status FailedAndRetry. Event 46099 completed with message 'Patch-PeriodicCatchUp: Code patch operation for Repository Id 96f8a036-54bc-47dc-9c37-08fc23d4a07b failed with error Error code -----> MaxAcceptableFractionOfFailedItemsInCrawlerThresholdBreached Exception message -----> Could not download content of lots of files, hence failing this Job. Failed Items Count: 38, Total Items: 38. Exception type -----> Mic...

A customer contacted me this week explaining that they noticed very long wait times before a build or release was triggered when using CI. These kind of triggers are scheduled using the TFSJobAgent queue, so I expected that for an unknown reason the queue was saturated with too much work. I asked them to collect some data using the TFS administration pages . Here is snapshot of the data we collected: These charts tell us clearly there is an issue with the TFVC_RepositoryCodeIndexing job responsible for indexing our source code and making them available for search through ElasticSearch. This is a clear indication that there is something wrong in ElasticSearch itself. Time to dig into the ElasticSearch logs…but we’ll leave that for another post.

After upgrading to TFS 2018 and reconfiguring our build servers, some of the TFS Version Control based builds started to fail with the following error message: Exit code 100 returned from process: file name ‘tf’, arguments ‘vc workspace /new /location:local’ The line above we noticed the following informational message: The path {pathname} is already mapped in workspace {workspacename} During the migration we switched URI’s which seemed to confuse TFS. To solve it, we removed all existing workspaces for our build account: tf vc workspaces /collection:https://tfs/DefaultCollection /remove:* We also had to delete the TFS client cache, which can be found here: "%localappdata%\Microsoft\Team Foundation\7.0\Cache\"

After upgrading to a new TFS server and going full HTTPS, our git based builds started to fail with the following error message: 2018-06-12T21:28:45.8463514Z ##[command]git init "D:\Builds\dev-agent-1\_work\5\s" 2018-06-12T21:28:45.9154418Z Initialized empty Git repository in D:/Builds/dev-agent-1/_work/5/s/.git/ 2018-06-12T21:28:45.9219662Z ##[command]git remote add origin https://tfs/DefaultCollection/_git/Sample 2018-06-12T21:28:45.9821910Z ##[command]git config gc.auto 0 2018-06-12T21:28:46.0407163Z ##[command]git config --get-all http.https://tfs/DefaultCollection/_git/Sample.extraheader 2018-06-12T21:28:46.0994413Z ##[command]git config --get-all http.proxy 2018-06-12T21:28:46.1508406Z ##[command]git -c http.extraheader="AUTHORIZATION: bearer ********" fetch --tags --prune --progress --no-recurse-submodules origin 2018-06-12T21:28:46.3636860Z fatal: unable to access 'https://tfs/DefaultCollection/_git/Sample/': SSL cert...

After replacing one of our build servers, we got the following error when trying to execute a custom XAML build Workflow activity: System.InvalidCastException: Unable to cast COM object of type 'System.__ComObject' to interface type 'Microsoft.Web.Administration.Interop.IAppHostWritableAdminManager'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{FA7660F6-7B3F-4237-A8BF-ED0AD0DCBBD9}' failed due to the following error: Interface not registered (Exception from HRESULT: 0x80040155). at Microsoft.Web.Administration.ConfigurationManager.CreateAdminManager[TClass,TInterface](WebConfigurationMap webConfigMap, Boolean isAdminConfig) at Microsoft.Web.Administration.ConfigurationManager.CreateConfiguration(WebConfigurationMap configMap, String configPathToEdit, Boolean isAdminConfig) at Microsoft.Web.Administration.ConfigurationManager.GetConfiguration(String rawConfigurationPath, String cacheKey, ...

Microsoft created a consolidated resource site with all kinds of information related to DevOps. It not only contains learning material about DevOps practices but also about Agile Methods, Git version control. You can also see how Microsoft is applying DevOps for their own products and assess you own DevOps maturity.