Skip to main content

Refactor an Azure DevOps pipeline into multiple stages

While helping a team getting their failing build back up and running I noticed that they were using one big build pipeline that only consists of one stage. This not only made the pipeline more difficult to understand but also makes the build time very long and forces you to rerun the full build if one specific step fails.

Let me walk you through several scenario’s how we logically split this pipeline into multiple stages but before I do that, here is the original YAML pipeline:

In the example above we have a single stage containing all the steps. Time to refactor…

Approach 1 - Build → Test → Publish Stages

Description

This is the most straightforward approach:

Build Stage:

  • NuGet tool installation
  • Package restore
  • Build all solutions
  • Angular npm install and build

Test Stage:

  • Run unit tests (.NET)
  • Run Angular tests

Publish Stage:

  • Publish all applications
  • Publish build artifacts

Advantages

  • Clear separation of concerns: Each stage has a distinct purpose
  • Early failure detection: Build failures stop the pipeline before testing
  • Easy to understand: Follows natural CI/CD flow
  • Good for debugging: Clear failure points
  • Scalable: Easy to add deployment stages later

Disadvantages

  • Sequential execution: No parallelization between major phases
  • Longer total time: Each stage waits for the previous to complete
  • Resource inefficiency: Agents may sit idle between stages

Example

Approach 2 - Technology-Based separation

Description

In this scenario we split the pipeline by technology stack:

Backend Stage:

  • NuGet operations
  • .NET restore and build
  • .NET unit tests
  • .NET application publishing

Frontend Stage:

  • Angular npm install
  • Angular testing
  • Angular build

Packaging Stage:

  • Publish build artifacts

Advantages

  • Team specialization: Backend and frontend teams can own their stages
  • Technology-specific optimizations: Different pools, tools, or configurations per tech stack
  • Parallel execution: Backend and frontend can run simultaneously
  • Clear ownership: Easier to assign responsibility for failures

Disadvantages

  • Potential duplication: May need to repeat setup steps
  • Complex dependencies: Frontend might depend on backend APIs
  • Resource overhead: Multiple agents running simultaneously
  • Integration testing gaps: Harder to test full-stack integration

Example

Approach 3 - Application-Specific stages

Description

In this scenario we create a stage for each major application component:

Foundation Stage:

  • General setup (NuGet, restore, build all solutions)
  • Run unit tests

BOSS.Intern.Web Stage:

  • Publish BOSS.Intern.Web

SMIL2.Extern.API Stage:

  • Publish SMIL2.Extern.API

SNapp2 Stage:

  • Angular operations
  • Publish SNapp2.Extern.Web
  • Publish SNapp2.WorkerService

SmilGo Stage:

  • Publish SmilGo.Extern.Web

Artifacts Stage:

  • Publish all build artifacts

Advantages

  • Independent deployment: Each application can be deployed separately
  • Granular control: Fine-tuned control over each application's pipeline
  • Selective builds: Can trigger builds for specific applications only
  • Microservice-friendly: Aligns well with microservice architecture

Disadvantages

  • Many stages: Can become unwieldy with many applications
  • Shared dependency complexity: Managing shared libraries and dependencies
  • Longer pipeline: More stages mean more overhead
  • Maintenance overhead: More stages to maintain and configure

Example

Approach 4 – Parallel execution

Description

With this scenario, we focus on optimizing for speed with parallel stages:

Foundation Stage:

  • NuGet, restore, build solutions

Parallel Testing Stage:

  • Job 1: .NET unit tests
  • Job 2: Angular tests

Parallel Publishing Stage:

  • Job 1: Web applications (BOSS.Intern.Web, SNapp2.Extern.Web, SmilGo.Extern.Web)
  • Job 2: APIs and Services (SMIL2.Extern.API, SNapp2.WorkerService)

Artifacts Stage:

  • Collect and publish all artifacts

Advantages

  • Fastest execution: Maximum parallelization reduces total pipeline time
  • Resource efficiency: Better utilization of available agents
  • Independent failures: One component failure doesn't block others
  • Scalable: Easy to add more parallel jobs

Disadvantages

  • Complex orchestration: More complex dependency management
  • Resource contention: May require more agents simultaneously
  • Debugging complexity: Multiple parallel failures can be harder to diagnose
  • Agent pool limitations: May exceed available agent capacity

Example

Conclusion

There are probably some other approaches that could work as well. But I would suggest starting with Option 1 (Build → Test → Publish) because it:

  • Follows the natural CI/CD flow
  • Provides clear failure points
  • Is easy to understand and maintain
  • Allows for easy addition of deployment stages later

Each stage can have dependencies on the previous one, and you can add conditions to skip stages based on branch or other criteria. You can also run some stages in parallel where there are no dependencies between them.

More information

Stages in Azure Pipelines - Azure Pipelines | Microsoft Learn

Labels:

Popular posts from this blog

Podman– Command execution failed with exit code 125

After updating WSL on one of the developer machines, Podman failed to work. When we took a look through Podman Desktop, we noticed that Podman had stopped running and returned the following error message: Error: Command execution failed with exit code 125 Here are the steps we tried to fix the issue: We started by running podman info to get some extra details on what could be wrong: >podman info OS: windows/amd64 provider: wsl version: 5.3.1 Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM Error: unable to connect to Podman socket: failed to connect: dial tcp 127.0.0.1:2655: connectex: No connection could be made because the target machine actively refused it. That makes sense as the podman VM was not running. Let’s check the VM: >podman machine list NAME         ...
Read more

Cache stampede: when our cache turned against us

While investigating some performance issues, we ran into an ASP.NET Core API that cached a fairly expensive aggregation query for 60 seconds. Under normal load, that was fine: one request rebuilds the cache, everyone else reads from it. Under peak load, dozens of requests would arrive in that same expiry window, all see a cache miss, and all fire the same expensive query in parallel. The database didn't like that. That was the moment when our caching layer stopped helping and started hurting. A burst of requests comes in at the same time, all miss the cache, and all go hammer the database or the downstream API at once. That's a cache stampede . The cache was supposed to protect our backend, and for a few hundred milliseconds it did the opposite. Why this happens IMemoryCache.GetOrCreate (and its async sibling) looks like it protects you, but it doesn't add any locking on its own. Look at the naive version: public async Task<Report> GetReportAsync(string key) ...
Read more

A complex system designed from scratch never works

A few years ago, I worked as an architect on a big mainframe rewrite. I still count it as one of my failures. Not because the technology was wrong, but because I couldn't convince the management team to simplify the approach. Years later, the organization is still struggling to get the new system up and running. I left the project at the time, because I couldn't put my name behind an approach that would take very long and cost a lot of money without a working system to show for it along the way. Gall’s Law That memory keeps coming back to me, because it's a textbook case of Gall's Law playing out in real life. Gall's Law , from John Gall's Systemantics , states it plainly: A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works, and it cannot be patched to make it work. You have to start over with a simple system that works. What does that mean in practice,...
Read more