Azure DevOps

Randy PagelsAzure Specialist - Application Development

US Great Lakes Region

How Microsoft Engineering uses Azure DevOps

MS Engineering uses 1ES to build Azure DevOps, Windows 10, Visual Studio, Bing…etc.…using Azure DevOps

…building 146,000 times per day…deploying 82,000 times per day

… for millions of users + most of Microsoft.

Microsoft’s DevOps Tooling – enhanced by GitHub

Security

Package Registry

Actions

The journey so far

Sprint 1August 2010

Sprint 29June 2012

VSTS Preview

Sprint 67June 2014

1ES

Sprint 102May 2017

Windows on Git Azure Pipelines

Sprint 140Sep 2018

Sprint 162Nov 2019

April 2015

VS CodeJuly 2014

Microsoft on GitHub

Nov 2016

Join Linux

FoundationOct 2018

GitHub

AcquiredMarch 2010

400 Open Source

Projects

8.8k open source projects

25k employees contributing

Before Azure DevOps

Microsoft Confidential

One Engineering System (”1ES”)

Microsoft Confidential

One Engineering System with Azure DevOps

1ES Growth

0

20,000

40,000

60,000

80,000

100,000

Engineer Users

Non-engineering Users

DevOps at Microsoft

Data: Internal Microsoft engineering system activity, November 2019

442kPull Requests per

month

2.4m Private Git commits per

month

82,000Deployments per day

4.6m Builds per month

155pb Build artifacts managed

8.8kOpen Source Repos

24kEmployees contributing

to open source

28kWork items created/day

500kWork items updated/day

Azure DevOps is the toolchain of choice for Microsoft engineering with over 110,000 active internal users

https://aka.ms/DevOpsAtMicrosoft➔

Azure Networking

One Engineering System with Azure DevOps

Microsoft Confidential

Microsoft Confidential

Software Products

500GB+Source code

6000+Repos

Microsoft Confidential

7,300 Developers making code changes

11,000 Topic branches

367,000 Commits

33,000 Pull requests

9,700 Branch Integrations

10 commits per minute

1,100 pull requests per day

On Average, Each Month

For 1 Day of Automated Lab Testing

Customer Obsessed

Production First Mindset

Team Autonomy + Enterprise Alignment

Shift Left Quality

Infrastructure as Flexible Resource

Customer Obsessed

Production First Mindset

Team Autonomy + Enterprise Alignment

Shift Left Quality

Infrastructure as Flexible Resource

Quantitively & Qualitatively

Original estimate

Completed hours

Lines of Code

Team capacity

Team burndown

Team story point velocity

# of bugs found

% code coverage

Things we don’t watchAcquisition

Engagement

Satisfaction

Churn

Feature Usage

Customer Usage

Time to Detect

Time to Communicate

Time to Mitigate

Which customers affected

Incident Prevention Items

Aging Prevention Items

SLA per Customer

Customer Support Metrics

Live Site Health

Time to Build

Time to Test

Time to Deploy

Time to Improve

Failed and flaky automation

Pipeline Velocity

Customer Obsessed

Production First Mindset

Team Autonomy + Enterprise Alignment

Shift Left Quality

Infrastructure as Flexible Resource

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

Assume Breach

Initially double-blind test

Over time, eliminated blue team

Our defenders need to be our defenders

vs.

Shifted left to prevent top risks

Credential theft

Secret leakage

OSS vulnerabilities

Started with war games to the learn attacks and practice response

Customer Obsessed

Production First Mindset

Team Autonomy + Enterprise Alignment

Shift Left Quality

Infrastructure as Flexible Resource

Alignment

Autonomy

“Let’s try to give our teams three things…. Autonomy, Mastery, Purpose”

Teams

Physical team rooms

Cross discipline

10-12 people

Self managing

Clear charter and goals

Intact for 12-18 months

Own features in production

Own deployment of features

UI

API

Data

UI

API

Data

Employee choice, not manager driven

Typically <20% change, but 100% get to make a choice

Cross-pollinate talent and micro-culture

Sticky Note Exercise - Self Forming Teams

Sprint3 weeks

1

Plan3 sprints

3

Season6 months

6

Strategy12 months

Teams are responsible for the detail

Leadership is responsible

for driving the big picture

3 weeks

SpringFallSpring Fall

Customer Obsessed

Production First Mindset

Team Autonomy + Enterprise Alignment

Shift Left Quality

Infrastructure as Flexible Resource

Managing the pipeline:

go fastHow do you

and

notbreak things?

engineers on

your team# 5 ?x =

Rule: If your bug count exceeds your bug cap… stop working

on new features until you’re back under the cap.

5 50x =10

Composing Isolation Mechanisms

Old way of branching

Fixes Hot fixes

Release Service Pack

Integration/Main

Team A

Team B

Team C

Feature 1

Feature 2

Release

Using Trunk Based Development to avoid Merge Debt

Release Flow

Your aim won’t be perfect.

Control the blast radius.

1

Customer Obsessed

Production First Mindset

Team Autonomy + Enterprise Alignment

Shift Left Quality

Infrastructure as Flexible Resource

Shared abstraction layer

Single master branch, multiple release branches

Update 1

Update 2

Update N

Azure DevOps (SaaS)

Azure DevOps Server

One code base with multiple delivery streams

After• 3-week sprints

• Vertical teams

• Team rooms

• Continual Planning & Learning

• PM & Engineering

• Continual customer engagement

• Everyone in master

• 8-12 person teams

• Publicly shared roadmap

• Debt paid as incurred

• Mockups in PPT

• Inner source

• Flattened organization hierarchy

• User satisfaction determines success

• Features shipped every sprint

Before• 4-6 month milestones

• Horizontal teams

• Personal offices

• Long planning cycles

• PM, Dev, Test

• Yearly customer engagement

• Feature branches

• 20+ person teams

• Secret roadmap

• Bug debt accumulated

• 100 page spec documents

• Private repositories

• Deep organizational hierarchy

• Success is a measure of install numbers

• Features shipped once a year