let's make the pain visible!

Janelle Kleinopenmastery.org @janellekz

Let’s Make the PAIN Visible!

Tour Speaker

http://@openmastery

What’s the Problem?

source: Domics - “What’s a Shrug?”

Every few years we rewrite our software…

Start%Over%

Unmaintainable%So0ware%

Across the Industry

We Start with the Best of Intentions

High Quality Code

Low Technical Debt

Easy to Maintain

Good Code Coverage

Then This Happens!

Stages of Escalating Project Risk

Product Owner: “We’ve got more important things to do.”

Deferring(Problems(

Deferring(Problems(

Painful(Releases(

Manager: “Good job everyone! Keep up that great work ethic!”


Deferring(Problems(

Painful(Releases(

Thrashing)

Manager: “We need to go faster! Let’s hire more developers.”


Deferring(Problems(

Painful(Releases(

Thrashing) Project(Meltdown(

Developer: “I give up. I don’t care anymore if the project fails.”


RESET

“A description of the goal is not a strategy.”

-- Richard P. Rumelt

What’s wrong with our current strategy?

Our “Strategy” for Success

High Quality Code

Low Technical Debt

Easy to Maintain

Good Code Coverage

RESET“A good strategy is a specific and coherent response to—and approach for overcoming—the obstacles to progress.”

-- Richard P. Rumelt

The problem is we don’t have a strategy...

What are the biggest obstacles that prevent us from breaking the cycle?

Start%Over%

Unmaintainable%So0ware%

Why Can’t We Break the Cycle?

Consulting with Failing Projects

Engineers: “We’re going to CRASH!”

Manager: “What do I do? We can’t miss these deadlines.”

Obstacle #1: Managers and Developers might as well be

speaking different languages.

Obstacle #2: We spend tons of time working on improvements that

don’t actually make much difference.

The ROOT CAUSE of BOTH PROBLEMS:

Lack of Visibility

Let’s Make the PAIN Visible!

Great Team Disciplined with Best Practices Constantly Working on Improvements+

Project FAILURE

About 8 years ago…

Technical Debt Mistakes

I thought the main obstacle was Technical Debt

?Most of our mistakes were in the

most well-written parts of the code.

Mistakes

We made significantly more mistakes in code that we didn’t write ourselves.

Lower Familiarity

More Mistakes=

There had to be more to the story...

Complex(So*ware(

PAIN

This is what I knew...

What made development feel painful?

Unexpected Behavior

Problem Resolved

Measure Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

The amount of PAIN was caused by…

Likeliness(of((Unexpected(Behavior(

Cost(to(Troubleshoot(and(Repair(

High(Frequency(Low(Impact(

Low(Frequency(Low(Impact(

Low(Frequency(High(Impact(

PAIN(

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

Most of the pain was caused by human factors.

What causes PAIN?




Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?


Semantic Mistakes





Copy-Edit Mistakes



False Alarm




Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

Semantic Mistakes





Copy-Edit Mistakes



False Alarm





Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

PAIN is a consequence of how we interact with the code.

Semantic Mistakes





Copy-Edit Mistakes



False Alarm

The Consequences of Problem-Breakdown

7:01

Iterative Validation with Unit Tests7:010:00

14:230:00

Skipping Tests and Validating at the End

7:01

Iterative Validation with Unit Tests7:010:00

14:230:00

Skipping Tests and Validating at the End

Urgency Leads to High-Risk Decisions

If I make no mistakes I save ~2 hours.

If I make several mistakes I lose ~8 hours.

The Consequences of Problem-Breakdown

Product(Requirements( So1ware(Task(

Possible(Decision(Paths(

So1ware(

Software is a Reflection of our Decisions


High%Risk%Decision%Habits%

So1ware((and(all(our(problems)(

(

RESET





(

Technical Debt is just a Symptom of the Problem


PAIN occurs during the process of understanding and extending the software

Complex(So*ware(

PAIN

Not the Code.

Optimize “Idea Flow”

My team spent tons of time working on improvements that didn’t make much difference.

We had tons of automation, but the automation didn’t catch our bugs.

My team spent tons of time working on improvements that didn’t make much difference.

We had well-modularized code,

but it was still extremely time-consuming to troubleshoot defects.

The hard part isn’t solving the problems it’s identifying the right problems to solve.

“What are the specific problems that are causing the team’s pain?”

measures the time spent on:

Idea Flow

x

Troubleshooting

x

Learning

x

Rework

Quality Risk Familiarity Risk Assumption Risk

Why Measure These Things?

x

Troubleshooting

x

Learning

x

Rework


Idea Flow

The Rhythm of Software Development

Write a little code.

Work out the kinks.


Work out the kinks.


Work out the kinks.

The Rhythm of Software Development

Conflict Confirm

Rework'

Learn'

Validate(

Modify'

Progress Loop Conflict Loop

Troubleshoot'

Confirmed Expectations

Our brain works like a giant prediction engine…

Violated Expectations

Our brain works like a giant prediction engine…

Conflict Confirm

Rework'

Learn'

Validate(

Modify'


Troubleshoot'

The Rhythm of Idea Flow

Write a little code. Work out the kinks.

Conflict Confirm

Rework'

Learn'

Validate(

Modify'


Troubleshoot'

The Rhythm of Idea Flow

“Friction” in Idea Flow

Conflict Confirm

Rework'

Learn'

Validate(

Modify'


Troubleshoot'

Idea Flow Mapping Tools

(Open Source, Supported GA ~June 2016)github.com/ideaflow/tools

http://ideaflow.org/tools

Typical Idea Flow Maps

Single Problem

Multi-Problem

Reading Visual Indicators in Idea Flow Maps

Le#$Atrium$

Le#$Ventricle$

Right$Ventricle$

Right$Atrium$

What’s$causing$this$pa7ern?$

Similar to how an EKG helps doctors diagnose heart problems...

...Idea Flow Maps help developers diagnose software problems.

Problem-Solving Machine

Reading Visual Indicators in Idea Flow Maps

= Solution Strategy

The “Heart” of Software Development(the problem-solving machine)

= Solution Strategy


Choose a general strategy

= Solution Strategy


Understand the system

= Solution Strategy


Code & work out the kinks

= Solution Strategy


Back to the drawing board

The type of friction we experience depends on where the disruptions are in the problem-solving process

Trial and Error

The Friction is Here

Troubleshooting

Progress

Learning

Rework

Struggling with Understanding the Code


Troubleshooting

Progress

Learning

Rework

Struggling with Feedback


Troubleshooting

Progress

Learning

Rework

Every task is a problem-solving task.

The Ten Pains of Software Development

Troubleshooting

Progress

Learning

Rework


Design-Fit Pain - When the new feature doesn’t fit well into the existing design.


Requirements Pain - Bad assumptions about what functionality to build


Collaboration Pain - Challenges collaborating with other developers on the team.


Modeling Pain - When it’s difficult to build a conceptual model of the existing software.


Cognitive Pain - Challenges with complexity and intense thinking


Alarm Pain - Challenges with false alarms and test maintenance


Experiment Pain - Challenges with getting feedback by running experiments


Execution Pain - When changing the code is highly mistake-prone


An amplifying effect on other problems

100 hours

50 hours

Troubleshooting

Learning

Rework

Why categorize problems this way?

Focused Effort = Visible Improvement

The hard part isn’t solving the problems it’s identifying the right problems to solve.

Visibility Changes Everything!!

Troubleshooting

Progress

Learning

7:070:00

0:00 19:52

12 year old project after all original developers left.

Case Study: Huge Mess with Great Team

70-90% of dev capacity on “friction”

The Team’s Improvement Focus: Increasing unit test coverage by 5%


Tags: #problemtag

Tag the Examples by Type

“What seems to be our biggest cause of pain?”

Add up the Friction by Tag

#ReportingEngine

#Hibernate

#MergeHell


1. Test Data Generation

2. Merging Problems

3. Repairing False Alarms

1000 hours/month

The Biggest Problem: ~700 hours/month generating test data

“Experiment Pain”

16:10

What’s causing all the Experiment Pain?

Experiment Time

Our Perception of Time was WAY OFF

Setup Experiment Analyze Results & Decide Next Experiment

Execute

waiting - time goes slow

doing - time zooms by

Experiment*Pain*

!Numerous!Feedback!Loops!

Itera&ve(Narrowing(

Ambiguous(Clue(

Non6determinis&c(

Lots(of(Test(Cases(

Lots(of(Bugs(

Bad(Input(Assump&on(

Uncertainty(

Difficult(to(Interpret(

Experiment*Pain*

!Slow!Feedback!Loops!

Long%Execu+on%

Noisy%Output%

Limited%Output%

Complex%Behavior%

Non<determinis+c%Behavior%

Lots%of%Code%Changes%

Difficult%to%Interpret%

Test%Input%Genera+on%

Environment%Setup%

Environment%Cleanup%

Have%to%significantly%modify%a%test%

Debugger%

Breakdown the Causes of Pain into Patterns

“Experiment Pain”

16:10

Optimizing for Human Cycles instead of Execution

Distill Lessons Learned into “Decision Principles”

Answers two questions

How do I evaluate my situation?

What should I optimize for?

Trade-off Decisions

The Code Sandwich Principle

The thickness of the sandwich increases

troubleshooting difficulty

Behavior Complexity

Observability

Ease of Manipulation

Optimize for low diagnostic difficulty.

“Alarm Pain”

Six hours of troubleshooting and repairing tests when nothing is broken.

The Waxy Coating Principle

Optimize the signal to noise ratio.

Software (before)

Software (after)

Tests are like a waxy coating poured over the code.

Friction in the BRAND NEW microservices code…

27:15

“Why is there so much friction?”

This is Painful!

Application

Usability Problems

3rd Party Dependencies

Upstream Components

Usability problems cascade through dependencies

90% 10%This is Painful!

Application

Usability Problems

3rd Party Dependencies

Upstream Components

The Complexity is in the Supply Chain

The Challenge of our Industry: Growing Complexity

Cost &

Risk

Complexity

Cumulative Complexity in Software Development

Human Limitations

Cost &

Risk

Complexity

Human Limitations

Cumulative Complexity in Shared SW Supply Chain (i.e. shared dumping ground of crappy software)

Bad software has become the new environment pollution.

The Challenge of our Industry: Growing Complexity

Make the PAIN Visible!

What Can We Do About It?

“The Idea Flow Factory” (supply chain model)

Optimize the Rate of Idea Flow

Across the Organization (or the Industry)

18 months after a Micro-Services/Continuous Delivery rewrite.

Troubleshooting

Progress

Learning40-60% of dev capacity on “friction”

0:00 28:15

12:230:00

Case Study: From Monolith to Microservices

They had this problem:

Long-Term Pain

0%

100%

Release 1 Release 2 Release 3

Troubleshooting

Progress

Learning

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

(extrapolated from samples)

0%

100%



Figure out what to do Learning is front-loaded

Troubleshooting

Progress

Learning

Long-Term Pain

0%

100%



Rush Before the Deadline Validation is Deferred

Troubleshooting

Progress

Learning

Long-Term Pain

0%

100%



Pain Builds Baseline friction keeps rising

Troubleshooting

Progress

Learning

Long-Term Pain

0%

100%



Chaos Reigns Unpredictable work stops

fitting in the timebox

Troubleshooting

Progress

Learning

Long-Term Pain

Developers tried to explain to management: “Technical debt is building up in the code!”

Managers: “We’re already behind schedule!!”

PAIN

Eventually the problems got so bad, they couldn’t be ignored.

Long-Term Pain

Builds were breaking

Releases were painful

Productivity slowing to a crawl

Begging for time

The Team’s Focus: Reducing technical debt in the micro-services code + test automation

The Biggest Problem: Integration problems across teams leading to environment down time.

1000 hours/month!!Cost:

Case Study: From Monolith to Microservices

The cost of bad architecture in the microservices world

is EXTREMELY HIGH.

Visibility gives us a way to manage long-term technical risk.

Risk is the Bridge Language Between Managers and Developers

Quality Risk Familiarity Risk

0%

100%



x

Troubleshooting

x

Learning

x

Rework


Visibility gives us a way to manage long-term technical risk.

Quality Risk (Troubleshooting)

Likelihood)of))Unexpected)Behavior)

Cost)to)Troubleshoot)and)Repair)

High)Frequency)Low)Impact)

Low)Frequency)Low)Impact)

Low)Frequency)High)Impact)

PAIN)

Familiarity Risk (Learning)

Likelihood)of))working)with)Unfamiliar)

Code)

Cost)to)Learn)

High)Frequency)Easy)to)Learn)

Low)Frequency)Easy)to)Learn)

Low)Frequency)Hard)to)Learn)

PAIN)

Assumption Risk (Rework)

Likelihood)of))making)a))

Bad)Assump4on)

Cost)to)Correct)Decisions)

High)Uncertainty)Low)Delay)

Low)Uncertainty)Low)Delay)

Low)Uncertainty)High)Delay)

PAIN)

How much work does it take to mop the floor?

How much work does it take to complete a software task?

= Side-Effects from Ignoring the Risk

Trade-off Decisions

Writing Unit Tests Troubleshooting MistakesDirect Cost Indirect Costs

or

We make high-risk decisions because the indirect costs are hard to quantify.

Likeliness of Event

Potential Impactx=

Risk

Troubleshooting Learning Rework

Side-Effects of High-Risk Decisions

Ignore Tango(risk not obvious)

Escala&ng)Risk)

Time)Pressure)

Defer)Problems)

Increase)Risk)&)Impact)

of)Extra)Work)

More)Frequent)Bugs)and)Higher)

Task)Effort)

IgnoreRisk

Cycle of Chaos

TangosMultiply

Escala&ng)Risk)

Time)Pressure)

Defer)Problems)


of)Extra)Work)


Task)Effort)

IgnoreRisk

Cycle of Chaos

More MoppingWork

Escala&ng)Risk)

Time)Pressure)

Defer)Problems)


of)Extra)Work)


Task)Effort)

IgnoreRisk

Cycle of Chaos

Further Behind on Deliverables

Escala&ng)Risk)

Time)Pressure)

Defer)Problems)


of)Extra)Work)


Task)Effort)

Productivity plummets because of the overwhelming side-effects

IgnoreRisk

I can’t go any faster!

Cycle of Chaos

Fewer%Problems%to%

Fix%

Stop%%and%Think%

Mi8gate%the%Risk%

Increased%Produc8vity%

Safety'Identify Risk

Cycle of Safety

Fewer%Problems%to%

Fix%

Stop%%and%Think%

Mi8gate%the%Risk%


Safety'Put Tango on a Leash

Cycle of Safety

Fewer%Problems%to%

Fix%

Stop%%and%Think%

Mi8gate%the%Risk%


Safety'

Predictable Completion

Cycle of Safety

Fewer%Problems%to%

Fix%

Stop%%and%Think%

Mi8gate%the%Risk%


Safety'

We improve productivity by improving control not by trying to go faster

Slack Time

Cycle of Safety

Troubleshooting Learning Rework

Optimize Idea Flow by systematically mitigating risk.

Control = Risk Mitigation

Quality Risk Familiarity Risk

0%

100%



“How do we control the risk?”

Risk is the Bridge Language Between Managers and Developers

Process Control in Manufacturing

This is “Out of Control”

Lower Variability = Better Control

“Pain Control” in Software Development

Average Pain per Incident

This is Control

Target

Control Limit

This is “Out of Control”

16:10

This is “Control”

Idea Flow Learning Framework

Input: Decision Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

1. Visibility

2. Clarity

3. Awareness

F ocus!

Output: “Friction” in Idea Flow

Improve Quality of Decisions




1. Visibility

2. Clarity

3. Awareness

F ocus!


Target - The direction of “better”






1. Visibility

2. Clarity

3. Awareness

F ocus!


Input - The constraints that limit our short-term choices…





1. Visibility

2. Clarity

3. Awareness

F ocus!


Output - The pain signal we’re trying to improve





1. Visibility

2. Clarity

3. Awareness

F ocus!


Focus on the biggest pain…

F ocus!





1. Visibility

2. Clarity

3. Awareness

F ocus!


1. Visibility - Identify the specific patterns.

1. Visibility





1. Visibility

2. Clarity

3. Awareness

F ocus!


2. Clarity - Understand cause and effect.

2. Clarity





1. Visibility

2. Clarity

3. Awareness

F ocus!


3. Awareness

3. Awareness - Learn strategies to avoid the pain.





1. Visibility

2. Clarity

3. Awareness

F ocus!




“The Idea Flow Factory” (supply chain model)

Optimize the Rate of Idea Flow

Across the Software Supply Chain

Brevity?

What should we be optimizing for?

Modularity?

Code Coverage?

What does “better” really mean?




(

Technical Debt is just a Symptom of the Problem


Data-Driven Software Mastery




1. Visibility

2. Clarity

3. Awareness

F ocus!




Idea Flow

x

Troubleshooting

x

Learning

x

Rework


Idea Flow gives us a universal definition of effective practice.

x

Troubleshooting

x

Learning

x

Rework




What are the consequences of our decisions?


Idea Flow gives us a universal language for sharing our experiences.


What strategies tend to minimize the friction in Idea Flow?

Idea Flow gives us a universal language for sharing our experiences.

And in what contexts?

Idea Flow gives us the capability to learn together as an industry.


We have an opportunity to learn together like we’ve never had before.

Idea Flow gives us the capability to learn together as an industry.

Why? Because this REALLY Sucks.

Seeing your creation get stomped on by organizational dysfunction is an emotionally damaging experience.

Let’s take responsibility,

learn how to get there,

then teach the industry how to succeed.

Collaboration Platform

IFM Tools Team Mastery Platform

TeamJoe Sally Mark Eric

Industry Collaboration Platform

Anonymized Data

Integrated #HashTag Glossary

ProjectTiger

ProjectBear

(REST)

Industry Collaborative Learning Network

Change Starts with Making the PAIN Visible!


http://@openmastery


Check out openmastery.org for details.

Read my Book.

Think About It.

FREE with Reading GroupBuy It

How to Measure the PAIN in Software Development

Janelle Klein

http://@openmastery

http://openmastery.org

Why read Idea Flow? Because Rene and Matt said so!

“If you just want to read ONE book about software engineering, this year, read this one: leanpub.com/ideaflow #ideaflow”

Rene Gröschke, Gradle Inc.

“If you don't know about @janellekz's work on Idea Flow Learning, you're missing out. leanpub.com/ideaflow”

Matt Stine, NFJS Tour Speaker

@breskeby

@mstine

Check out openmastery.org for details.

https://t.co/ZLq2lXRpeY

https://twitter.com/search?q=%23ideaflow

https://twitter.com/janellekz/

http://leanpub.com/ideaflow

http://openmastery.org