- Jon Smart
Agility: Build the right thing
From discrete Output to continuous Outcomes
Milestones mark miles. They are made of stone. They are hard to move and are no longer valid once moved. They don’t exhibit agility, nor do they need to. They resemble tombstones and are often referred to as ‘dead-lines’, or ‘drop dead’ dates.
They don’t convey any expectations on why you want to get to where you’re going, how you might get there, how else you might get there, if you’ll make it there at all (the level of safety en-route), the quality and experience of your journey nor if it’s even the right destination, given your intent.
They are one dimensional. The distance between two fixed points along one fixed route. Originating 2,300 years ago in the Roman Empire, milestones were set more than 2 feet into the ground, stood 5 feet tall and weighed more than 2 tons. They measured 1,000 paces. They were immovable.
You don’t know when you’re going to reach a milestone. The last 20% of the journey might take 80% of the time. There might be roadworks, an accident or a flooded river crossing. You only know you’ve reached a milestone when you reach a milestone. And even then, given your intent, you might find that it’s not the optimal place to be.
Are milestones a good mental construct to use for unique change with a rapidly changing terrain? I think not.
This is the fourth post in a series sharing a number of observed antipatterns and corresponding patterns on the topic of organisational agility, for organisations who want to deliver Better Value Sooner Safer & Happier.
We are in a new Deployment Period in a 50 year cycle, in the Age of Digital, as well articulated by Carlota Perez in ‘Technological Revolutions and Financial Capital’.
Today, for complex knowledge work, many large organisations are still applying a management model from the early 1900s optimised for manual labour in factories.
With new means of production and consumption, with on-demand compute, information and communication power which is in the cloud and in our pockets, if traditional organisations want to survive and thrive, want to keep up with competitors and challengers, there is a need to adopt new ways of working to maximise the outcomes from the new means of production.
Survival is not mandatory
This is of course optional, as survival is not mandatory, a path that UK department store Debenhams has taken. Debenhams, founded in 1778, went into administration 241 years later, on 9th April 2019 with 165 stores and 25,000 employees. To quote Richard Lim, chief executive of industry analysts Retail Economics,
“Put simply, the business has been outmanoeuvred by more nimble competitors, failed to embrace change and was left with a tiring proposition. The industry is evolving fast and it paid the ultimate price.” (source).
Time will tell if Debenhams is following in the footsteps of Toys R Us, Maplins, BHS, House of Fraser, HMV, Kodak, GM, Blockbuster, Nokia and Sears as firms who have chosen, implicitly, to not be a going concern either at all or in what was their core market.
This post focuses specifically on how organisations choose (implicitly or explicitly) what work to do and how it is approached, in the context of product development, as organisational agility increases or needs to increase.
A scenario often observed is that processes upstream of product development teams become the primary constraint to valuable flow, especially as product development teams increase their agility.
How do large, traditional, organisations make sure that teams are working on the best possible things as agility increases? Done badly, and anything can be done badly, it’s fragile rather than agile. As empowerment and autonomy are increased, the system of work could become chaotic and disconnected. Or, capital ‘A’ Agile is imposed with a culture of fear, with all change still having drop-dead predictive output milestones, waterfall dressed up as agile, which doesn’t maximise the benefits of agility.
First we take a look at four antipatterns: (1) Local Optimisation & the Urgency Paradox (2) Milestone Driven Predicted Solutions (3) Headless Chickens and (4) Start Starting. Then we look at the corresponding pattern per antipattern, namely: (1) End to End Flow (2) Outcome Hypotheses (3) Strategy Alignment and (4) Stop Starting, Start Finishing.
These antipatterns and patterns are based on lessons learnt through doing, through learning, through running experiments (the only failed experiment is one which generates no learning. Even then that itself is a learning). This includes being a servant leader on better ways of working, applying agile, lean, DevOps, systems thinking and so on, across a large (80,000 people), old (300+ years old), global, not-born-agile, highly regulated enterprise. I have been delivering change with an agile mindset, principles and practices since the early 1990s, about a decade prior to the Agile Manifesto, when the term ‘lightweight processes’ was used. The antipatterns and patterns are also based on learnings from many horses (rather than unicorns) on similar journeys.
Antipattern 1: Local Optimisation & the Urgency Paradox
The following scenario is a common one I’ve experienced, especially with business areas early in the journey to delivering Better Value Sooner Safer and Happier.
I had met with a software development team who were rightly proud of their progress with increasing their agility. The Team Outcome Lead (aka Scrum Master) showed me their improvement of Cycle Time, from the time work is pulled off the backlog to development done. It was an impressive improvement with the 85th percentile Cycle Time approximately halved from what it used to be 12 months previously. During this time the team had become multidisciplinary, limiting Work In Progress with more smaller stories, which had reduced complexity per change, de-risked delivery and led to faster feedback and learning.
However, something wasn’t quite right. The recipients of the change were complaining that they weren’t seeing the claimed benefits of agile. Things were taking just as long as before.
I asked what happened after development and testing was done to get the product into the hands of users. “Well, we need to wait for integration testing because we’ve got lots of dependencies, we haven’t decoupled the architecture, and we’ve got three or four or five systems that all need to be deployed together. So we have to wait for integration”.
Okay, so when you’ve done integration testing, are you then good? Are you then ready for production? “Well, no, because we then have to wait for user acceptance testing”.
Okay, fine, so you do your acceptance testing, and then anything else before you can then put that into production? “Actually yes, because we batch up releases. So we have to wait for the release cadence, we have to wait for that to happen and for IT Ops to be happy to do the release.”
Okay, is there anything else after that in order to reduce the time to market? “No, after that, then we’re done”.
And so then I’ll ask, okay, so what’s the cadence on this, what’s the frequency?. “Well, integration testing is monthly, acceptance testing is quarterly, and releasing is quarterly”.
I ask, to the left, prior to the work getting to the dev team what happens?
“Well, prior to the work getting to the dev team there is a Product Backlog with epics which need to be refined, analysed and broken down into stories. In some other teams that work is predictively planned into multiple iterations over a quarter, with a focus on commitment of output, rather than experimenting to best achieve outcomes, which feels kind of traditional”.
“Prior to that, a detailed design needs to be reviewed by the organisation-wide Technical Design Authority and before that there is a Detailed Business Case which needs to be completed for each programme or project, funding agreed and approved by a business line steering committee“.
“Before all of that there is an Outline Business Case approval step to filter the business cases going into Detailed Business Case review and ahead of that there is Idea Triage”.
How often do these happen, I ask.
“That’s a good question. It’s monthly for the Idea Triage, quarterly for the Outline Business Case and an annual planning process which typically takes 6 months to complete, using the Detailed Business Cases, which determines what programmes and projects we work on for the following calendar year. It’s always been this way, we start planning in September for the following year, with endless cycles of big up front planning and around March we know our level of funding, on the premise that it delivers the big up front plans, which by the end of the year are 18 months old. Usually there’s gaming of the system, going in high, knowing that it will be knocked back, usually ending up the same or just lower than last year”.
However, when you speak to the dev teams they say:
‘WE’RE SO FREAKING AGILE, YAY!”
“We’re so freaking Agile, yay!” (credit: Klaus Leopold)
Software development teams can improve all they like, however the time from customer need identified to need met, the time to pivot based on a market event or learning, hasn’t materially reduced. Often I’ve heard recipients of change comment, quite rightly, that they have not seen an improvement in time to market.
“A system of local optimums is not an optimum system”, Eli Goldratt, The Goal
Delivery risk is lower with faster learning within development, however it might be the wrong thing being built. Time to learning, time to value, time to delighting customers, flow, has not materially changed end to end. This was a lesson learnt in manufacturing a long time ago. Don’t improve the performance of a non-bottleneck as it will stack up more inventory at the bottleneck and not improve end to end flow of value. There is, however, a human benefit in the local optimisation, where it is likely making working lives more humane and rewarding within the local optimisation, even if the overall human endeavour is not generating broader benefits.
What was also witnessed was an Urgency Paradox (Don Reinertsen & Preston Smith, “Developing Products in Half the Time”). Valuable ideas sit in 12m to 18m of big up front planning with no sense of urgency, no questioning if they might add more value than what has already been locked in the plans for the year. As soon as they reach the product development team they are urgent. However, often by then the Cost of Delay profile has plateaued.
The Urgency Paradox
As per Douglas Hubbard, author of “How to Measure Anything”, the most valuable data for investment decisions is (1) whether the product will be used at all and (2) how quickly it will be used. These two factors trump all other data, including expected cost (source).
This adds further weight to the Urgency Paradox of not waiting one rotation of Earth around the Sun before deciding what to invest in for the next orbit. A lack of flow end to end and an inability to pivot reduces the probability that teams are working on the most valuable thing, both due to a lack of feedback and an inability to respond to a changing environment.
Why wait one rotation of the Earth around the Sun before deciding what to invest in next?
Antipattern 2: Milestone Driven Predicted Solutions
Traditionally, milestones in a Gantt chart focus human endeavour on activity completion, on output, predicted at the time of knowing the least, instead of continually thinking about how to maximise outcomes based on strategic intent and ever increasing knowledge of the landscape, in the context of unique product development.
“Have we achieved the activity”, rather than “What shall we do next, given what we’ve learnt so far, in order to try to best achieve the desired outcome”. The former is order-giver and order-taker with thinking put on hold. The latter requires everyone to use their brains.
“Don’t mistake activity for achievement”, John Wooden
The Urgency Paradox
Milestones are an unsuitable metaphor to use for unique product development (which is emergent), within organisations (which are emergent), to continually delight customers (who are emergent), in a landscape (you guessed it, which is emergent), where they are all changing faster than ever. The metaphor, in this context, traditionally has been overloaded to predict distance (work done), the quality of the destination and time.
Add in managers vs. workers, business vs. IT, low psychological safety with fear, blame and reprisals, individual incentivisation over team incentivisation, tribal identity by task based job role and not surprisingly, success is low at 29% (Standish Group, 2015, source) and employee engagement, while rising, is low with 34% of people actively engaged, 53% not engaged and 13% are actively disengaged (Gallup, 2018, source). The Standish Group survey also reports that an agile approach is four times more successful than a waterfall approach.
This is not to say that fixed dates don’t exist, they do. More on this later. Specifically, the word and metaphor of “milestone” is sub-optimal, perpetuating old behaviours and outdated ways of thinking for unique change.
I’ve often heard milestones being referred to as ‘drop dead dates’. With a long lead time, big bang, traditional approach, the activities to reach the drop dead date have at times been called a ‘death march’. Indeed, the term deadline has its origins from prisoner of war camps in the American Civil War in 1864, where the ‘dead line’ was literally a railing or a ditch which if crossed would result in death.
In one organisation I worked at, project managers set deadlines, fixed output milestones, in advance, without consulting those doing the work, at the point of knowing the least. There was big batch, feast to famine work passing by role, with finger pointing. The deadlines never moved. During the ‘feast’ phase of work per role, people were working unsustainably. I asked some of the people why they were still working there. The answer I got, said seriously: “We are united through a common suffering”. A form of organisational Stockholm Syndrome. An undercurrent of do this thing by this date or die. How’s that for colleague engagement and satisfaction at work?
In the Age of Digital, with a new means of production, with a rapidly changing terrain, there are better ways of working, if an organisation wants better outcomes. This is of course optional. Surviving and thriving is not mandatory.
Antipattern 3: Headless Chickens
In this scenario, product development teams have adopted agile ways of working and have become a Feature Factory churning out stuff, disconnected to company strategy and typically incentivised by activity or output (e.g. velocity, Say Do ratio, commitment), rather than by Outcomes. Teams become a self-fulfilling prophecy of backlog replenishment.
Teams are disconnected from strategic intent, yet continue producing stuff
This could be because there is a lack of flow coming from upstream, as the PMO, portfolio management, investment case approval and technical design authority processes are big batch and infrequent.
It could be due to a lack of clear strategic alignment, a lack of clear North Star, no link between the work being done and strategic goals. There is low alignment and high autonomy, resulting in brownian motion.
It could also be due to disconnect from the customer, a lack of seeking or responding to feedback. A behaviour I’ve observed is Product Owners (especially new POs who used to be project managers) being so focussed on ‘I OWN this Product’ that they forget to speak to the customer and understand what might be most valuable for them.
As an aside, to help address this point, we’ve been experimenting with renaming the Product Owner role to Stakeholder Outcome Lead, to make the self identity more clear. The Stakeholder Outcome Lead is a servant leader focussed on the What, an outward focus on the customer, economic, societal and environmental outcomes (Value. See Triple Bottom Line and this significant statement from the Business Roundtable signed August 2019 by 181 US CEOs, moving away from short-term shareholder primacy to all stakeholders, including society and the environment). This complements the Team Outcome Lead (aka Scrum Master in Scrum) who is a servant leader with an inward focus on the How and building the muscle of continuous improvement as a daily habit practiced by everyone (Better Sooner Safer Happier).
Another possible reason for this anti pattern of teams churning out stuff agnostic of strategy, feedback or value, is a traditional mindset around resource utilisation with a view that people need to always be busy, leading to overproduction in siloes, rather than coming together, swarming and helping to alleviate the constraints of a lack of flow and strategy alignment, coming from upstream.
“An [operation] in which everyone is working all the time is very inefficient”, Eli Goldratt
If there is a traditional mindset around resource utilisation, with a desire for people to always be busy, this will have the opposite effect to that desired, in that Lead Time rises exponentially as utilisation increases in a system of work with variability, as is the case with product development. There is no time for continuous improvement, for resolving impediments, for reflecting, for pivoting, for swarming and the system of work goes slower. People are fully occupied pushing the square wheels. For a great online simulation of the effect of utilisation on lead time, see Troy Magennis’s interactive article here.
Antipattern 4: Start starting
In this antipattern, organisations keep on ‘start starting’ instead of ‘stop starting, start finishing’. This is like adding more cars to an already busy motorway, the cars go slower.
Blissfully unaware of the capacity of the system of work and treating it as a push based rather than pull based system, organisations keep on starting more initiatives, more projects, saying yes to customers or stakeholders demands.
In one specific case, an area was saying yes to twice as many requests from an internal customer, than the natural capacity of the system of work, such that each month the backlog of committed work grew by another month. Applying traditional thinking, the focus was on SLAs at each sequential stage and people incentivised to work harder to meet the SLA, a series of local optimisations, pushing work and actually making the end to end lead time longer by having unlimited work in progress, building up queues at each stage.
The result in this context was an acceptance by the recipients of change that things take a long time to get done, driving a desire to start more things otherwise they’ll never get done (‘the best time to plant a tree is 20 years ago, the second best time is now’), an us and them behavioural norm and growing piles of invisible inventory. This results in lower quality as people try to work harder rather than smarter to meet local optimisation SLAs, and lower engagement. It’s a no win situation with a lack of awareness of the system of work.
The most important measures for teams, and teams of teams, to know are their Flow measures: Lead Time, Work In Progress, Throughput and Flow Efficiency. This determines the health of the system of work. For more on this see Know Your Flow.
Pattern 1: End to end Flow
Where the antipattern is local optimisation, the pattern is to focus on end to end flow, looking at the whole system of work.
First, identify long lived value streams. A value stream is producing something which is of value to one or more internal and or external customers, going from needs identified to needs met. It is why you are doing what you are doing.
It is likely that the organisational structure of the business is already set up as value streams, for example, mortgages, savings, investments, health insurance, travel insurance, luxury bags, jet engines, helicopter engines, renewable energy generation, immigration, recruitment, finance, internal audit and so on. Key is that a value stream should be end to end and ideally with as few dependencies as possible. That is, the value stream should have high cohesion and low coupling. Importantly, value streams are nested. For example, Bank ➜ Investment Bank ➜ Capital Markets ➜ Trading ➜ Equities ➜ Market Making.
Value Streams are represented horizontally, with a logical flow of value from left to right, from need identified to need met, from hypothesis to validated learning. Value streams have a quality of service.
Personas (e.g. young person, family, retired couple) are not Value Streams. They consume the Value Streams. Organising by persona can lead to duplication of the actual value production. Personas are a facade, a specific way of serving up the value streams aligned to the customer segment.
Customer journeys (e.g. sign up, onboard, search, purchase, renew, query, close) are also not value streams, as whilst some customer journeys are specific to a value stream, other customer journeys straddle multiple value streams (where, again, a value stream is the reason you’re in business, e.g. transport, fashion, financial services, energy, insurance and so on). Customer journeys are an action, or an operation, on one or more value streams. For example, when I become a customer of bank, I might want to open a current account, a tax-free index tracker investment account, a credit card, a business account and take out travel insurance. Whilst the experience of account opening is important, the raison d’etre of an organisation is unlikely to be having the best new customer on-boarding process, and then having low quality, long lead times and a lack of innovation for the value propositions themselves, which are the reason I became a customer in the first place.
The primary alignment is value (the reason you’re in business), with customer journeys serving up that value. The serving up of the value is enabled via distribution channels (e.g. mobile, web, mail, phone, store), which implement the customer journeys, optionally aligned to persona. There is a ‘business oriented architecture’, with most people organised around customer need and customer value and a smaller percentage of people focused on how that is served up via the distribution channels. In the Age of Digital that value can be (or in the case of regulation such as the Payment Services Directive 2 (PSD2) in Europe, has to be) served up via each value stream having APIs (Application Programming Interfaces). This is relevant in the context where the customer journey can be automated, for example, the transfer of money from one account to another account, or the online purchase of ‘vanilla’ insurance.
In addition, rebadging existing role based teams, fiefdoms and calling them Value Streams, are likely also not Value Streams.
Value Stream Mapping is a technique which can be used to shine a light on the steps involved in the end to end value creation flow, including wait time vs. value adding time. This provides a Flow Efficiency measure. For most large organisations, Flow Efficiency for knowledge work is very low at around 10%, which means that work is waiting 90% of the time.
The value streams should be as self contained as possible, with dependencies eliminated or mitigated over time, part of a network of interdependent services. This won’t be the case at the beginning, there will likely be many dependencies. Spend time breaking dependencies rather than just managing them. Ways to do this for software products include shared code ownership, internal open source, service virtualisation and breaking monolithic balls of mud into microservices.
In my experience typically 80% of value streams are customer facing and 20% are internal shared services, for example HR and Finance.
Long Lived Products
Having identified the value streams that enable the organisation to economically, socially and environmentally serve its purpose, next, map long lived products to the long lived value streams.
These products are often IT systems. For example an Equities Trading System which enables the Equities Market Making value stream, or an HR system which supports the onboarding of new employees, or a smartphone app which enables management of personal finances or to shop online or to publish a photo of your lunch. The mapping of IT products to value stream shines a light on duplication of systems and eases simplification and decommissioning efforts.
It also shines a light on big monolithic IT systems which straddle multiple value streams and inhibit agility due to in-built hard dependencies. In some cases, the mapping of products to value streams leads to an organisation actually having a product inventory for the first time.
Long Lived Teams
The long lived value streams should have long lived small (<10 people) multi-disciplinary teams on them. This results in teams going through forming, storming, norming, performing (Tuckman) and staying together, unlike temporal project teams. The teams get to understand the unarticulated needs of the customers. The multidisciplinary nature of the teams means that most of the time they have all the skills necessary to deliver end to end value. Team members are ‘T-shaped” generalising specialists. This minimises dependencies on individuals and other teams and hence enables flow, fast feedback and learning. It should not require multiple teams to get value to production, early and often.
The alignment of long lived teams to long lived value streams, creates a tribal identity around the value stream, away from a tribal identity of ‘I’m business’ or ‘I’m IT’ or by job role. Instead ‘we are mortgages’ or ‘we are luxury bags’ or ‘we are helicopter engines’. The nature of the organisation of human endeavour has people working together daily to iterate towards common business outcome hypotheses.
Funding is aligned to the value streams. Each value stream, with long lived teams, is capacity funded. Funding is a constraint on which to maximise the value curve. For knowledge work, the main cost for value streams are people, who are aligned to value streams. Funding is value stream aligned, with a rolling roadmap of value outcome hypotheses.
Long lived Value Streams with long lived multidisciplinary teams . Value Streams are funded.
Pattern 2: Outcomes Hypothesis
“It’s better to do the right thing wrong than the wrong thing right. The righter you do the wrong thing the wronger you become”, Russell Ackoff (source)
Where the antipattern is big up front milestone driven predicted solutions, the pattern is to focus on outcome hypotheses with experimentation. From deterministic output to outcome hypotheses.
We can’t travel in time, product development is emergent, it is unknowable, what people want, how it’s going to be built, what obstacles we’ll come across on the way and how external market forces will change over time. Acting in the space, changes the space. The North Star should not be a fixed solution or output by a date. It is not a knowable straight path from A (current condition) to B (desired outcome). The bigger the distance from A to B is, the less knowable the journey is. It is beyond our knowledge threshold. A traditional approach is placing a fixed bet at the beginning of the race when the least is known.
Instead it is a wiggly journey from A to B, optimised with fast, real, feedback and learning, with safe to fail experiments, in order to test the Outcome Hypothesis. For unique work, we want to maximise variability and run multiple experiments, maximising learning, early and often. This approach enables changing bets on the race while the race is running.
Toyota Improvement Kata, Mike Rother
The outcomes are nested, at multiple cadences. Daily, weekly, monthly, quarterly, yearly, multi-year. The key focus is the quarterly Business Outcome (also known as an OKR). A three month time period is not too close and not too far away. This is broken down into monthly Features, which are experiments to test the hypothesis. These are in turn broken down into Stories which are daily and optionally into hourly Tasks. Therefore there is learning and innovating many times a day, on the desired outcome hypothesis.
The Business Outcome hypothesis is written along the lines of the following:
Each quarterly Business Outcomes has a Portfolio Epic parent, a higher level North Star outcome hypothesis. Each business area has a handful of Portfolio Epics, limiting work in progress. This is strategy alignment. The <12 month Portfolio Epics are nested within multi-year Portfolio Objectives, which are business unit level strategic items. And for larger firms, they are contained within a handful of Strategic Objectives which is the top, group-level, strategic intent, spanning all business units.
The nested Outcomes are represented vertically, from multi-year strategic intent down to daily stories. Typically each level has its own Kanban board, with WIP being limited at all levels.
Value, which is unique and not aggregatable, is expressed in the Lagging Value Measures. It is important that these are measurable, that they are quantitative.