This article, written by me, was first published in November 2016 on Hack & Craft News:
It may be controversial to say it but much of what passes for “agile” in the enterprise is anything but. Agile was intended to replace waterfall development processes and it has – the result is many mini-waterfalls and, with that, a level of inefficiency that we don’t need to accept.
I want to argue that there are many better techniques than agile. Rather there are many techniques that add up to something better.
The rise of “flow”
It is worth thinking of this new method as “flow”, and by flow I mean a whole new cultural approach to work, not just a methodology that gets applied to tasks and workflow.
People in a flow culture are able to accept that many of the decisions that guide their work may be up in the air. They are excited by the presence of uncertainty. There is something very different about flow and it is this. Decisions that need to be made about features and products will emerge from the social interaction of the people doing the work. In fact, one can iterate and focus on value, right up to the final moment of commit when an efficient handover to another member of the group is made. The boss, the CIO or CTO is there to create the right conditions for people and allow them to focus on getting the best possible outcomes.
There may never be a flow manifesto but if there was it would stress one thing above others.The primary mechanism for creating good flow is extreme visibility. The social interaction of good people excited by their work can be guided towards great results if we keep on pushing the boundaries of visibility. Now let me explain all that.
The problem with agile
What’s wrong with agile? It has been dominated by the discussion about software methods. It has been the main protagonist for waterfall. It may even have been effective in dismantling the era of monolithic software. Those are all commendable achievements.
However, time moves on. In many of the environments where I’ve worked agile could better be described as mini-waterfall. One of the reasons for this involves the roles people play in productive environments and an overlooked star of the software show is the Business Analyst.
In waterfall software projects, analysts help to build a specification for a large system; that specification is then allocated to developers who code the features. The whole process is overlooked by a system engineer who assembles the code into a monolithic package.
We certainly don’t want to go back to those days. But look what’s missing once the process moves to coding. The analysts, the people who have helped describe the need have disappeared.
In agile, the situation is pretty much the same. Analysts help define a specification and all that really changes is the division of work. Of course in the agile manifesto the division of work should be done in a way that allows developers to interact more with customers and therefore to adapt some of the analyst’s assumptions.
The customer is meant to be king or queen. In reality what happens is that much of the agile process is dependent on the smaller work packages and the imposition of sprints. Sprints are the new currency in many enterprises. Work is queued in planning sequences prior to the developer team sprinting to the next milestone very couple of weeks or so. This doesn’t prioritise a looser form of decision making, a wait and see what works best approach or a real engagement with the end user. The analyst is often the proxy for the customer and as often as not is left out in the cold.
The importance of extreme visibility
In the new world of flow, the emphasis is quite different. Flow is well adapted to a newer production environment where continuity of delivery is important and where architectures are becoming more dependent on multiple, even smaller, packages of software, usually referred to as microservices.
When you create architectures that are dependent on microservices you need a much higher awareness among teams of what’s happening at the architecture level. Everybody needs the overview because they need to know where their pieces fit.
You also need a much stronger emphasis on the architecture itself because the CIO/CTO role is more about integration. Those analysts that get left in the cold in other methods becomes essential to the analysis of how the architecture might fit together and where the packages are coming from.
But the very complexity of a microservices architecture also makes it imperative that people simply know a whole lot more about what is going on. A microservices architecture is often too big for one brain. It needs the most talented people in the team and sometimes the most specialised people in the team to keep reviewing it.
That observation is the beginning of an extreme visibility principle.
What do I mean by that. Well, you can’t just ask your best people or specialists or friends to take an occasional look over the workflow and architecture and give them a thumbs up or down.
They really do need to be involved and willing to commit their intelligence and creativity. You are sunk without it.
Over the course of the past five years I have found that making everything visible helps create the flow culture where everyone is involved and everyone can live with uncertainty.
Here are some examples of what I mean by extreme visibility.
- The Portfolio Wall. Portfolio Management is often seen as the scourge of Agile. But that’s because this vital part of Business prioritisation is often at odds with agile processes. But using a visual board, at the project level, can allow the same adaptive, flexible, experimental method of team discussion and prioritisation, that you get at the task level, right up to the last the very last moment of commit when resources are allocated and work starts. Also, this wall is very visible to all staff and is in fact a true transparent representation of a business strategy in progress, or should I say in flow.
- The Kanban Wall. Around our office there are hundreds of post-it notes related to projects and the different tasks that we have in progress. Everyone can see them and many people do. There is nothing invisible about what we are doing. That of course is good Kanban and it works as a starting point for getting people out of their own personal silos.
- Team Kanban Boards. Don’t let the very few post-it’s on these boards confuse you. This is the “actual” work in progress. One post-it per person and only the tasks in hand, not those that are blocked. Therefore, if it’s not on this board, it is not being worked on and therefore product owners can’t be fobbed off (the cheque is the post syndrome) but more importantly, working with the analyst, they can change the flow into the team at will before the task is allocated for work.
- Risk & Issues Wall. Here any team member across all the various teams can raise risks and issues that they need the technical leadership to address. Everything from a delay in building servers to concerns about a process or lack of resources. The leaders must address the risks or in fact accept them and take our chances. It’s as simple as that. Sometimes you can’t fix everything but knowing that a leader supports you, can give confidence to the team.
- Fun Walls. Not all extreme visualisation needs to be process based. Look across our office and you will see a Jobs Wall. It is a situations vacant space but it’s more than that. Employees can come here and request a job, what they would like to work on next; post their ambitions, where they want to go next. It is a social interaction space for the most important part of the work we do, the people who do it. There is also a Thank You wall. It’s important that people’s appreciation is just as visible as their tasks. If folks want to say thank you to colleagues they can post it here. I can too and frequently do. It let’s people know that they are appreciated but it also let’s other people see that appreciation. I’m currently working on a concept called the Cool Wall (as per the Top Gear TV show) whereby team members can post things that they feel are really cool and that we, as a company, should investigate such as containers, scalable programming languages or indeed DevOps. This is to celebrate innovation and creativity and bring it out from underground activities in order to make it real. However, the other end of the wall (not so cool) may actually contain many of our existing assets and I’m not too sure if that’s too negative. But we’ll see.
- Obeya. My ultimate dream was to have a room whereby I could visualise all the critical projects in progress and provide a white space for ideation, innovation and interaction with people from all over the business. I now have this where I currently work and it provides the forum whereby key decisions are made. Almost like a war room but in constant use and spanning a number of initiatives. Obeya is the forgotten poor cousin of the lean world but one which is vital to overall process. The biggest obstacle in most companies is finding a room!
What does all this add up to? In many production environments people see a degree of structure that they assume will guarantee quality and delivery. But structures as we have known them to date require a lot of planning, agreement and sign off.
The fact is that we are are moving at such a fast pace now on many fronts, there isn’t the time to create plans and to update them. You need to be in the flow. You need to have tasks up there on the wall where you can switch them out when things change. You need people who hate ambiguity embedded in a support mechanism that helps them to excel. You need to have people’s attention and engagement because all those eyes are going to spot mistakes, see things differently or suggest a better way. More than anything you need a system, a process, a flow that gives you ultimate flexibility and that let’s you add in all the new learning that floats around our heads, on the Internet in particular, but at conferences or in conversation with startups. We all have to admit that we are a “work-in-progress” and just go with the flow.