In their iconic book “Peopleware”, DiMarco and Lister said “I can’t tell you what to do to get a team to gel, but I can tell you 100 things that will prevent it”
A similar thing applies to Agile. A lot has been written about how to become agile – what practices to use, and what principles and behaviours to adopt. Little, however, has been written about what can stop an agile transformation in its tracks. There are a number of things that can undermine or derail an agile team or company, especially one in transition. One day I might write that book, but the focus of this post is just one such factor – individual KPIs (Key Performance Indicators).
I was recently made aware of an organisation that did an ‘audit’ of user stories and concluded that they were not of a very good quality. Management, typically, decided that to improve this practice, KPIs were needed. So someone asked the members of the Business Analysts practice to come up with their own KPIs.
Here are some of them, and why I think they are a Very Bad Idea :
1. Percentage of projects that have prioritised and estimated requirements – If the perception is that 100% is perfection, teams will be motivated to put any priority and estimate against the stories, just so they don’t ‘fail’ the metric. So in some cases, the estimates and priorities will be meaningless. They could also come up with just a handful of stories with priorities and estimates at the time the metric is taken. There – passed!
2. Percentage of requirements fully implemented – This is the same as fixing the scope of a project at the start, something that all agile methods are against. Meeting this measure means teams are motivated to prevent change (in contravention of the fourth value in the Agile Manifesto). And assuming the requirements set is baselined at the end of Foundations, meeting it would encourage teams to spend a lot longer defining the set of requirements and be excessively pessimistic about their estimates to ensure that they are all delivered.
3. Quality of Requirements Index – The intention was to measure whether stories adhered to the INVEST criteria for writing good stories. But this cannot be done reliably on an automated basis, and manual reviews are tedious, time-consuming and subjective. This measures the mechanics of story generation, not the quality. How do I know whether a team’s story is Independent or not without understanding the rest of the requirements set in total? Is a story Small enough? I can’t tell that either. Is Testable measured by the presence of acceptance criteria? Very easy to game that, isn’t it?
4. Number of missing requirements – It is normal – indeed encouraged – that requirement detail emerges over time, that stories are decomposed as the project progresses, that new requirements will also emerge as the product is developed. Are these classified as ‘missing’ requirements? Like measure No 2, this also encourages fixing the scope of the project at the end of Foundations, leaving the project manager with no contingency, and pushing back on any attempt by the business to change the requirements.
These are not things we want to encourage, and improving the quality of a set of requirements cannot be achieved by setting these sort of measures – quite the opposite. Gaming them, especially when they are automated, is easy. Remember, you get what you measure, and managers are all too quick to measure things and present figures that show themselves in a good light because the numbers are going up, but without thinking of the consequences of these measures on the teams and individuals involved. Agile is a team game, after all. No, the answer lies in education and coaching teams on how to do it properly and the benefits they get from doing so.
In conclusion, I’d like to share my opinion that measuring individuals on an agile project team is in itself harmful. How would a team feel if just the Business Analyst was subject to such measures? There is potential for the needs of the BA to meet the KPIs to be at odds with the team’s needs, and this is detrimental to teamwork. There are a lot of things that are worth measuring in an IT organisation. Individual KPIs like those above are not in that list.