Here we are, rolling out SCRUM across our teams, having teams practice the process and deliver customer value. We needed to figure out a simple way to measure the effectiveness of our rollout. Measuring the effectiveness has two parts:
- Part 1 How effective were we in changing team behaviors, making our team members and product owners comfortable with the process, and helping institute the correct practices.
- Part 2 – How effective are our teams at delivering customer value. What do we tell the rest of the organization about measuring the effectiveness of our changes on productivity and delivery of customer value?
Measuring Team Behaviors
To help us measure team behaviors, we have taken to anonymously surveying the scrum teams 3 times a year. In these surveys we ask each person to rate their team on their team’s effectiveness at the various scrum ceremonies and practices, the scrum/agile behaviors we want the teams to exhibit, and how the team rates the folks performing key roles.
Armed with this information, we can see how the teams are evaluating themselves and learn what things we as a management team want to improve. We can work with each team’s scrum master (or a scrum coach), with the people managers involved, or the product owner to attack each of the areas.
By focusing on the effectiveness of the practices and ceremonies, separate from the behaviors we can identify if it is a person or team improvement that is necessary. We can also get independent evaluation from the team on the performance of the key roles (Scrum Master, PO, Management) in helping the team be successful.
We also ask about the success of SCRUM as a net promoter score (NPS) question to see how the teams feel overall. Using this data we can look at both this moment in time as well as review the results over repeating surveys to identify trends. We get about an 80% participation rate which gives us a very good feeling about the data.
Team Productivity Metrics
We have normalized on 3 easy to collect and measure team based metrics.
- Sprint Velocity
- Sprint Health
- Time to Customer Value
Sprint Velocity should be obvious to anyone who is running SCRUM. This is simply a measure of velocity achieved by each time during each sprint. We run in two week sprints and each team at the end of the sprint records the velocity achieved in story points for the sprint.
Sprint Health is defined as the committed sprint velocity divided by the actually achieved sprint velocity. While adopting scrum, one of the learning principles is learning how to get each team to groom their backlog and plan their sprints well. Because we story point each user story and we have a strong definition of done, there is variation between what the team thinks can be done in a sprint and what actually gets done. I have found that this measure helps me understand how each team is learning, correcting, and implementing SCRUM.
Time to Customer Value is defined as the time it takes for a story to be totally consumable to its customer from start to finish. We measure by calculating the time the story enters a sprint until the time that the story is consumer by the customer specified in the story. We then average this duration for every story over a rolling 6 months. Like any organization going through an Agile transformation, even with our CI/CD pipelines and rapid deployments, getting a story into the hands of customers takes longer than a sprint. It could be because it is deployed but hidden behind a switch, or because it is stuck waiting for a marketing campaign, or because a PO is waiting for further work before releasing.
One of our goals is to start measuring how long it takes to get stories to our customers. Can we get SCRUM to help us deliver more customer value faster? Just by measuring, I have noticed our teams are asking themselves the right questions. With two-week sprints, a really great answer would be 12-14 days. You can actual over achieve and get numbers less than that if you deploy each time a story is complete. We take this average at the end of any sprint and report on it. Just by watching this number, I can observe teams asking themselves the right questions about:
- breaking work down
- deploying more often
- reducing the overhead of deployment
- automating more testing
- overall working to deliver greater value.
Each team produces these metrics each sprint and reports them each month. I have found not only the raw data useful, but also graphed as a moving timeline, it is very easy to discern patterns and help teams improve.
With these very light weight metrics, we can provide the teams themselves as well as engineering management metrics to help us understand how our teams feel SCRUM is working for them, how well our SCRUM is impacting our delivery, and where we might have issues within the teams that need to be addressed.
Are you an engineer or engineering leader who is looking to work and learn in an Agile culture that works on leading edge technology and is focused on delivering customer value for our Small Business Customers? Check us out at jobs.constantcontact.com.