Introduction
As a project manager, it is often easy to get caught in the day to day activities of managing a project’s delivery and we fail to take a step back and measure the work we are doing. By taking a holistic view and quantitatively measuring the effectiveness of you and your team throughout the project, you are able to identify and resolve pain points and improve your overall delivery over the course of a project as well as future projects.
A key performance indicator (KPI) is a measure of performance. In the context of project execution, KPIs allow project managers to understand where their projects are succeeding and where there is room for improvement. Having distinct KPIs and metrics to measure a project is essential for effective delivery. Here we will discuss some common KPIs to consider when running a traditional waterfall delivery model as well as in Agile.
Waterfall vs Agile
Before diving into the key performance indicators for delivery it is important to call out the high-level differences between Agile and the traditional waterfall life cycle. Waterfall is a linear life cycle divided into distinct phases (i.e. design, development, testing, etc.) which are completed sequentially. Agile on the other hand offers more flexibility with an iterative and incremental approach in which your different software development phases are performed, producing a useable product increment, as the outcome of each sprint. It is important to take this into consideration when we measure project performance.
4 KPI Areas
In any project, the effectiveness of your delivery can be measured by considering 4 areas: Schedule, Scope, Budget, and Quality. This is often referred to as the triple constraint in project management, where the quality of work is tied to a project’s budget, deadline, and scope.
Scope in Waterfall
Work Delivered vs Work Planned
When discussing scope, one of the first things that comes to mind is what work is being delivered. In a traditional model where the approach is to complete the development of all requirements identified at the planning phase and deliver a final product, work delivered is a direct measure of completing the scope planned against the scope delivered. This includes any change requests approved throughout the project.
How to Measure:
- Score/Weigh work using a common scale
- Map team member capacity to a measure of the work they can complete given their availability
- Planning will yield a breakdown of work to which each team member commits for a given work cycle. The summation of the tasks measured provides a measure of the “work planned”
- The measure of the work delivered is determined when the team is “pencils down” in the project
Often times complexities associated with delivering a piece of scope will emerge after the team has started its work. New information may reduce the importance of a scope item considering the time/dollars required to complete it, so it is important to take this into consideration in terms of your planned work.
Planned Capacity vs Realized Capacity vs Scope Delivered
The purpose of tracking capacity is to understand the relationship between the project staffing model of the organization and the ability to achieve staffing needs for its execution goals. As a PM, it is important to understand the project team’s available capacity before the start of any project, understanding prior commitments or competing priorities. By measuring each resource’s capacity and utilization with a uniform metric, the PM is able to track capacity throughout the project as work is delivered and new needs arise. For a waterfall model it is important to plan resource capacity in the planning phase prior to the official kickoff so that it can be tracked throughout the project execution It is also important to proactively manage the schedule to ensure that resource capacity aligns to any schedule adjustments.
Scope in Agile
Work Delivered vs Work Planned
In an Agile model where the team operates using an iterative approach with each iteration delivering an incremental product, the work delivered is directly associated with the backlog items assigned to the given sprint. However, this also serves as a metric that teams can use to better understand their planning and ability to target scope and deliver each sprint, allowing the team to plan more accurately in future iterations.
Productivity Trajectory
As mentioned above, in an Agile model a team’s planning can improve for each iteration as the team gains a better understanding of the impact of known constraints and how to manage potential blockers through prior experience. This leads to another useful metric, Productivity Trajectory, which is a measure of whether or not the team is getting more done over time relative to the scope of work in each iteration.
There are two ways to look at this KPI: Project Team vs. Organizational Team
- Project Team – comprises individuals from different organizational teams who work together for the duration of the project life cycle. The KPI indicates how well the team is working together (or not) over the life of the project with the goal being that the project can demonstrate improvement.
- Organizational Team – comprises individuals who are “homed” within the same organizational unit within a company (e.g. Application Development, Product Development, DevOps, Finance, etc.) and who work on multiple projects with individuals from other teams. The KPI measures the overall performance of the organizational team on meeting its responsibilities across the myriad of projects to which its constituents attend.
Capabilities or Features Delivered
Scope for an iteration is typically chosen so that a specific feature(s) or set of capabilities is delivered at the end. By tracking the business-defined features which are accepted and delivered for each iteration, you can understand your progress towards the end goal.
Planned Capacity vs Realized Capacity vs Scope Delivered
In Agile, the project is typically handled by a dedicated team which remains the same at every sprint. However, team members are able to reflect on their prior commitments and their ability to honor them. Did they deliver by their committed date or with time to spare? Did they take on too much work? Did their performance impact the overall performance of the project? It is also important to note that depending on organizational structure, teams are often a composition of fractionally allocated members instead of truly dedicated, which makes understanding a team member’s commitments and actual capacity even more important.
Budget in Waterfall
Cost Variance
Cost Variance, a popular measure when discussing Earned Value Management, serves to measure how a project is aligning to its spend plan throughout the duration of the project. With knowledge of where the project is in its life cycle, cost variance can be used as a performance indicator. Measuring cost variance allows us to understand how far along we are in project spending and if we are trending over or under budget, allowing us to determine if any adjustments are needed.
Budget in Agile
Agile Team Cost
This refers to the cost of the Agile team as a unit over time (Agile typically consists of a dedicated team so team size should remain constant). With a constant team size, this measure can be used to determine the cost of work overtime by the development team if the amount of time needed to complete the work can be estimated. As part of the Agile team cost it can be worthwhile to understand the breakdown of resources by function (programmers, engineers, architects, etc.)
Requirements Cost
With an understanding of the Agile team’s structure and recurring costs, and the scope of work to be targeted, it is possible to breakdown cost estimates by specific features, epics, user stories, or general capabilities to be delivered as long as you have an estimate of the level of effort for each. By tracking this information, you are able to track not only the total product estimate but also the MVP estimate.
Some cost measures that also provide insight into value delivered include:
- Cost per Delivered Feature or Capability = Total Cost to Date / Total Delivered Features or Capabilities
- Cost per Story Point = Cost to Date / Total Delivered Story Points
An organization will often provide project funding for an Agile project based on the amount of dollars estimated to deliver a set number of features, so it is important to track the actual feature cost once delivered and compare it to the originally estimated cost.
Cost Burn Rate
This is the tracking of costs over a set period of time (sprint, month, release, etc.).
Schedule in Waterfall
Schedule Variance (SV)
Measured similarly to cost variance, schedule variance indicates how ahead or behind a project is progressing against its planned schedule. By measuring schedule variance over the course of a project, the team will identify schedule issues or risks and determine whether schedule compression techniques should be implemented (e.g., fast-tracking or crashing).
Schedule in Agile
Velocity
Velocity for a Scrum team is a key planning tool and describes the quantity of work a team can be expected to deliver at the end of a sprint. From this velocity, the product owner conducts release planning to predict when a feature or multiple features can be delivered. This is often measured through a burndown or burnup chart at the sprint level. Example below:
Velocity Predictability
Velocity predictability is measured as the difference between planned and completed velocity, which is the difference between the total planned and completed story points for a given sprint. By analyzing the fluctuations in velocity for a given sprint, one is able to understand whether there are persistent interruptions (competing priorities, blockers, unplanned work, etc.) that are slowing down productivity.
Quality in Waterfall
Defect Distribution
Defect distribution is key to measuring and improving product quality as well as decreasing potential bug fixing costs later on in the project. In a traditional model, all development will typically be completed prior to the QA (Quality Assurance) phase which should typically capture any major bugs prior to UAT (User Acceptance Testing), but any post-production defect statistics will serve as a measure for the QA/UAT and the development quality of the project.
Defect distribution can also be used to capture the test effectiveness which highlights the difference between the number of defects found by development and QA vs the number of defects found post launch
Defect data can be distributed across multiple charts and tools for further analysis (Pareto charts, histograms, pie chart, etc.) and can be used to pinpoint areas that are not meeting your project’s quality standards. Below is an example of a Pareto chart showing the distribution of defects by reported cause:
Customer Satisfaction
Another measure of a project’s quality is through customer satisfaction. This can be quantitatively measured depending on the project’s end result through metrics such as a net promoter score and user churn rate if your product is a service. On a more qualitative note, customer satisfaction can be measured by what the project solves, is it improving or replacing an existing process? Does it reduce costs? Has it received positive customer reviews? Customer satisfaction does not only pertain to the end user, it is important in a traditional project to keep the Sponsor and key stakeholders involved to understand if the project achieved its original objectives. Defining the objective and KPIs upfront allows the PM to measure throughout the project and make any necessary adjustments, to ensure achieving the objectives upon project completion.
Quality in Agile
Defect Distribution and Test Coverage
In Agile, testing and development occur in the same iteration allowing for the team to identify and fix bugs continuously hence lowering defect rates over the course of the project. Tracking defects and eliminating defects over time will lead to a more production-ready final product. Having insight into the amount of testing being conducted and correlating it with the value delivered will improve overall project quality.
These are some variations to consider when measuring test coverage:
- Percentage of tests that are automated
- Number of automated tests per capability, feature, epic, and/or story
- Percentage test coverage of capabilities, features, epics, and/or stories
- Percentage test coverage of code (by module or line)
First-Time Pass Rate
The first-time pass rate is a measure of the number of stories, features, or capabilities that pass after the first test. Tracking this pass rate provides insight into how well user requirements are captured, understood, developed, integrated, and delivered. It is a good measure of the effectiveness of the end-to-end delivery process.
Summary
Conclusion
The KPIs mentioned here are essential metrics when measuring the effectiveness of your delivery. Although many of these KPI are relevant for both frameworks it is clear that the benefits of these KPIs and how to measure them can range depending on whether you follow an Agile or traditional model. In a traditional project, where the focus is on a one-time final delivery, these KPIs reiterate the importance of planning and measuring effectiveness over the course of the project. With a more flexible and iterative framework like Agile, it is possible to use these KPIs for gradual improvement over each iteration and improve your overall delivery.
