How to make tough software decision with a decision matrix
Changes within a monolith can have far reaching, unintended consequences. This makes it extremely difficult, if not downright impossible, to safely make foundational changes to a system. At Gusto we’ve been building towards a modularized monolith, but we still have some work to do.
When my team decided to tackle a project that would fundamentally alter the way we reconcile taxes, there were a few concerns. How does a payroll engineering organization give the green light for a basal change to their tax system when the consequences reach many domains, features, and teams?
For a few weeks we circulated the problem to various engineering teams. We asked for input and our repository of creative solutions continued to grow. After a few weeks, despite the engaging conversation, we were left with many potential solutions to shipping this change but no clear answer as to how to decide between them. Each solution had benefits, but each also came with non-negligible risks.
This type of analysis paralysis has the ability to seriously slow down a project, or even bring it to a complete halt. Luckily, my manager suggested we consider using a decision matrix. With a decision matrix we were able to break down our options, clearly compare solutions, and find a path forward.
How to build a decision matrix
A decision matrix is a tool that provides a weighted comparison between many options. When there are many different factors and risks to consider, but there can only be one path forward, a decision matrix is a great way to analyze many options and find consensus as a team.
Define the evaluation criteria
Instead of starting with solutions, start with the evaluation criteria. The first step to building a decision matrix is clarifying the metrics of a successful project. With our tax reconciliation project, for example, the most obvious criteria for success was that the work improved tax accuracy. Upon further consideration we realized it was also important that the impact to the change was predictable. We had a strict timeline for the project. Our team would like all of our projects to move us towards our long term goals.
After carefully considering what would make this tax reconciliation project successful, we were able to come up with these requirements:
- Change improves tax accuracy
- Impact is predictable
- Change is reversible
- Change is not noticeable to customer in app
- Project can be completed within two months
- Solution moves us towards our other long term goals
As this is a highly technical decision, many of the criteria came from engineers. However we also gathered some critical requirements from cross-functional stakeholders. Customer facing stakeholders were able to provide input our engineering team had not considered.
Weight the evaluation criteria
Listing out your evaluation criteria is a helpful first step, however each criteria does not carry equal weight. For example, while our team would prefer a solution that “moves us towards our long term goals”, we care much more that the “change is not noticeable to the customer in app”.
The next step to a decision matrix is weighting each of your listed criteria. Each criteria carries a different level of importance and our decision matrix needs to reflect this. You can select any range to apply weighted values. Our team chose 1-10, with 1 being the least important and 10 being the most important.
Defining and weighting the criteria should be a team activity. Discussing what success looks like for the project will bring uncertain areas to light and give the team an opportunity to clarify incongruences early on.
Rank the solutions against each criteria
Once the evaluation criteria are clearly defined and weighted, the team can score each solution against each criteria. We allowed for scores 0-3, with 0 meaning the option does not fulfill the criteria and 3 meaning the criteria is very likely to fulfill the criteria.
When completing the ranking process we wanted to avoid anchoring bias, the tendency for humans to be influenced by the first piece of information they hear. Each teammate completed the ranking process solo in order to ensure as little bias as possible. We then came together to discuss each evaluation criteria ranking vs each option and agree on a ranking as a team. This was a lengthy process and should not be rushed.
Calculate the final solution score by multiplying each criteria weight by the solution ranking for that criteria and adding together the total.
solution_score = sum_for_each_criteria(weight * solution_ranking)
We finally had it. Our solutions were scored by weighted criteria and our path forward was clear.
Make a decision and keep building
With the help of a decision matrix, my team was able to make a fundamental change to our tax system. We do not build software in black and white. Any complex (and therefore interesting!) problem typically comes with a multitude of potential solutions, all with unique benefits and risks. A decision matrix is a powerful tool for teams to bust through their analysis paralysis, see the problem clearly, and find a path forward.
