If you’ve read along so far, we’ve summarized the process of evaluating and choosing something new. We then helped you to define a specific problem to be solved. Whether you need a specific niche new tool for the toolbox, or are replacing some software completely across your organization, evaluations are critical. Getting through the evaluation process isn’t trivial, and in order for the process to be truly successful, both the buyer and seller should feel like they got what they wanted. This doesn’t always mean that the seller will make a sale, or that the buyer will find what they’re looking for. But at this point, what’s critical is having a repeatable, objective, and defensible way to make a decision based on specifically defined success criteria.
Are We There Yet?
I remember before the days of pocket computers and maps a la Google Maps, parents feared getting in the car with their kids on vacation. The dreaded “Are we there yet?” plagued parents everywhere. The reality is that today we have tiny computers in our pockets that tell us just how close we are to our final destination – and can help us understand the journey. When you’re working through an evaluation of some new product or piece of software, it would be awesome to have this same kind of tool. In fact, we do. Creating success criteria is important for evaluations… so important in fact, that at previous companies at which I worked, senior management had to sign off on success criteria in order to ensure fair and equitable evaluations. I approve of this approach enough to write about it.
The keys for success criteria are as follows, in no special order:
- They must be objective – There is nothing more frustrating than having to debate with someone what ‘better’ means. Have you ever tried to convince someone from New York that the thing they call pizza pales in comparison to the culinary gift that is Chicago deep dish? Odds are you’ve got different criteria for what you like. They probably like cardboard and giant floppy slices (see inset), while you’re a fan of hearty, cheesy, goodness. Evaluating pizzas based on subjective criteria is silly, just as evaluating security tools is. You will need some way of defining what it is you’re looking for in a manner that everyone can agree on, that gets you to a point where there is no room for personal opinions or biases. And there’s the bit we need to be concerned about. Everyone has their inherent biases and opinions. Maybe half of your team likes one vendor, and another half a different vendor. Unless your success criteria are objective, you’re going to have an impossible time getting to a consensus as opinions will insert themselves, and it’ll be ugly.
- They must be binary – Based on the above (must be objective) key, you’ll want to come up with a method to get to a binary yes or no. Does this pizza have flavor? (This is a trick, because having flavor is subjective.) What you want is to agree that the proper thickness of a pizza is somewhere between .75” and 1.25”, that way you can measure the pizza and determine that yes it does, or no it does not, meet the criteria. You’ll want to apply this to your evaluation by defining a series of success criteria that you can answer yes or no to. This is very difficult. Getting these criteria is the hardest part, but once you’ve got them and have agreement from the evaluation team, the evaluation will be significantly less encumbered by opinion.
- They must be developed for the specific use-case – Your success criteria must be based on the use-case that the thing will serve. I know this feels obvious, but it’s a little more difficult than that. We, as human beings, tend to think of all the wonderful things we could do with whatever we’re evaluating. That pizza could be breakfast tomorrow, or a midnight snack later on tonight. That means we need to make sure that it’ll be good cold, and that reheating won’t be a problem. Even though those are interesting things to think about, the use-case is right now at dinner. So you want to put those off into the edge or “supplemental criteria”, if you choose to have such a thing (in case of a tie breaker). Whatever the use-case you’re buying or evaluating for, that should comprise your primary success criteria. Period.
Defining Solid Success Criteria
So it’s easy, right? Here’s my three-step formula:
- When defining objective success criteria you can use, use an odd number of them. The odd number of success criteria helps prevent ties. Make sure your success criteria can be answered “yes” or “no”, and do your best to avoid “good, better, best” situations.
- Make sure your evaluation criteria have binary responses, yes or no, on or off, does or does not. That pizza is either within tolerance, or it’s not. The toppings either meet the predefined objective standard, or they don’t. If you ever find yourself saying “well, sort of”, you’ve gone astray.
- Figure out the use-case you’re evaluating for. What do you want to do with the tool or software you’re evaluating to purchase. Whatever that use-case is, frame your mind in that manner.
Let Them Eat Pizza
Now that you’ve got the success criteria part down… you should go eat. All these pizza references have me missing a good Chicago style deep dish. Try my favorite, Lou Malnati’s… you won’t regret it! Celebrate your progress while you can, because the hard work is just beginning. Check back soon for the next part in this series, “Stage 3 – Execution of Testing”, where our questions start to get some real answers.
Credit: Images are of actual Lou Malnati’s deep dish pizza and from their site.
Rafal Los serves as the VP of Solution Strategy at Armor. He’s responsible for leading the various technical functions associated with designing, developing and delivering next-generation cloud security-as-a-service solutions to our clients. Rafal is also the Founder & Producer of the Down the Security Rabbithole Podcast. He previously worked as the Managing Director, Solution & Program Insight at Optiv Inc.; Principal, Strategy Security Services at HP Enterprise Security Services; and Senior Security Strategist at HP Software.
As an IT security professional, Rafal gained experience in some of the world’s most challenging business environments. His responsibilities included budgets, risk analysis, process creating and adoption, internal audit and compliance strategies. His professional experience has taken him from budding “.com” companies, to a security boutique shop, to one of the world’s largest and most complex enterprises – always meeting challenges head-on and with a positive attitude. He has been the catalyst for change in many organizations, building bridges across enterprises and developing permanent successful strategies for growth and prosperity.highlight industry infosec los project success