Wrapping up this series on evaluating technologies and services is actually fairly simple. Determining the outcome of an evaluation is, in fact, mechanical. That is to say, that you shouldn’t have any emotion or subjectivity in your determination of the outcome.
When I think back at all the evals I’ve led or participated in over my 20+ years (damn, I’m getting old… get off my lawn), I can separate them all into two categories: those that went well and those that didn’t. If I’m truly honest with myself, the reason for the difference is fair and easy to see. That difference is subjectivity. The moment you add feelings into an evaluation you’ve lost.
Outcome How To
So how do you do evaluations without feelings? Spreadsheets, of course.
Set up your spreadsheet with a table of 4 columns, and as many rows as you have evaluation criteria. Now you’ll remember that in The Evaluation: Stage 2 – Definition of Success Criteria (the one with the Lou Malnatti’s pizza example), I told you that you should have an odd number of success criteria. What I didn’t tell you is how many you should have… the answer here is “enough”. Actually anything more than 11 (remember, odd numbers) is good.
So column 1 is the name of the success criteria.
Column 2 is the weight; you’ll use 1, 10, 100 for “nice to have”, “should have”, and “must have” respectively. So for criteria that are the “can’t do withouts” – the ones that support the core of the use-case you’re trying to solve – you give that a 100, and so on.
Column 3 is the binary (pass/fail) metric and in that column you either give a 1 or 0. Nope, not 1 through 5 … 1 or 0.
Column 4 is the multiplication of columns 2 and 3. So if you have a “must have” item that passed, it gets 100 (100×1) points. If you get a must have item that failed, you get 0 (100×0) points.
Now, when you’re done doing hard math on all those rows you add up columns 4 together to get the total for that product/service you’re evaluating.
You should have a separate tab in your spreadsheet, or a separate table, for each item being evaluated. You should also do them each independently. Do NOT perform them at the same time, so you’re not tempted to cheat subconsciously.
That’s it. Simple. Done.
The beauty and effectiveness of this system is in the simplicity. If you were truly honest throughout the entire process (which you can review using the links below), then it’s just simple math at this point. I promise, it may feel a little overwhelming the first time you run through it, but the more times you run this process the more intuitive it’ll be. And this can work for all evaluations… internal or external, products or services, security or otherwise.
We’re done! Good luck, and I always welcome feedback. If you’ve got a way to improve this process or have suggestions on where it could go wrong, I’d love to hear from you. This will not only help me but also help the next person do it better. You’re welcome to use this, any part of it, and call it your own… although we both know.
Articles in this series:
- The Evaluation – Four Phases to Finding “Better” Solutions
- Stage 1 – Definition of the Problem to be Solved
- Stage 2 – Definition of Success Criteria
- Stage 3 – Execution of Testing
- Stage 4 – Determining the Outcome
Rafal Los serves as the VP of Security at Armor. He’s responsible for leading the various technical functions associated with designing, developing and delivering next-generation cloud security-as-a-service solutions to our clients. Rafal is also the Founder & Producer of the Down the Security Rabbithole Podcast. He previously worked as the Managing Director, Solution & Program Insight at Optiv Inc.; Principal, Strategy Security Services at HP Enterprise Security Services; and Senior Security Strategist at HP Software.
As an IT security professional, Rafal gained experience in some of the world’s most challenging business environments. His responsibilities included budgets, risk analysis, process creating and adoption, internal audit and compliance strategies. His professional experience has taken him from budding “.com” companies, to a security boutique shop, to one of the world’s largest and most complex enterprises – always meeting challenges head-on and with a positive attitude. He has been the catalyst for change in many organizations, building bridges across enterprises and developing permanent successful strategies for growth and prosperity.2019 business evaluation industry infosec los