Welcome back. In the last few articles we discussed how to set up an evaluation that’s both defensible & provides value to your immediate needs, and, maybe, more clearly defined what good pizza is. In this edition, I’ll address the importance of the process of the execution of testing.
We’ve already talked through defining the problem, and the success criteria. Now it’s time to actually run tests that will give you results you can defend and live with. The good news is, there are rules that govern this sort of thing, and they’re pretty good. I’ll break the sub-components of testing down into three parts: define the testing framework, execute with consistency, trust the results. Let’s talk through those here.
Define the Testing Framework
The key to testing is making it fair and equitable. Everyone you bring to the table should have a fair shot of winning your approval within the rules of engagement. Whoops! I almost forgot Rules of Engagement.
So Step 1, defining Rules of Engagement. When you bring 3 widgets into a test, what will the rules be? Will you work off of a stock configuration? Will you allow tuning? Will the participants be given knowledge of the testing environment? How much knowledge? What happens when something fails, does the participant get a tune and retry? These and a billion more are the right questions to answer ahead of time. Everyone in your evaluation should have to play by the same rules, or the thing is a farce.
So now that you have your rules of engagement, it’s time to build a test harness. If you’re evaluating a network-based intrusion prevention platform, you’ll want to make sure you have a mock network segment designed that someone can plug into that accurately mirrors your real production environment. Otherwise, what’s the point?
I remember when I was selling AppSec testing tools and the customer would give us their “apps” to demo on. Our tool was particularly good at .Net code, while a competitor may have excelled at Java apps. That said, if the customer threw together some demo app from the Internet to assault and test that didn’t mirror their environment and the types of apps their developers were building, what the heck was the point? The truth is in that case the customer would buy the best damn race car to run on their mudding course. Whoops.
Consistency in Execution of Testing
So here’s the thing. There are a few rules to getting good results.
First, you have to run the test more than once to prove that the first and second (and maybe third) tests weren’t a lark. Then, testing must be scripted, so that anyone (that is not you, because you likely have a horse in the race) can do it. Let someone else run the tests, that has no skin in the game. If your tests are designed so that only you can run them, you’ve failed. Lastly, you must have documentation of your results. If you need to record the tests, their outputs, or whatever, to prove yourself, then that’s what you do. “Trust me, this one won” is a horrible idea. You will not like the end results. Consistency is king. Do it. Be good at it.
Trust the Results
This one is hard. Especially when you go into an evaluation thinking you really like widget A because your friends work there, and the result is that widget B is cheaper and more effective at solving the problem you’ve defined. Human nature makes you want to add “gut feel” to evaluations in those cases. Don’t!
When you’ve set up the tests properly, consistently, and fairly – you have to trust the results. There is no other way to operate. Testing yields impartial, empirical results. You must trust that those are the correct results.
Document your results, present them impartially, and get a group to validate your understanding and outcome. That’s how you’ll get to a state where you have a way to solve problems, rather than pick favorites.
Here are my tips for a framework for your execution of testing:
- Must mirror your environment, for production, as closely as is reasonable
- Must expose the thing you’re testing to as much ‘real’ scenario as possible, because synthetic situations are inherently… synthetic
- Don’t alter your Rules of Engagement when your favorite product won’t work within your predefined rules
- Script everything, so it’s repeatable
- Document everything, so there is evidence of your results
- Run the tests more than once, by more than one person
Off you go. Don’t over-complicate it. This part is pretty simple. I could probably write an entire book about it, but I’m making a point of the fact that it needs to be concise and factual. Hopefully it helps you! Soon, we’ll wrap this up in the last and final section, “Stage 4 – Determining the Outcome”.
Credit: Lego Executioner Photo by Johnson Cameraface
Articles in this series:
- The Evaluation – Four Phases to Finding “Better” Solutions
- Stage 1 – Definition of the Problem to be Solved
- Stage 2 – Definition of Success Criteria
- Stage 3 – Execution of Testing
- Stage 4 – Determining the Outcome
Rafal Los serves as the VP of Solution Strategy at Armor. He’s responsible for leading the various technical functions associated with designing, developing and delivering next-generation cloud security-as-a-service solutions to our clients. Rafal is also the Founder & Producer of the Down the Security Rabbithole Podcast. He previously worked as the Managing Director, Solution & Program Insight at Optiv Inc.; Principal, Strategy Security Services at HP Enterprise Security Services; and Senior Security Strategist at HP Software.
As an IT security professional, Rafal gained experience in some of the world’s most challenging business environments. His responsibilities included budgets, risk analysis, process creating and adoption, internal audit and compliance strategies. His professional experience has taken him from budding “.com” companies, to a security boutique shop, to one of the world’s largest and most complex enterprises – always meeting challenges head-on and with a positive attitude. He has been the catalyst for change in many organizations, building bridges across enterprises and developing permanent successful strategies for growth and prosperity.evaluation highlight industry infosec los