Risk Analysis is a Voyage

From the Society of Information Risk Analysts mailing list, I was introduced to the

OWASP Risk Rating Methodology.  If you haven’t read about this methodology, I highly encourage that you do.  There is a lot of material there to talk and think about.

To be completely honest, my first reaction is “what the fudge-cake is this crud?”  It symbolizes most every challenge I think we face with information security risk analysis methods.  However, my pragmatic side steps in and tries to answer a simple question, “Is it helpful?”  Because the one thing I know for certain is the value of risk analysis is relative and on a continuum ranging from really harmful to really helpful.  Compared to unaided opinion, this method may provide a better result and should be leveraged.  Compared to anything else from current (non-infosec) literature and experts, this method is sucking on crayons in the corner.  But the truth is, I don’t know if this method is helpful or not.  Even if I did have an answer I’d probably be wrong since its value is relative to the other tools and resources available in any specific situation.

But here’s another reason I struggle, risk analysis isn’t easy.  I’ve been researching risk analysis methods for years now and I feel like I’m just beginning to scratch the surface – the more I learn, the more I learn I don’t know. It seems that trying to make a “one-size fits all” approach always falls short of expectations, perhaps this point is better made by David Vose:

I’ve done my best to reverse the tendency to be formulaic.  My argument is that in 19 years we have never done the same risk analysis twice: every one has its individual peculiarities.  Yet the tendency seems to be the reverse: I trained over a hundred consultants in one of the big four management consultancy firms in business risk modeling techniques, and they decided that, to ensure that they could maintain consistency, they would keep it simple and essentially fill in a template of three-point estimates with some correlation.  I can see their point – if every risk analyst developed a fancy and highly individual model it would be impossible to ensure any quality standard.  The problem is, of course, that the standard they will maintain is very low.  Risk analysis should not be a packaged commodity but a voyage of reasoned thinking leading to the best possible decision at the time.

-David Vose, “Risk Analysis: A Quantitative Guide”

So here’s the question I’m thinking about, without requiring every developer or infosec practitioner to become experts in analytic techniques, how can we raise the quality of risk-informed decisions? 

Let’s think of the OWASP Risk Rating Methodology as a model, because, well, it is a model.  Next, let’s consider the famous George Box quote, “All models are wrong, but some models are useful.”  All models have to simplify reality at some level (thus never perfectly represent reality) so I don’t want to simply tear apart this risk analysis model because I can point out how it’s wrong.  Anyone with a background in statistics or analytics can point out the flaws.  What I want to understand is how useful the model is, and perhaps in doing that, we can start to determine a path to make this type of formulaic risk analysis more useful.

Risk Analysis is a voyage, let’s get going.