But does it work? ISO 31000 conference in Paris in May

In search of something other than Kool-Aid to refresh in the spring warmth in Paris. 

I have been reviewing the conference programme for G 31000 in Paris in May and I am a little bemused.  ISO 31000 was introduced in 2009.  The basic principles have been around since before the original AS/NZS 4360 in 1995, which drew on earlier UK-based work in engineering and risk assessment.  I searched on the programme for the slot that discussed ‘evidence that ISO 31000 is effective at improving firms’ management of risk’ but couldn’t find it.

With more than 2 years and 17 years between them, respectively, these two standards should by now have generated a solid body of research on efficacy.  At least, there should be evidence of relative strengths and weaknesses; that is, not stories of effective implementations, but evidence that the programmes in place have improved corporate performance by managing effectively the unexpected and potentially catastrophic.  On the programme in Paris, I would have expected at least to see ‘Review of evidence on successes and failures in practice and how we can improve the standard’.  No, again.

Looking through a downloaded provisional programme, I see one speaker who referred to ISO 31000 as “a guideline, not a standard.”  A sensible view, but I note he is no longer appearing as originally advertised.  Tim Leech, always balanced, remains, but where are the voices of criticism or of doubt?  Where is the objectivity?

Risk management is not a religion.  It is a management art.  We need to treat it with the skepticism and dispassionate enquiry we do any developing area of human knowledge.  Fundamental to that is the presentation of efficacy.  Not the presumption of efficacy or reasoning that it should be effective; not the evidence that you can implement it easily, but that it works sustainably in practice at its stated objectives.  Where is the critical evidence that ISO 31000 actually works?  Not in Paris in May, it would seem.  Expect the odd “Hallelujah!” from the benches.

Rather than disciples picking their corners – ISO or COSO or AS/NZS as was or whatever – we need systematic enquiry from both the academic and practitioner communities jointly and severally about what works.  ISO 31000 has flaws and some pretty fundamental ones.  There, I’ve said it.  There is little or no evidence available that it is effective in practice (which may not mean that it is not).

What we need is a more realistic and naturalistic approach that rejects lazy presumptions like the need for a common language that is within the control of the guardians of risk semantics.  Or ideas like risk = probability x impact.  Or risk matrices at all.  Or that we can create an ‘effective risk culture’ or that there even is such a thing.  We need more thought, a greater recognition of the value of reflection and more humility about our ability to understand culture or predict the future or to prepare meaningful reductionist models of reality.  A conference looking at how to do these things would be worth attending.  If I want religion, I’ll go to church.

Advertisements

15 thoughts on “But does it work? ISO 31000 conference in Paris in May

  1. Peter:

    You raise a very good point. It isn’t just the IS0 31000 crowd that doesn’t spend enough time objectively assessing whether generally accepted andwidely practiced paradigms actually work as well as they should/could. It applies to the whole of the accounting and audit professions since they were created. Far more time and money needs to be spent studying root causes of accounting and auditing failure. Regulators keep throwing mud at the wall hoping some will stick with little regard for studying whether the expensive “solutions” they mandate in laws and regulations are worth the cost. In my 30 years in the auditing profession I can’t recall a single serious research study with a decent sample to try and identify root causes of external audit failure. I am also not aware of a single serious study by the IIA to research why thousands of internal audit departments have missed or been unable to communicate serious risk exposures in their organizations to their boards.

    I am going to Paris to the first international ISO 31000 conference in the hopes I can help convince the ISO 31000 community to not make the same mistakes made by COSO, the IIA, and external audit profession of creating yet another “feel good club”. “Feel good clubs” spend far too much time promoting why things are great and very little serious time studying whether major changes are needed to better serve stakeholders and taking aggessive and sometimes radical steps to address failings when needed. I have recommended to the IIA CEO, Richard Chambers, that a full conference should be dedicated each year to speakers that want to promote radical change in the internal audit profession. Unfortunately, all too many conferences today are presentations on topics and approaches that have been around for many decades and, unfortunately, often haven’t really worked very well. This needs to change. It’s time that we actually practice the tough and objective assessment approaches on ourselves that we claim are cornerstones of the audit profession.

  2. Tim

    Thanks for your comment.

    This is a point you have made before, repeatedly, and more formally in other settings. It bears repeating by all who encounter the problem, over and over and over and . . . until the committees who put money behind these initiatives realise they have an intellectual and a practical responsibility to verify their assertions of efficacy. It is as if we skipped the whole Enlightenment through to Thomas Kuhn and progressed from the Dark Ages to the wisdom of ISO (or COSO or . . . ). It’s almost enough to make me quote Francis Bacon.

    You are speaking at the conference in Paris. I have no doubt you will raise the concerns.

  3. I have struggled to find objective materials on ISO 31000 implementations. Some LinkedIn site members were able to point me to a few examples, but by and large there is a dirth of materials as you point out. We will all need to work toward education, training and sharing to make this work better at/for our companies.

    Mike

  4. MIke

    Thanks for your comment. How do you see this happening?

    As Tim points out above, this is not the only issue in risk and control in which we observe this phenomenon. Perhaps if more academics served as skeptical and objective documenters and observers of reality and effectiveness, we would move ahead. Far too many seem to have become poster-bearers for one or other approach without engaging in valid empirical research or rigorous questioning of assumptions.

    I see this as a potential pressure point for pushing for greater empirical research and analysis and, well, realism. Do you have any ideas? How can we move the agenda forward?

    • I would think the academics would have been all over this for a long time. North Carolina State and Harvard here in the US come to mind and I am sure many others in Australia, New Zealand, South Africa, Canada and UK. There are ISO working groups and various associations involved but not sure they are independent as they charge for publications, training etc. and I am not sure what non-profit is any more. There are open source movements?

      Mike

  5. My understanding is as follows:

    NC State: any work done here is survey-related and aimed at the parties that have developed and implemented the frameworks. Unreliable.

    Harvard: Anette Mikes has done some interesting review work but it looks at case studies rather than across significant numbers of firms.

    Australia / NZ / South Africa: No, not that I am aware of.

    Canada: No . . .

    UK: No . . .

    One paper (Hoyt & Liebenberg, 2008) notes in its abstract: “Despite the heightened interest in ERM by academics and practitioners, there is an absence of empirical evidence regarding the impact of such programs on firm value.” This paper looked at ERM in insurers.

    A recent paper by a couple of Dutch academics tackles the issue, but they note some methodological problems relating to their data.

    Not really a happy picture.

  6. Peter
    I think the question needs to be asked in two parts. A) how well is risk management implemented and B) how effective is it on improving business performance. The latter part is more difficult to prove since you cannot easily create a control organisation or project. Unsurprisingly therefore most of the comment and research is targeted at answering the first part of the question, based on an assumption that if this can be proven, the answer to the second part is better organisational performance. Anecdotal evidence exists on the factors which contribute to business failure, for example the results of the recent financial crisis. Whilst the lack of experimental controls and repeatable results erodes the scientific basis to say how instrumental a lack of risk management was in contributing to failure, perhaps this is the best we can expect.

    In the project world, there is a fair amount of anecdotal evidence of contributory factors to project failure and poor risk management seems to be one of them. Perhaps this is a good area to start, since what is true for a project organisation, should in principle hold true for a larger enterprise.

    • Simon

      I wholly concur that understanding failure, causes and correlates thereof, in the project world would be enormously useful. Certainly more useful than the current PRINCE-type, register-based approaches that seem currently to be in favour. I find CObIT much more valuable, and SEI-CMM better again. But these frameworks all suffer from a common limitation – the reliance on and event-based survey of what we already know rather than looking at project performance and success types and making decisions based on probability-adjusted expectations of outcomes. In effect, this is the reverse of what you suggest – the ultimate organisational protection approach (precautionary and signalling capital) scaled down to the project level.

      Re your comment “Anecdotal evidence exists … best we can expect,” I like your line of thought. Over the next few days, I’ll be preparing another blog based on a paper by an American practitioner and academic, Kathleen Locklear. The original is here. Have a read and let me know what you think.

  7. Peter, I am also developing my practice to help companies preserve and create value by among other things managing risk. But, there are other management aspects of value creation and preservation that ISO 31000 or COSO ERM just do not address like innovation, organizational agility, use of technology and investment optimization. I find that most CEO’s and BOD’s talk about value creation to their stakeholders. Evolving the practice, one client at a time.

  8. That’s an interesting point Michael. I’ve recently been looking at the link between risk management and innovation. Stephen Shapiro, a US innovation guru has identified the first step in building an effective, repeatable innovation culture is by implementing a portfolio of challenges. He says,”the first step in creating a culture of innovation is to surface, identify, and codify challenges. And then
    you must become masterful at valuing, prioritizing, and framing these challenges.” Sounds a lot like risk management to me, and it’s the effective solutions to these challenges, when implemented, which preserve and create value.

  9. PeterB it would be great if you can point out what are the fundemental flaws of ISO31000. MichaelJa and PeterG, should the ISO or any other Risk guideline and/or framework address innovation, organizational agility, use of technology and investment optimization? IMHO the ISO give a common risk language to consider ( I would admit nt everyone perceive it or understands those definitons the same) and process guidance how to implement your risk management infrastructure. The ISO people is now busy with ISO31004 to give implementation guidance…Will it be the last word on subject? I don’t think so!

    • Hi Julian, looking foward to ISO 31004 as the lack of real world evidence (training, tools, research, software, whitepapers) is quite a surprise for something that has its roots 17 years ago and now almost 3 years in the current state. In fact, I have been waiting for several months now for Active Risk Manager to show their ISO 31000 solution they have written about.

      My point is that some of the most ardent ISO 31000 evangelists and you know who they are have suggested that ISO 31000 is all you need to model/run your business. I do not believe this. Managing risk and associated governance have an important role. But innovation, agility, investment optimization and change management to name a few – are also critical organization capabilities for success.

      PeterG – It sounds to me like value management. What do we need to do to meet these challenges to achieve and succeed. I am afraid that risk management in many companies is about events with negative outcomes.

      Mike

  10. Hi Julian, I think depending on your interpretation of ISO31000 it already does address innovation and investment optimisation (though certainly not explicitly). This is dependent on the context setting phase. I look at this as articulating the organisations business goals which would optimise value i.e. a description of the ideal end result/outcome. This is very different to an objective which is SMART. Risk identification to objectives has a tendency to be event or time based and as Mike says only talks to the downside.

    By framing risk as the variance to this optimum outcome, and risk management as the identification, analysis and response to challenges which are preventing the attainment of this outcome (threats and opportunities to that outcome can then be directly compared for value), management should then constantly be striving for new ways to achieve better performance, and with thanks to the risk ranking system they should concentrate on solving those challenges which will deliver the most value (in a systematic rather than intuitive way).

  11. Pingback: I love Paris in the springtime, but not this week « The risk debate

  12. New to 31000, spent the last 33 years doing nonprofit risk management, etc., etc. Great post…too many times solutions are thought up but not thought out (e.g., PPACA, among a long list of others). Thanks, and will visit again.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s