ERM, ISO et al: response to Arnold Schanfield


I will attempt to answer your concerns point by point, but hopefully briefly (regrettably, I have failed on the brevity objective – Ed.)

Re focused on the wrong thing: my point is that there is no use in debating the detail of specific approaches if the whole concept behind the approach is flawed.  Take, for example, GRC software.  I am yet to come across GRC software that is useful for analysing alternative assumptions behind Monte Carlo simulations on valuation of debt.  Yet this is something every firm issuing debt should do.  GRC software is usually populated (at least in the dozens of implementations I have observed up close and personal, as it were) using risk workshops where groups of people offer their opinions on rating a risk on a 1-5 or similar scale.  If such ratings are completely invalid (and I am not stating here that they are), then the various strengths and weaknesses of alternative software alternatives to aggregate the assessments is a moot point.  A major local authority (think multiple county) CEO observed to me recently that his biggest risk was his decision to remove a couple of his senior executive team mis-firing.  As he pointed out, this would never make it on a risk register (or even a ‘top 10’ list).  What use GRC software there?

Re “no rigorous attempt to test or even explain the utility of risk management”: it is not incumbent on me to provide sources for that – that’s my whole point; there really aren’t any.  There are dozens of anecdotes (which we will come to) where risk management has been implemented, but almost no robust, methodologically defensible studies demonstrating the efficacy of these COSO / ISO type approaches.  You are the academic; surely you acknowledge it is up to those positing efficacy to demonstrate it.  The few studies that fit that description that have been conducted show very mixed effectiveness (cf. Paape and Speckle, forthcoming but available online in pre-published form).

Re the PwC study: COSO made PwC the authority (quite indefensibly, I believe).  No, I learned almost nothing from reading it, but the same is true of most quasi-evangelical risk or control work published; most of it, just like the PwC study, relies on opinion which will, for a variety of reasons, be systematically biased; it is not evidence of anything, per se, other than the fact of the opinions themselves.  Yes, it did dawn on me that it is self-serving – a point I have made (and that you have agreed with) elsewhere.

The F. Scott Fitzgerald reference I like.  But it is not really the point.  The point is that it is necessary to resolve these to act or to accept action in the face of ambiguity or ‘equivocality’ in Weick’s phrase.  Dilemmas in efficacy of risk management are seldom resolved by intellect alone; propensity to act is a vital and often overlooked ingredient in management efficacy.

My observation about your ‘integration’ point would be that this is how YOU see ISO 31000.  That term is not used in ISO 31000 to refer to forming a holistic view across risk classes; it is used only to talk of integrating risk management into other ‘processes’.  However, most people see ISO 31000 as a process framework, as depicted in its figure 3, which is drawn directly from AS / NZS 4360, which in turn, drew it from earlier work by the Royal Society in the UK, originally drafted by a UK engineer and academic (if my memory serves me rightly).  The ‘linearity’ of the approach arises in two senses: first, and most obviously, in the depiction of risk in figure 3 titled ‘Risk Management Process’.  Clearly the progression from ‘Establishing the context’ to ‘Risk treatment’ is linear and uni-directional.  Secondly, there is a presumption of ability to map risks to objectives that is linear in the sense of being able to represent these stably at a point and, to a certain extent, through time.  However, the reality is a complex ‘soup’ or ‘mess’ of relationships and interrelationships that can and do constantly result in unexpected outcomes.  Here, linearity contrasts complexity in the systems sense.  QED.

Amplification is necessary here.  ISO spend quite a bit of text backing away from the depiction in figure 3; that I will grant you.  But what do most people take away?  Risk management is a process (and a linear one, at that) as depicted in figure 3.  Implementation of a process involves doing . . . figure 3.  Whatever else may occur, that is what is required.  Again, what is written is not wrong, but it sets itself up to be misinterpreted or partially interpreted.  That is an error in construction or drafting.

My experience with COSO ERM runs to sitting down and debating the issues with a panel member pre-publication.  And to working with numerous clients who have attempted to implement it only to find their approach runs out of steam or relevance very quickly.  Not that this is relevant, I have worked with risk in one context or other all my career, initially at NZ Treasury then a major bank then a statutory monopoly accident insurer.  I started working with these approaches in 1993 when asked the question by the CFO of the major telco (owned by a Bell Atlantic / Ameritech consortium) in which I was working: what should it do about risk?  I sought to find out.  In one capacity or another, mainly as an independent consultant or advisory firm partner, I have worked with them – the frameworks – continuously since (except for a three-year spell in emerging markets venture capital).  Most of my work has been undoing the damage they have done.  At a rough calculation, that makes about 24,000 hours working in and around these things (“fieldwork in the trenches,” in your phrase) – fortunately not all of it having to think about COSO or AS/NZS 4360 or ISO or equivalent standards or other ‘complete solutions’.  The short answer is ‘yes’.  I am a practitioner and advisor, not an academic.

As an aside, I do not consider it either attractive or edifying to question experience of a discussant in these areas; we all have experiences on which to draw.  A person can work his whole life without ever questioning the validity of the basis of the work he does.  As Bertrand Russell pointed out: “most people would rather die than think.”  Stating that one does think is no more valid; as you may recall, Enron’s catch-line was “Ask why.”

Re Alpaslan’s points, these are his observations; I would not seek to contradict them in his experience; being a utility theorist by training, I am well versed in ‘the incomparability of interpersonal utility preferences’ –  you cannot validly compare how much he likes something with how much I like it.  However, I can provide counter-examples for each of his observations from risk and ERM implementations in which I have been involved in one way or other remedially.  But that neither proves nor disproves anything.

You have cited Mars, BHP Billiton and Hydro One.  Of these, I have only engaged with BHP Billiton, and then only at NED and secretariat level.  I do not doubt that they or “100 other examples” have been “successful in implementing ISO 31000.”  That does not tell us anything about the efficacy of the approach in practice in those firms, only that they have determined that they have embarked on the process of implementing a framework and have deemed themselves at some arbitrary (but potentially entirely valid) point to have done so to their satisfaction.  Was their management risk after that more effective?  Against what criteria?  What did not work as anticipated?  What failed?  What risks were not managed effectively?  What happened to their corporate value subsequently, relatively to similar entities?  Was their subsequent performance attributable to their risk management actions or risks averted?  And so on.  Most importantly: will it remain so?  These questions are neither answered, nor even addressed in most analyses of risk management.  But they did implement successfully, at least according to those whose performance was being judged, at least in part, according to whether or not they implemented successfully the chosen risk framework.

Here, you cannot have your cake and eat it too.  Commending such firms – the 103, if you will – is exactly what PwC does in its report with the firms it interviews.  No causal link is offered in terms of effectiveness.

As I indicated in the other blog stream in which you asked me the question, I have not read your other blogs.  However, I find your observation that you “find it odd that I (sic) would not do some research into the subject matter before making such statements, ” referring to reading your blogs, a touch . . . hmmmm . . . bold.  It is all the more an astonishing statement coming from an academic who, in another context, called for ideas to have been tested in peer-reviewed journals; consistency is everything.  Reading other people’s blogs is not research; this is not research.

I hope these comments have been sufficiently specific.  I remain happy to debate the issues, but let’s keep it to issues and evidence.

I retain a completely open mind to evidence of ISO or COSO effectiveness or effectiveness of other linear risk and control approaches, but reserve the right to pick to pieces any purported evidence if I find it flawed or unconvincing!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s