I love Paris in the springtime, but not this week

More on ISO 31000, efficacy of risk management and research

As a young student of French, I spent several months on a student exchange in Tahiti, French Polynesia, in 1980.  It was one of the most idyllic experiences of my life and I loved every minute of it.  Although I was well on the way already, the experience turned me in to a life-long francophile.  But that love has been put to the test more than once, and not only at rugby World Cups.

In July 1985, I was a final-year undergraduate student in international relations in Wellington, NZ’s capital city, which has a justified reputation as being windy and cold in the winter months.  I have no recollection of what I was doing on the evening of 10 July of that year, but events that occurred elsewhere in the country are etched in the memory of every New Zealander of then-sentient age.  Late that evening, at close to midnight, agents of the French secret service (DGSE) bombed Greenpeace’s flagship, the Rainbow Warrior, that was tied up in Auckland, due shortly to sail to Moruroa Atoll in French Polynesia to attempt to disrupt the French nuclear testing programme.  In the bombings, a 35-year old Portuguese photographer, Fernando Pereira, was killed.  As much as the act itself, the French government’s subsequent realpolitik stained France’s reputation in New Zealand and further afield.

Though I’m no Cole Porter fan, I love Paris in the springtime (and I am an Ella Fitzgerald fan).  But I am not there this week for the ISO 31000 international conference.  In a blog in April, I expressed my concern at the lack of balance in the programme for the conference and the apparent absence of contrary views.  My intervention, while deliberately provocatively written, was met with decidedly mixed reactions.  From some, there was support for the idea that evidence of efficacy should be sought and assembled.  Some noted the difficulties of collection of evidence in social sciences.  Some, however, seemed to reason that the best form of defence was attack: in the discussion thread that followed, direct, ad hominem attack ended up displacing reason all too easily, as it so often does.

In the original blog, I advocated research on the efficacy of risk management approaches generally and ISO 31000 in particular: “we need systematic enquiry from both the academic and practitioner communities jointly and severally about what works,” I wrote.  The problem is that such “systematic enquiry” is no easy matter.  What would it look like?  Is it even possible?

It was John Stuart Mill who, in A System of Logic published in 1843, first outlined formal rules for induction.  Mill introduces his formulation very simply:

The simplest and most obvious modes from among the circumstances which precede or follow a phenomenon, those with which it is really connected by an invariable law, are two in number.  One is by comparing together different instances in which the phenomenon occurs.  The other is by comparing instances in which the phenomenon does occur with instances, in other respects similar, in which it does not.  These two methods may be respectively denominated the Method of Agreement and the Method of Difference.

In illustrating these methods, it will be necessary to bear in mind the two-fold character of enquiries in to the laws of phenomena: which may be either inquiries in the cause of a given effect or into the effects or properties of a given causes.

From these relationships, Mill derives five ‘canons’, adding to his methods, (iii) Joint Method Agreement and Difference, (iv) Method of Residues and (v) Method of Concomitant Variation.  Mill’s schema was hardly the final word on the causal relationships or the nature of correlation and causation.  But from it, scientific method was born.

The problem is that, in assessing the efficacy of risk management in modern organisations, we are not dealing with physical science.  In management, we are dealing with people, with human behaviour, that behave altogether less predictably than the physical world.  But that does not mean at all that Mill’s insights are not valid.  Nor does it mean that we must address the problem without the assistance of valid research methods.

Any discussion of research of business practice in the practitioner community seems to attract a wealth of proponents of surveys; because of the potentially large number of respondents a survey can reach, they are often viewed as rigorous.  However, survey-based research will not reveal utility or efficacy.  At best, it will reveal opinions about utility and efficacy.  There are plenty of opinions already.   Most importantly, surveys cannot isolate cause and effect.

So where can we turn for robust social science method?  There are many candidates.  One of my favourites is the eminent American political scientist James Q. Wilson, who died of leukemia earlier this year, aged 80.  Wilson’s work was invariably empirical and thoughtful.  From his work on crime and deprivation in New York to his analysis of the performance and function of bureaucracy (which is the work I first encountered), he dealt with the world as it was, rather than as we might wish it to be.  This is exactly the sort of dispassionate enquiry we need in relation to the performance of systems of risk management in corporate settings.

In a landmark article “Thinking about crime” in Atlantic Monthly in 1983 discussing analysis of the deterrent effect of punishment on criminality, Wilson wrote:

Whenever we try to discover a relationship between hard-to-measure factors that operate deep inside a complex social structure, we are well advised not to rely on any single method of analysis, and we are particularly well advised not to rely on statistical studies using aggregate data. We should attack the problem from a number of angles, using different kinds of data and various methodologies. Above all, we should look at what happens to individuals . . .

In forming his view, Wilson discussed what he called the “weight of evidence” which comprised “statistical analyses, evaluations of experiments and quasi-experiments, and studies of individual behavior” including individuals’ behaviour in groups.

While recent work on structural equation modeling provides a quantitative base for analysis of causality (e.g. Pearl, 2000), in the social sciences, statistical analyses are fraught with problems. Never (at least to my knowledge) has this argument been more cogently presented that by the great liberal economist Friedrich von Hayek in his Nobel Prize lecture in 1974.   Discussing evidential standards in social sciences, specifically economics, Hayek stated:

the social sciences, like much of biology but unlike most fields of the physical sciences, have to deal with structures of essential complexity, i.e. with structures whose characteristic properties can be exhibited only by models made up of relatively large numbers of variables.

In some fields, particularly where problems of a similar kind arise in the physical sciences, the difficulties can be overcome by using, instead of specific information about the individual elements, data about the relative frequency, or the probability, of the occurrence of the various distinctive properties of the elements.  But this is true only where we have to deal with . . .  “phenomena of unorganized complexity,” in contrast to those “phenomena of organized complexity” with which we have to deal in the social sciences.  Organized complexity here means that the character of the structures showing it depends not only on the properties of the individual elements of which they are composed, and the relative frequency with which they occur, but also on the manner in which the individual elements are connected with each other.   In the explanation of the working of such structures we can for this reason not replace the information about the individual elements by statistical information, but require full information about each element if from our theory we are to derive specific predictions about individual events.

So, how about Wilson’s second category – experimentation?  It is worthwhile to review the purpose of experimentation (as defined by Herbert Simon):

It is a procedure for systematically varying the conditions that hold, and hence for producing observations of situations in which not all the conditions in a complete set of laws can hold simultaneously.  The observations permit an empirical choice to be made between the two sets of laws that determine the same state.

In other words, experimentation allows us to observe differences in the strength of cause and effect relationships.  But genuine opportunities for experimentation are limited in operational behavioural contexts such as working organizations, mainly due to the phenomenon of “organized complexity” discussed by Hayek.  While there are many lessons on behaviour to emerge from experiments (e.g. prospect theory), the task in organisational research is to understand their applicability in theory and implications for practice.

By the time Wilson wrote “Thinking about crime,” the study of individual and group behaviour had been transformed by the work of two sociologists Barney Glaser and Anselm Strauss (1967).    Their formulation of “grounded theory” – the process of ‘discovery’ of theory from qualitative data – has provided the methodological basis for qualitative research since its publication.  While not without its own controversies, grounded theory provides a robust framework for qualitative social research.  A key element of Glaser’s and Strauss’s research schema was the need for what they called “theoretical sensitivity” and related the danger that researchers’ pre-existing knowledge and preference may influence the research process.

Over the same period, the growing influence and popularity of business schools was seeing a related strand of research emerge: the case study.   In case studies, the debt to grounded theory is clear and often (though not always) acknowledged.  Perhaps reflecting the on-going preference for ‘scientific’ research, business school professors often seem to ignore (if they are even aware of) the caution urged by Hayek.  However, with the case study, a new and rich research strategy opened up.  The case study, as Yin (1981) noted:

attempts to examine (a) a contemporary phenomenon in it real-life context, especially when (b) the boundaries between phenomenon and context are not clearly evident.

As Yin observed, “experiments differ in that they deliberately divorce a phenomenon from its context.”

Our choice of research method to examine the efficacy of risk management practice in general and specific methods or standards in particular will owe much to the insights of these writers and academics.   While it is likely to involve each of the methods identified by James Wilson, it should lean heavily towards qualitative study of individuals’ and groups’ behaviour, validated and “triangulated” (Eisenhardt, 1989) against quantitative data where available, built around a series of case studies.  As I said in the original blog, it will need to examine “what works, what does not and why.”

At the ISO 31000 conference this week, there are few examples of conflicting or contrary views on offer.  As I asked in my original blog on the conference “where are the voices of criticism or of doubt?  Where is the objectivity?”  And yet, understanding what Kathleen Eisenhardt (1989) termed “the conflicting literature” is critical to robust qualitative research.  Is it realistic to expect the pro-ISO 31000 (or, for that matter, the pro-COSO) communities to examine critically their own, preferred approaches?

Any risk management approach must be robust to the ‘Rainbow Warrior’ problem; what Nassim Taleb called the ‘Black Swan’ and to Bertrand Russell’s chicken’s problem – the limits of induction.  Before July 1985, I sincerely doubt that “sabotage by agents of an allied government” would have been on the radar of the port authority in Auckland.  And yet the approach to risk management that emerged from that part of the world – via AS/NZS 4360 : 1995 – and has morphed into ISO 31000 is no more likely to assist a user to identify such a possibility than any other.  While that does not vitiate ISO 31000 relative to other process-based approaches to ‘comprehensive’ identification of risk, it does call in to question the entire construct of a standard for risk management.  But, without properly constructed and executed research, the debate will continue to degenerate to assertion and name-calling; we will continue to chase our tails: so much more ‘he said, she said’.

The responses to the discussion thread asking if a systematic enquiry is needed give reason for some optimism.  While the lack of specific suggestions of methods probably reflects the disciplinary background of risk managers and auditors more widely, there was considerable enthusiasm for the idea – albeit tempered with the usual concern over the validity of qualitative research methods.  While there were clear flaws in the inferential logic in some of the posts, there were plenty of receptive noises.  However, the preference seemed to be for a research question addressing ISO 31000 versus other formulaic approaches and for survey instruments.

To be useful, any qualitative research will need to consider Mill’s Method of Difference and rules of inference, Wilson’s variety of approaches to social research, Hayek’s caveats on quantitative research, Glaser’s and Strauss’s grounded theory, Yin’s boundaries between phenomenon and context and Eisenhardt’s exhortation to consider ”conflicting literatures.”  That is, it will involve cross-firm case studies combining qualitative and quantitative data in an organized and systematized research protocol using experienced social science researchers with a background in risk management and organisational behaviour.  Easy, then.

If I have three wishes for the ISO 31000 conference this week, they would be, first, that the participants will emerge with more open minds to the flaws of the Standard and to the logical fallacies of a standard-based approach to what I have described elsewhere as ‘irreducible uncertainty’.  Certainly, the issues around what Glaser and Strauss called ‘theoretical sensitivity’ necessitate an open mind.

Second wish:  that the participants resolve to adopt a more realistic approach to the use of language.  As Orwell observed,

If thought corrupts language, language can also corrupt thought.

From following the discussions leading up to the conference, the problems associated with the contorted and rule-bound use of language in the Standard still loom large.  John Adams has addressed these issues eloquently.

Thirdly, I hope that the participants enjoy Paris in the springtime and travel safely.  And enjoy tonight’s dinner.

There is much to do.  Why not start with a good meal?