How Repeatable is Quantitative Risk Analysis – European Benchmark Studies

I was at PSAM11/ESREL12 this week, and I plan to write a few posts on topics I found interesting. First though there were some papers that I mentioned during my own talk that are not in the bibliography of the accompanying paper. Normally, this would cause me to start questioning just how  comprehensive my work was – after all, I was making claims about the lack of empirical evaluation of quantitative analysis, and here are three relevant papers that I hadn’t even seen. On the other hand, no one in my research group had heard of the studies either, and they aren’t well known in the whole Probabilistic Risk Assessment community. Important work, bypassing its key audience. Maybe the research councils have a point when they insist on an impact plan in grant applications.

These papers report three investigations into variation in risk assessment. In the first study, 11 teams were asked to quantify the risk presented by an ammonia plant. All teams had access to the same information, but the results were evenly spread over six orders of magnitude. In the second study they tried to pin down what was causing the variation. At several stages during this study they reduced the variation by standardising assumptions and methods, but the teams (using similar source information to the first study) still spread over four orders of magnitude. Once you take into accound the reductions in variation along the way, this was a good (but not completely independent) replication of the first study’s results.

The third study is to do with the ability of reviewers to interpret studies. Multiple reviewers were given the same study, and asked to draw conclusions from it.

Anyway, the three papers are:

S. Contini, A. Amendola, and I. Ziomas, ‘Benchmark Exercise on Major Hazard Analysis. Vol. 1. Description of the Project, Discussion of the Results and Conclusions’, 1991. [Online]. Available: [Accessed: 22-Jun-2012].
K. Lauridsen, I. Kozine, F. Markert, A. Amendola, M. Christou, and M. Fiori, ‘Assessment of Uncertainties in Risk Analysis of Chemical Establishments The ASSURANCE project’, Final Report, Risø, 2002.
L. Fabbri and S. Contini, ‘Benchmarking on the evaluation of major accident-related risk assessment’, Journal of Hazardous Materials, vol. 162, no. 2–3, pp. 1465–1476, Mar. 2009.