Training and consultancy for testing laboratories.

Archive for February, 2020

Uncertainty of measurement – “Bottom-up” vs “Top-down”

At the recently concluded Eurachem/PUC training workshop on “Accreditation of analytical, microbiological and medical laboratories – ISO/IEC 17025:2017 and ISO 15189:2012”, the following important pointers were noted during the presentation of Dr Steve Ellison of LGC UK on the to subject : Measurement Uncertainty – “Bottom-up” vs “Top-down”:

  1. Measurement uncertainty assessed in analytical chemistry is either through the use of the Law of Propagation of Uncertainty from uncertainty budgets (or inputs or contributors) as per GUM (bottom-up) method, or adopting the method performance (or validation) data (top-down);
  2. Using the GUM approach with a mathematical model, the laboratory is to assess and sufficiently quantify significant uncertainty contributors in the test procedures.  This can be done by (a) using descriptive statistical data through repeated experiments (Type A), or (b) any other means, such as certificates of analysis by a third party, theory or professional judgement (Type B);
  3. It has been stated that testing laboratories tend to underestimate measurement uncertainty using the GUM method in almost measurement fields, as one cannot comprehensively identify and quantify all important uncertainty inputs;
  4. Use of any one of the top-down approaches with the use of validation data is a better bet in the evaluation of measurement uncertainty because the actual dispersion of test results in extended experiments can be observed; the major uncertainty source data can come from (a) long term precision (intermediate reproducibility), (b) bias uncertainty based on reliable certified reference materials, and (c ) any other additional important effects which are not part of the method’s mathematical equation;
  5. By definition, uncertainty is a range which includes the true value.  Therefore, any significant bias should not be ignored.
  6. Empirical methods are operationally defined.  In the top-down approach, relevant  reference material should be used to estimate laboratory bias as an input of uncertainty.  In this case, only matrix bias is to be taken care of and method bias is not relevant.
  7. Eurachem opines that the bottom-up GUM method is appropriate for metrology laboratories, whilst the top-down approaches are best for testing laboratories.

Expressing MU for qualitative testing?

I would like to share some of the ideas picked up at the 2-day Eurachem / Pancyprian Union of Chemists (PUC) joint training workshop on 20-21 February 2020, titled “Accreditation of analytical, microbiological and medical laboratories – ISO/IEC 17025:2017 and ISO 15189:2012”, after flying all the way from Singapore to Nicosia of Cyprus via a stop-over at Istanbul.

Today, let’s see whether there is a requirement for an expression of uncertainty in qualitative analysis. In other words, are there quantitative reports of uncertainties in qualitative test results?

Qualitative chemical and microbiological testing usually fall under the following binary classifications with two outcomes only:

  • Pass/Fail for a targeted measurand
  • Positive/Negative
  • Presence/Absence
  • “Above” or “Below” a limit
  • Red or yellow colour
  • Classification into ranges (<2; 2 – 5; 5 – 10; >10)
  • Authentic or non-authentic

Many learned professional organizations have set up working groups to study on expression of uncertainty for such types of qualitative analysis for many years and have yet to officially publish guidance in this respect.

The current thinking refers to the following common approaches:

  1. Using false positive and negative response rates

In a binary test, we can get result to be a true positive (TP) or a true negative (TN).  There are two kinds of errors associated in such testing, giving rise to a false positive (FP) or a false negative (FN) situation.  

A false positive error occurs in data reporting when a test result improperly indicates presence of a condition, such as a harmful pathogen in food, when in reality it is not present (being a Type I error, statistically speaking), while a false negative is an error in which a test result improperly indicates absence of a condition when in reality it is present (i.e. a Type II error).

Consequently, the false positive response rate, which is equal to the significance level (Greek letter alpha, α) in statistical hypothesis testing, is the ratio of those negatives that still yield positive test outcomes against the total observations. The specificity of the test is then equal to 1−α.

Complementarily, the false negative rate is the proportion of positives which yield negative test outcomes with the test.  In statistical hypothesis testing, we can express this fraction a letter beta β (for a Type II error), and the “power” or the “sensitivity” of the test is equal to 1−β.

See table below:

  2.     Alternative performance indicators (single laboratory)

The alternative performance indicators are actually reliability measures involving several formulae, as summarized below:

Reliability Measure Formula
False positive rate FP/(TN+FP)
False negative rate FN/(TP+FN)
Sensitivity TP/(TP+FN)
Specificity TN/(TN+FP)
Positive predictive value TP/(TP+FP)
Efficiency (TP+TN)/(TP+TN+FP+FN)
Youden Index Sensitivity + Specificity – 1
Likelihood ratio (1-False negative rate)/False positive rate

There are many challenges to evaluate qualitative “uncertainty”.  Although the idea of estimating uncertainty for such binary results is sound, the most problematic one is how to collect hundreds of experimental data in order to make reasonable statistical estimates for low false response rates.  Another challenge is how to confidently estimate the population probabilities in order not to be bias.  A sensible suggestion is to ask laboratories to following published codes of best practices in qualitative testing where they are available and to ensure the conditions of testing are under adequate control.

At this moment, quantitative (i.e. numerical) reports of uncertainties in qualitative test results, involving strict metrological and statistical calculations, are not generally expected by the accreditation bodies.