Training and consultancy for testing laboratories.

Archive for the ‘Measurement uncertainty’ Category

Dilemmas in making decision rules for conformance testing

Dilemmas in making decision rules for conformance testing

In carrying out routine testing on samples of commodities and products, we normally encounter requests by clients to issue a statement on the conformity of the test results against their stated specification limits or regulatory limits, in addition to standard reporting.

Conformance testing, as the term suggests, is testing to determine whether a product or just a medium complies with the requirements of a product specification, contract, standard or safety regulation limit.  It refers to the issuance of a compliance statement to customers by the test / calibration laboratory after testing.  Examples of statement can be:  Pass/Fail; Positive/Negative; On specification/Off specification. 

Generally, such statements of conformance are issued after testing, against a target value with a certain degree of confidence.  This is because there is always an element of measurement uncertainty associated with the test result obtained, normally expressed as X +/- U with 95% confidence.

It has been our usual practice in all these years to make direct comparison of measurement value with the specification or regulatory limits, without realizing the risk involved in making such conformance statement.

For example, if the specification minimum limit of the fat content in a product is 10%m/m, we would without hesitation issue a statement of conformity to the client when the sample test result is reported exactly as 10.0%m/m, little realizing that there is a 50% chance that the true value of the analyte in the sample analyzed lies outside the limit!  See Figure 1 below.

In here, we might have made an assumption that the specification limit has taken measurement uncertainty in account (which is not normally true), or, our measurement value has zero uncertainty which is also untrue. Hence, by knowing the fact that there is a presence of uncertainty in all measurements, we are actually taking some 50% risk to allow the actual true value of the test parameter to be found outside the specification while making such conformity statement.

Various guides published by learned professional organizations like ILAC, EuroLab and Eurachem have suggested various manners to make decision rules for such situation. Some have proposed to add a certain estimated amount of error to the measurement uncertainty of a test result and then state the result as passed only when such error added with uncertainty is more than the minimum acceptance limit.  Similarly, a ‘fail’ statement is to be made for a test result when its uncertainty with added estimated error is less than the minimum acceptance limit. 

The aim of adding an additional estimated error is to make sure “safe” conclusions concerning whether measurement errors are within acceptable limits.   See Figure 2 below.

Others have suggested to make decision consideration only based on the measurement uncertainty found associated with the test result without adding an estimated error.  See Figure 3 below:

This is to ensure that if another lab is tasked with taking the same measurements and using the same decision rule, they will come to the similar conclusion about a “pass” or “fail”, in order to avoid any undesirable implication.

However, by doing so, we are faced with a dilemma on how to explain to the client who is a layman on the rationale to make such pass/fail statement.

For discussion sake, let say we have got a mean result of the fat content as 10.30 +/- 0.45%m/m, indicating that the true value of the fat lies between the range of 9.85 – 10.75%m/m with 95% confidence. A simple calculation tells us that there is a 15% chance that the true value is to lie below the 10%m/m minimum mark.  Do we want to take this risk by stating the result has conformed with the specification? In the past, we used to do so.

In fact, if we were to carry out a hypothesis (or significance) testing, we would have found that the mean value of 10.30%m/m found with a standard uncertainty of 0.225% (obtained by dividing 0.45% with a coverage factor of 2) was not significantly different from the target value of 10.0%m/m, given a set type I error (alpha-) of 0.05.  So, statistically speaking, this is a pass situation.  In this sense, are we safe to make this conformity statement?  The decision is yours!

Now, the opposite is also very true.

Still on the same example, a hypothesis testing would show that an average result of 9.7%m/m with a standard uncertainty of 0.225%m/m would not be significantly different from the target value of 10.0%m/m specification with 95% confidence. But, do you want to declare that this test result conforms with the specification limit of 10.0%m/m minimum? Traditionally we don’t. This will be a very safe statement on your side.  But, if  you claim it to be off-specification, your client may not be happy with you if he understands hypothesis testing. He may even challenge you for failing his shipment.

In fact, the critical value of 9.63%m/m can be calculated by the hypothesis testing for the sample analyzed to be significantly different from 10.0%.  That means any figure lower than 9.63%m/m can then be confidently claimed to be off specification!

Indeed, these are the challenges faced by third party testing providers today with the implementation of new ISO/IEC 17025:2017 standard.

To ‘inch’ the mean measured result nearer to the specification limit from either direction, you may want to review your measurement uncertainty evaluation associated with the measurement. If you can ‘improve’ the uncertainty by narrowing the uncertainty range, your mean value will come closer to the target value. Of course, there is always a limit for doing so.

Therefore you have to make decision rules to address the risk you can afford to take in making such statement of conformance or compliance as requested. Also, before starting your sample analysis and implementing these rules, you must communicate and get a written agreement with your client, as required by the revised ISO/IEC 17025 accreditation standard.

Basis of decision rule on conformity testing

There are three fundamental types of risks associated with the uncertainty approach through making conformity or compliance decisions for tests which are based on meeting specification interval or regulatory limits.  Conformity decision rules can then be applied accordingly.

In summary, they are:

  1. Risk of false acceptance of a test result
  2. Risk of false rejection of a test result
  3. Shared risk

The basis of the decision rule is to determine an “Acceptance zone” and a “Rejection zone”, such that if the measurement result lies in the acceptance zone, the product is declared compliant, and, if it is in the rejection zone, it is declared non-compliant.  Hence, a decision rule documents the method of determining the location of acceptance and rejection zones, ideally including the minimum acceptable level of the probability that the value of the targeted analyte lies within the specification limits.

A straight forward decision rule that is widely used today is in a situation where a measurement implies non-compliance with an upper or lower specification limit if the measured value exceeds the limit by its expanded uncertainty, U

By adopting this approach, it should be emphasized that it is based on an assumption that the uncertainty of measurement is represented by a normal or Gaussian probability distribution function (PDF), which is consistent with the typical measurement results (being assumed the applicability of the Central Limit Theorem),

Current practices

When performing a measurement and subsequently making a statement of conformity, for example, in or out-of-specification to manufacturer’s specifications or Pass/Fail to a particular requirement, there can be only two possible outcomes:

  • The result is reported as conforming with the specification
  • The result is reported as not conforming with the specification

Currently, the decision rule is often based on direct comparison of measurement value with the specification or regulatory limits.  So, when the test result is found to be exactly on the dot of the specification, we would gladly state its conformity with the specification. The reason can be that these limits are deemed to have taken into account the measurement uncertainty (which is not normally true) or it has been assumed that the laboratory’s measurement value has zero uncertainty!  But, by realizing the fact that there is always a presence of uncertainty in all measurements, we are actually taking a 50% risk to have the actual or true value of the test parameter found outside the specification.  Do we really want to undertake such a high risky reporting? If not, how are we going to minimize our exposed risk in making such statement?

Decision rule and conformity testing

What is conformity testing?

Conformance testing is testing to determine whether a product, system or just a medium complies with the requirements of a product specification, contract, standard or safety regulation limit.  It refers to the issuance of a compliance statement to customers after testing.  Examples are:  Pass/Fail; Positive/Negative; On specs/Off specs, etc. 

Generally, statements of conformance are issued after testing, against a target value of the specification with a certain degree of confidence. It is usually applied in forensic, food, medical pharmaceutical, and manufacturing fields. Most QC laboratories in manufacturing industry (such as petroleum oils, foods and pharmaceutical products) and laboratories of government regulatory bodies regularly check the quality of an item against the stated specification and regulatory safety limits.

Decision rule involves measurement uncertainty

Why must measurement uncertainty be involved in the discussion of decision rule? 

To answer this, let us first be clear about the ISO definition of decision rule.  The ISO 17025:2017 clause 3.7 defines that: “Rule that describes how measurement uncertainty is accounted for when stating conformity with a specified requirement.”

Therefore, decision rule gives a prescription for the acceptance or rejection of a product based on consideration of the measurement result, its uncertainty associated, and the specification limit or limits.  Where product testing and calibration provide for reporting measured values, levels of measurement decision risk acceptable to both the customer and supplier must be prepared. Some statistical tools such as hypothesis testing covering both type I and type II errors are to be applied in decision risk assessment.

Decision rule and ISO/ IEC17025:2017

Notes on decision rule as per ISO/IEC 17025:2017 requirements

Introduction

The revised ISO/IEC 17025:2017 laboratory accreditation standard introduces a new concept, i.e., “risk-based thinking” which requires the operator of an accredited laboratory to plan and implement actions to address possible risks and opportunities associated with the laboratory activities, including issuance a statement of conformity to product specification or a compliance statement against regulatory limits.

The risk-based approach to management system implementation is one in which the breadth and depth of the implementation of particular clauses is varied to best suit the perceived risk involved for that particular laboratory activity.

Indeed, the laboratory is responsible for deciding which risks and opportunities need to be addressed. The aims as stated in the ISO standard clause 8.5.1 are:

  1. to give assurance that the management system achieves its intended results;
  2. to enhance opportunities to achieve the purpose and objectives of the laboratory;
  3. to prevent, or minimize, undesired impacts or interfering elements to cause failures in the laboratory activities, and
  • to achieve improvement of the activities.

The decision rule as required in ISO/IEC 17025:2017

On the subject of decision rule for conformity testing, the word of ‘risk’ can be found in the following relevant clauses of this international standard:

Clause 7.1.3

When the customer requests a statement of conformity to a specification or standard for the test or calibration (e.g. pass/fail, in-tolerance/out-of-tolerance), the specification or standard and the decision rule shall be clearly defined.  Unless inherent in the requested specification or standard, the decision rule selected shall be communicated to, and agreed with the customer.”

Clause 7.8.6.1:

When a statement of conformity to a specification or standard is provided,  the laboratory shall document the decision rule employed, taking into account the level of risk (such as false accept and false reject and statistical assumptions) associated with the decision rule employed and apply the decision rule.”

Clause 7.8.6.2

The laboratory shall report on the statement of conformity, such that the statement clearly identified:

  1.  to which results the statement of conformity applies;
  2. Which specifications, standards or part therefor are met or not met;
  3. The decision rule applied (unless it is inherent in the requested specification or standard).

From these specified requirements, it is obvious that clearly defined decision rules must be in place when the laboratory’s customer requests for inclusion of a statement of conformity on the specification in the test report after laboratory analysis.  Therefore, the tasks in front of the accredited laboratory operator are how the decision rules are going to be for a tested commodity or product, based on the laboratory’s own measurement uncertainty estimated, and how to communicate and convince the customers on its choice of reporting limits against the given specification or regulatory limits when issuing such conformity statement.

Examples on how to calculate combined standard uncertainty (edited)

Uncertainty calculation

It is very important for anyone interested in the evaluation of measurement uncertainty to fully understand the very basic principles in calculating the combined standard uncertainty.  Let’s look at some worked examples ….

Calculating standard uncertainties for each uncertainty contribution

In evaluating the combined uncertainty of a testing method from various sources of uncertainty, we need to ensure that we work on a platform of standard uncertainties expressed as standard deviations throughout, because in addition to the standard uncertainty (u) values obtained by our own evaluation (Type A uncertainty), we may also encounter the so-called Type B uncertainty contributions which are uncertainty (U) values given by a third party or from experience and other information in different forms.   Read on … How to calculate standard uncertainties for each source of uncertainty

 

Basic discussion on measurement uncertainty evaluation

MU with error

Currently many measurement uncertainty (MU) courses and workshops for test laboratories in this region are run by metrology experts instead of practicing chemists. Some laboratory analysts and quality control personnel have found the outcome after attending the two- or three-day presentations rather disillusion, leaving the classroom with their minds even more uncertain. This is because they cannot see how to apply in their routine works as there are no practical worked examples demonstrated to satisfy their needs…..  Read on  Measurement uncertainty – the very basic

 

Estimation of both sampling and measurement uncertainties by Excel ANOVA Data Analysis tool

Sampling and analysis

Estimation of sampling and analytical uncertainties using Excel Data Analysis toolpak

In the previous blog  https://consultglp.com/2018/08/22/a-worked-example-of-measurement-uncertainty-for-a-non-homogeneous-population/ ,  we used the basic ANOVA principles to analyze the total chromium Cr data for the estimation of measurement uncertainty covering both sampling and analytical uncertainties….

A worked example of measurement uncertainty for a non-homogeneous population

Sampling and analysis

A worked example of MU estimation on a non-homogeneous population

For sampling a non-homogeneous target population such as grain cargo, grainy materials or soil, random positions may be selected and split duplicate samples are taken with duplicate laboratory analysis carried out on each sample received.  This approach will be able to address both sampling and analytical uncertainties at the same time…..

A simple example of sampling uncertainty evaluation

IMG_6916

A simple example of sampling uncertainty evaluation

In analytical chemistry, the test sample is usually only part of the system for which information is required. It is not possible to analyze numerous samples drawn from a population. Hence, one has to ensure a small number of samples taken are representative and assume that the results of the analysis can be taken as the answer for the whole…..