Training and consultancy for testing laboratories.

Archive for the ‘Top-Down Approach’ Category

The concept of measurement uncertainty – a new perspective

Since the publication of the newly revised ISO/IEC 17025:2017, measurement uncertainty evaluation has expanded its coverage to include sampling uncertainty as well because ISO has recognized that sampling uncertainty can be a serious factor in the final test result obtained from a given sample ……

The concept of measurement uncertainty – a new pespective

Bottling_edited

ISO FDIS 17025:2017 – Impacts on accredited laboratories

ISO FDIS 17025:2017 – Impacts on accredited laboratories

As noted in my previous article on the final draft international standard FDIS 17025:2017 (https://consultglp.com/2017/10/30/iso-fdis-170252017-sampling-sampling-uncertainty/ ), the current ISO/IEC 17025:2005 version widely used by accredited laboratories around the world will soon be replaced by this new standard, expected to be published very soon.

The ILAC (International Laboratory Accreditation Cooperation), a formal cooperation to promote establishing an international arrangement between member accreditation bodies based on peer evaluation and mutual acceptance with a view to develop and harmonize laboratory and inspection body accreditation practices, has recommended a 3-year transition to fully implement this new standard from the date of its publication. At the end of the transition period, laboratories not accredited to the ISO/IEC 17025:2017 will not be allowed to issue endorsed test or calibration reports and will not be recognized under the ILAC MRA terms.

Today, there are over 90 member accreditation bodies from over 80 economies have signed the ILAC Mutual Recognition Arrangement (ILAC MRA). This new ISO standard therefore has a tremendous impact on all accredited calibration and testing laboratories of which their national accreditation bodies are signatory members of the ILAC MRA.

Each national accreditation body is expected to work out its own transition plan with actions to be taken to help the laboratories under its charge to smoothly migrate to the new practices. These actions might include, but not limit to, effective communication, scheduled seminars/training courses for laboratory managers and technical assessors, and mapping out a time table and policies to achieve the ultimate goal.

In the nutshell, the new standard has standardized and aligned its structure and contents with other recently revised ISO standards, and the ISO 9001:2015, in particular. It reinforces a process-based model and focuses on outcomes rather than prescriptive requirements such as the absence of familiar terms like quality manual, quality manager, etc. and giving less description on other documentation. It will allow more flexibility for laboratory operation as long as the laboratory’s technical competence can be assessed and recognized by the standard.

The following notes highlight major significant changes in the new revision as compared with those in the 2005 version:

  1. Standard format

Many requirements under the 2005 version remain unchanged but appear in different places of the document, under headlines like general requirements (Clause 4), structural requirements (Clause 5), resource requirements (Clause 6), process requirements (Clause 7) and management system requirements (Clause 8). Also, there are certain language updates to reflect the current standard practices and technologies.

  1. Laboratory activities

Under Clause 3 on Terms and Definitions, the term “laboratory activities” in sub-clause 3.6 has included “sampling, associated with subsequent testing or calibration” in addition to the existing “testing” and “calibration” activities. This is a major scope expansion of the laboratory activity for accreditation and will be a challenge for most testing laboratories which are engaged in field sampling. Sub-sampling of test sample in a laboratory prior to analysis is considered to be part of the test procedure.

  1. Risk-based thinking

The revision has incorporated a new “risk-based thinking” which requires the laboratory to plan and implement actions to address possible risks and opportunities associated with the laboratory activities. The laboratory is responsible for deciding which risks and opportunities need to be addressed. The aims are to:

a) give assurance that the management system achieves its intended results;

b) enhance opportunities to achieve the purpose and objectives of the laboratory;

c) prevent, or minimize, undesired elements

The word of ‘risk’ can be found in the following requirements:

Clause 4.1.4:   Identifying risk to impartiality

Clause 7.8.6.1:  “When a statement of conformity to a specification or standard is provided, the laboratory shall document the decision rule employed, taking into account the level of risk (such as false accept and false reject and statistical assumptions) associated with the decision rule employed and apply the decision rule.”

Clause 7.10:  Actions taken for nonconforming work based upon risk levels established by the laboratory

Clause 8.5:   Actions to address risks and opportunities

Clause 8.7:   Updated risk and opportunities when corrective action is taken

Clause 8.9:   Management review agenda to include results of risk identification

  1. Impartiality

The new standard has stressed on the laboratory’s impartiality.  Under Clause 4 General Requirements, the Sub-clause 4.1 requires the laboratory to identify risks to its impartiality on an on-going basis and if a risk to impartiality is identified, the laboratory shall be able to demonstrate how it will eliminate or minimize such risk.

  1. Decision rule

The term “decision rule” is new to this ISO standard. It first appears in Clause 3.7 under Terms and Definitions which states that “rule that describes how measurement uncertainty is accounted for when stating conformity with a specified requirement”. This is in relation to Sub-clause 7.8.6 on providing “Reporting statements of conformity”.

Before the laboratory provides any statement of conformity to a specification, it is required  to first assess the level of risk (such as false acceptance, false rejection and statistical assumptions) involved in the decision rule employed which has to be documented.  See Sub-clause 7.8.6.1.

  1. External provided products and services

The new standard combines current Sub-contracting, Supplies and External Services which affect laboratory’s activities under a new headline with requirements, controls and communication guidance given under Sub-clause 6.6

  1. Evaluation of measurement uncertainty (MU)

Clause 7.6.1 requires laboratories to identify the contributions to measurement uncertainty. When evaluating MU, all contributions which are of significance, including those arising from sampling, shall be taken into account using appropriate methods of analysis.

The standard in its Clause 7.6.3 Note 3 states that “For further information, see ISO/IEC Guide 98-3, ISO 5725 and ISO 21748”.  It is inferred that the laboratory has a choice in the MU evaluation methods, i.e. using either the conventional GUM (bottom up) or the holistic method performance (top down) approaches can be applied in the MU evaluation.

  1.   Options in management system requirements

The new standard allows the laboratory to implement a management system in accordance with Option A or Option B after meeting the requirements of Sub-clauses 4 to 7.

Option A asks the laboratory to address all its sub-clauses 8.2 to 8.9 as the minimum requirements, and Option B is for a laboratory which has already established and is maintaining a management system in accordance with the requirements of ISO 9001, and which is capable of supporting and demonstrating the consistent fulfillment of the requirement of Clauses 4 to 7, whilst fulfilling at least the intent of the management system requirements specified in Sub-clauses 8.2 to 8.9.

In conclusion, in addition to aligning with the other current international standards in its structural forms and wordings, the new version of ISO 17025 to be implemented introduces new laboratory activity scope on sampling and new requirements such as risks and opportunities, decision rule, sampling uncertainty and two management system options.

Laboratories will need to acquire new skills in carrying out risk assessment, making decision rule, evaluating sampling uncertainty and learning how to incorporate this uncertainty into the overall measurement uncertainty evaluation.

Now, accredited laboratories await for their respective national accreditation body to provide new laboratory accreditation guidelines and directives in this significant migration of ISO/IEC 17025 standards from the existing 2005 version to the latest one during this 3-year transition period.

ISO FDIS 17025:2017, sampling & sampling uncertainty

17025 Process

The international standards for accrediting laboratory’s technical competence has evolved over the past 30 over years, started from ISO Guide 25: 1982 to ISO Guide 25:1990, to ISO 17025:1999, to ISO 17025:2005 and now to the final draft international standard FDIS 17025:2017, which is due to be published before the end of this year to replace the 2005 version. We do not anticipate much changes to the contents other than any editorial amendments.

The new draft standard aims to align its structure and contents with other recently revised ISO standards, and the ISO 9001:2015 in particular. It is reinforcing a process-based model and focuses on outcomes rather than prescriptive requirements such as eliminating familiar terms like quality manual, quality manager, etc. and giving less description on other documentation. It attempts to introduce more flexibility for laboratory operation.

Although many requirements remain unchanged but appear in different places of the document, it has added some new concepts such as:

–  focusing on risk-based thinking and acting,

–  decision rule for measurement uncertainty to be accountable for when stating

conformity with a specification,

–  sampling as another laboratory activity apart from testing, and calibration, and,

–  sampling uncertainty to be a significant contributing factor for the evaluation of

measurement uncertainty.

The purpose of introducing sampling as another activity is understandable, as we know that the reliability of test results is hanged on how representative the sample drawn from the field is. The saying “The test result is no better than the sample that it is based upon” is very true indeed.

If an accredited laboratory’s routine activity is also involved in the field sampling before carrying out laboratory analysis on the sample(s) drawn, the laboratory must show evidence of a robust sampling plan to start with, and to evaluate the associated sampling uncertainty.

It is reckoned however that in the process of carrying out analysis, the laboratory has to carry out sub-sampling of the sample received and this is to be part of the SOP which must devote a section on how to sub-sample it. If the sample received is not homogeneous, a consideration of sampling uncertainty is to be taken into account.

Although FDIS states that when evaluating the measurement uncertainty (MU), all components which are of significance in the given situation shall be identified and taken into account using appropriate methods of analysis, its Clause 7.6.3 Note 2 further states that “for a particular method where the measurement uncertainty of the results has been established and verified, there is no need to evaluate measurement uncertainty for each result if it can demonstrate that the identified critical influencing factors are under control”.

To me, it means that all identified critical uncertainty influencing factors must be continually monitored. This will have a pressured work load for the laboratory concerned to keep track with many contributing components over time if the GUM method is used to evaluate its MU.

The main advantage of the top down MU evaluation approach based on holistic method performance using the daily routine quality control data, such as intermediate precision and bias estimation is also appreciated as stated in Clause 7.6.3.  Its Note 3 refers to the ISO 21748 which uses accuracy, precision and trueness as the budgets for evaluation of MU, as an information reference.

Secondly, this clause in the FDIS suggests that once you have established an uncertainty of a result by the test method, you can estimate the MU of all test results in a predefined range through the use of relative uncertainty calculation.

 

Which top down MU method is good for you?

Χ ± ∪

Which alternative top down MU method will work best for you?

There are several alternative top down methods available for the estimation of measurement uncertainty (MU) in chemical and microbiological laboratories. One or two will work best under your laboratory conditions.

All accredited laboratories would have implemented a robust laboratory quality management system in accordance to the ISO/IEC 17025 standards. In this scenario, you would have carried out certain method validation and verification on your in-house test methods and standard/official methods, respectively. You would also have consistently been running laboratory control samples (LCS’s) as and when you conduct a batch of sample analysis to monitor the accuracy of your test results. Similarly, you should also have been participating some proficiency testing (PT) programs at regular intervals as required by the national accreditation body.

With this in mind, I recommend the following top down approaches which will be suitable for your MU evaluation:

  1. For established standard test methods (like ISO, AOCS, BS, EN, ASTM, etc.) that you have been running routinely with readily available QC data, the use of data repeatability, reproducibility and trueness estimates will be fine for estimating the MU, such as following the ISO 21748:2010;
  2. If you have been using stable laboratory control samples (LCS’s) to monitor your test accuracy regularly, you can consider plotting the LCS data on a Shewhart control chart against time and apply the variance of the data moving average as the estimation of its standard uncertainty (see ASTM D6299-08). Certain pre-requisites however, do apply here, such as statistically confirming your QC data are completely random and independent by the Anderson-Darling (AD) or Shapiro-Wilk tests for normality, and visually checking the data trend of the Shewhart chart to follow a set of chart rules laid down by the standards;
  3. If your laboratory is not able to take part in any PT program because there is no such program to check your test parameters, you can use your within-lab reproducibility (or intermediate precision) data and the result bias estimates which are available in your method validation process. The relevant reference of this approach is ISO 11352:2012;
  4. For certain types of chemical tests involving a series of certified reference materials as calibration standards (e.g. the determination of total sulphur in petroleum product by X-ray fluorescence spectrometric method – ASTM D4294 – which is a direct read-out from the energy-dispersive X-ray fluorescence sulphur meter with 4-point or 5-point calibration using different sulphur reference standards), the laboratory concerned can plot a linear calibration curve involving the actual test results of these reference standards against the assigned reference values to estimate the uncertainty. See reference ISO 11095:1996(2012).
  5. The GUM component-by-component method is not suitable for the MU estimation of microbiological count experiment because the distribution of such data is not strictly normal whilst GUM has made this assumption. In fact, the Poisson probability distribution is better in this situation. The MU estimation for microbiological counting therefore is based on the holistic performance of the test method and we need to make logarithmic transformation of the within-lab reproducibility data before evaluating its variances. Various established documents are available for reference, such as ISO 10936:2006, BS 8496:2007, A2LA G108, NMKL Procedure No. 8, etc.

In my opinion, the ISO 11095 method in (d) is the most difficult one for a laboratory analyst who has acquired only elementary statistical skills, because of its application of more complicated statistical tools in estimating the constant and proportional variances. The other approaches are quite straight forward and will be easily appreciated by the analysts with average statistical knowledge.

 

List of published documents on top down MU methods

All logos

A list of published international documents in relation to the use of top down MU approaches (not exhaustive)

  • ISO 21748:2010 Guidance for the use of repeatability, reproducibility and trueness estimates in measurement uncertainty estimation
  • ISO 11095:1996 (2012) Linear calibration using reference materials
  • ISO 11352:2012 Water quality — Estimation of measurement uncertainty based on validation and quality control data
  • ISO 19036:2006 Microbiology of food and animal feeding stuffs — Guidelines  for the estimation of measurement uncertainty for quantitative determinations Amd1:2009: Measurement uncertainty for low counts
  • ISO 29201:2012 Water Quality – The variability of test results and the uncertainty of measurement of microbiological enumeration methods
  • ISO Guide 98-3/Suppl. 1 Uncertainty of measurement Part 3: Guide to the expression of uncertainty of measurement  Supplement 1 : Propagation of distributions using a Monte Carlo method
  • BS 8496:2007   Water quality. Enumeration of micro-organisms in water samples

  • ASTM D2554-07 Estimating and monitoring the uncertainty of test results of a test method in a single laboratory using a control sample program
  • ASTM D6299-08 Applying statistical quality assurance and control charting techniques to evaluate analytical measurement system performance
  • ASTM E2093-05 Optimizing controlling and reporting test method uncertainty from multiple workstations in the same laboratory organization
  • EuroLab Technical Report No. 1/2006 Guide to the Evaluation of Measurement Uncertainty for Quantitative Test Results
  • NORDIC Technical Report TR 537 Edition 3.1 Handbook for Calculation of Measurement Uncertainty in Environmental Laboratories
  • A2LA G108 – Guidelines for Estimating Uncertainty for Microbiological Counting Methods
  • NMKL Procedure No. 8 (2002) Measurement of uncertainty in microbiological examination of foods.
  • CNAS-GL34:2013 基于质控数据环境检测测量不确定度评定指南  Guidance for measurement uncertainty evaluation based on quality control data in environmental testing

 

 

 

A chat group set up at www.meetup.com website

Yeoh GH of GLP Consulting has at the http://www.meetup.com website set up a chat group, namely Interest Group in Measurement Uncertainty for Test Labs for better interaction with the laboratory and QA personnel in this region.

The very first Meetup session is scheduled on August 30 in Singapore with venue to be confirmed later. All are welcome to join this interest group at https://www.meetup.com/Interest-Group-in-Measurement-Uncertainty-for-Test-Labs/  even though you might not be in Singapore as we shall post summaries of discussions and useful pointers after the meetup.

Meetup 1

Another top down MU method – ISO 11352 made simple

Lab Picture 3

Another top down MU method – ISO 11352

 

 

Top down MU method – ISO 21748 explained simply

measurement-uncertainty-4-638

Top down MU method ISO21748 explained simply

 

 

Estimating MU for microbiological plate count – using Laboratory Control Samples with Target Values

A2LA (The American Association for Laboratory Accreditation) in its policy document P103b – Annex: Policy on estimating measurement uncertainty for life sciences testing labs reckons that when a test laboratory uses chemical, environmental or biological test methods traceable to published regulatory or consensus methods (such as FDA, EPA, AOAC, ASTM, ISO, BS, EN, etc), measurement uncertainty (MU) shall be estimated using available data, published information, and/or some designed experiments. That means uncertainty can be estimated using laboratory control samples (LCS), method validation studies, or by appropriate model for the propagation of error components.

ASTM E 2554-13: Standard practice for estimating and monitoring the uncertainty of test results of a test method in a single laboratory using a Control Sample Program indicates that LCS may be used to estimate MU, provided the samples are an appropriate matrix, stable and homogeneous in concentration distribution.

The A2LA G108 states that:

  1. When the LCS has been through all method steps, the laboratory can then use the pooled standard deviation (sp) from the LCS intermediate reproducibility sR’ (also called intermediate precision) data as an estimate of combined standard uncertainty. A relative standard deviation (RSD) or coefficient of variation (CV) may also be used.
  2. When the LCS have not been run through all method steps, then the laboratory should incorporate any appropriate additional uncertainty components as expressed in their respective standard uncertainty in its MU estimation. The additional components should be combined with sp using the root sum square (RSS) method:

Formula1

where sa, sb, … are the standard deviations of the other uncertainty components

3.   When a method has a known consistent bias that is inherent to the method (e.g.                 low recovery on difficult analytes), the bias must not be added to the uncertainty               calculations. The bias shall, however, be clearly stated and recorded along with                   the uncertainty estimate.

This method involving the use of LCS with same target values is the simplest amongst the three methods discussed so far. The other two methods that we had discussed are using the intermediate reproducibility duplicates, and the recovery replicates. It can be shown that this method, however, tends to produce a much bigger uncertainty as only single analysis data point is taken in the evaluation as compared to the other approaches using duplicates results or matrix spiked recoveries which provide better confidence in the outcome.

Image 1 below shows the A2LA example on 20 LCS replicates analyzed over a period of time under the laboratory intermediate reproducibility conditions:

 

A2LA Table 3A

This example has also covered most sources of uncertainty in the analytical process. It is recommended that 20 or more individual LCS data points are to be obtained to estimate the pooled standard uncertainty in terms of pooled standard deviation, sp. The evaluation of the combined uncertainty is then expanded by multiplying sp with a coverage factor k =2 for 95% confidence.

In concluding this series of MU estimation on microbial count testing, we should say that the three procedures described in this article and the other two earlier ones are having their own merits and disadvantages, and no one process is favored over the other.  It is up to the laboratory concerned to exercise its professional judgment in deciding whether the MU estimate obtained from any of these approaches is reasonable and whether it meets the needs of its customers.

 

Estimating MU for microbiological plate count – using Recovery Replicates method

In the last article, we showed how intermediate reproducibility duplicates data of the laboratory control samples (LCS’s) could be used to estimate the measurement uncertainty of an aerobic count method. If it is safe to assume that analysis recovery is reasonably constant for a particular organism in a given sample matrix, the uncertainty in a microbial enumeration test can be evaluated by studying the recovery data over time.

But, what is analysis recovery?

Recovery is defined as “Proportion of the amount of analyte, present in or added to the analytical portion of the test material, which is extracted and presented for measurement”. In a method validation protocol, analytical recovery studies are an essential component which cannot be ignored.

Recovery data R obtained as the ratio of the concentration of analyte found to that stated to be present, can be used to determine bias present in a particular test method by studying if the difference between the actual test result is found to be significantly different from the assigned or known value.  Even if the bias does exist, it is still not current conventional practice to correct for bias in quantitative microbiology results.

Therefore to get a recovery data, one has to add a known amount of analyte to a matrix and the whole matrix is then subject to the whole process of the test method. The amount recovered minus the original amount present should indicate the recovery factor.

In a perfect situation, R would be exactly unity (1) but in reality, circumstances such as imperfect extraction or serial dilutions often give observations that differ from the ideal.

Hence, we must take note of the sources of uncertainty in recovery estimation. Some of them are:

  1. repeatability of the recovery experiment
  2. uncertainties in reference material values
  3. uncertainties in added spike quantity (in terms of weight or volume)
  4. poor representation of native (originally present) analyte by the added spike
  5. poor or restricted match between experimental matrix and the full range of sample matrices encountered
  6. effect of analyte / spike level on recovery and imperfect match of spike or reference material analyte level and analyte level in samples.

Consequently, recovery differences over time reflect the various uncertainty contributors, including but not limiting to the following:

  • random error which is a common cause variability
  • analyst differences
  • different equipment
  • different environmental conditions

The aim of this method therefore is to use recovery variations of laboratory control samples over time to determine how various sources of uncertainty affect the final test results.

A2LA calculation method

A recovery test is conducted by plating the same amount of inoculum in petri dishes with and without the matrix of interest and counting the colony forming units (CFU’s) after growth in sterile nutrient agar at specified temperature over incubation time required.  The difference between the counts from the plate without matrix and the plate with matrix samples is a measure of recovery of the organism, usually expressed as a percentage of the CFU count in the inoculated sample.

A2LA carried out 20 replicates of great different inoculated levels and found the recovery was reasonably constant. Hence, it is assumed that the matrix used for this study is representative of those samples where an estimate of uncertainty is needed.

Image 1 below shows the data of the recovery replicates obtained over time and the subsequent calculation involving transformation of data to logarithmic values.

A2LA Table2

For example, for a result of 1,000 CFU, we can report the uncertainty interval as 606 to 1,647 CFU.

It may be noted that the actual calculated interval was 607 and 1,646, but for maximum coverage of 95% confidence interval, the results by convention are always “rounded down” for the value at the lower end of the range and always “rounded up” for the value at the upper end.

Like the method using the intermediate reproducibility data, this current method has also adequately captured in the estimate various types of uncertainty sources, namely: random error, counting error, dilutions, environment, equipment and analyst.