Jon Seltin – Brick Lane Brewing Co, Melbourne.

Interlaboratory Proficiency Testing For

Craft Breweries

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 2

Overview

• What is Proficiency Testing?

• Why is it a good idea?

• What schemes exist?

• Why make another one?

• What is BIRA?

• How does it work?

• Statistics and reporting

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 3

What is proficiency testing?

‘The primary purpose of proficiency testing is to help laboratories detect and cure any unacceptably large inaccuracy in their reported results’ The International Harmonized Protocol for the

Proficiency Testing of Analytical Chemistry Laboratories (IUPAC Technical

Report)

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 4

What is proficiency testing?

• Proficiency testing is a way of measuring (and continually improving)

proficiency in beer analysis

• Distribution of a homogeneous sample / artifact between participants

• Each participant performs a common set of analysis

• Results are collected centrally by the PT scheme provider, and analyzed

statistically.

• Results are published, and members receive a report outlining their

performance in comparison to their peers

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 5

What is proficiency testing?

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 6

Why is proficiency testing a good idea?

PT is a cornerstone of good quality assurance

• I ter al, regular QA easure. Ho tu ed i is our la ?

• Helps identify problems (especially systematic errors) that you may

otherwise be blind to in a lab producing otherwise precise results

• Demonstrates competence and proficiency

– To brewers and production staff

– To management

– To consumers

– To clients

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 7

Why is proficiency testing a good idea?

• Assists in setting meaningful product specifications

– Particularly important w acceptance on analytical measures across

labs/production facilities, such as in contract brewing and packing.

• Trigger for a Kaizen event

• Staff education, training and monitoring

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 8

Why is proficiency testing a good idea?

• Meet regulatory obligations

– FSANZ

– NMI

– Truth in labeling

– Excise reporting

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 9

What schemes already exist?

• Internal schemes for multi-site or multinational breweries operating

multiple laboratories

• BAPS/MAPS (Brewing/Malt Analytes Proficiency Scheme)

– Global, administered by Campden BRI and LGC Standards

– High participation from larger brewing labs in Aus/NZ

• ASBC Check Sample

– US based, with analytes covering Barley, Malt, Hops, Beer, Mycotoxins

– Packages available for craft breweries

• Sensory proficiency testing offered by Cara Technology / Aroxa

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 10

Why Another Scheme?

• Creating a domestic forum for discussion and promotion of excellence in

beer analysis

• O l i depe de t re eries parti ipati g i Australia e thi k…

• Limitations of existing schemes;

– Costly. Even schemes aimed at encouraging participation from small brewers

(such as the ASBC Sample Check) is US$270 + shipping.

– Long list of analytes beyond the reach of many smaller breweries (such as GC/

HPLC methods).

– Limited local support or re-testing

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 11

BIRA – the Brewing Interlaboratory Reference

Analytes • BIRA was created to promote participation amongst smaller breweries in

PT, and to drive continual improvement in the field of beer analysis

• Desig ed as a light eight s he e, ith lo er arriers to parti ipate, a d lower participation costs

• Analytes aligned with the requirement and capabilities of smaller breweries.

• Non-profit scheme run by volunteers

• Local support and troubleshooting, and retesting possible.

• Possibility of using alternative experimental design to BAPS to address inside-lab repeatability AND inter-lab reproducibility

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 12

BIRA – the Brewing Interlaboratory Reference

Analytes

• Established in 2017 at the Australian CBIA conference.

• Modeled in part on the Interwinery Analysis Group (IWAG), an Australian

scheme founded in 1983, and now with over 200 members internationally.

• 12 participating labs in Round 1, representing a significant volumetric

portion of independent craft beer production in Australia

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 13

Some Participants

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 14

BIRA – how does it work

• 4 rounds/pa, distributed in two shipments

• Beer supplied by member breweries (preferably in-pack pasteurized for greater stability)

• Analysis performed during the same 1-week window and reported online via portal

• Round 1 analytes

– Apparent Extract

– ABV

– Dissolved CO2

– Bitterness

– pH

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 15

BIRA – how does it work

• Members report results anonymously

• No prescriptive methods – free choice of methods by participants

• Results screened and statistically analyzed and assigned a z-score,

denoting their proficiency in that particular analysis.

• Annual recognition to the lab with best results

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 16

Statistics and Reporting

• BIRA aims to take the best bits from BAPS, ASBC and IWAG

• Different design to BAPS, in that BIRA uses paired samples, and Youden

charts to interpret results.

• Youden analysis represents a powerful statistical method, simultaneously

indicating inside-lab repeatability as well as inter-lab reproducibility (r95 vs

R95)

• More statistically useful results

• Helps eliminate operator bias

• Identifies systematic inside-lab errors

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 17

Statistics and reporting – BIRA specifics

• Statistics endorsed by ISO13528 – paves way for future accreditation

• Assigned values taken from the results submitted – not a reference value measured by a reference laboratory.

• Using a robust statistics (assigned value is the median) as opposed to a traditional approach (assigned value is the mean). – Less sensitive to outliers.

– Better for smaller data sets

• Gross outliers are screened and removed (typos, misplaced decimal points, spelling errors)

• No further outlier re o al differe t to the ASBC ethod, hi h uses Di o s outlier test)

• We re goi g MAD – using Median Absolute Deviation (MAD) for the assigned deviation.

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 18

Z-scores and results interpretation

• Lab individual performance can be assessed using a Z-score traditionally:

• 𝑥 = individual Lab result • X = Assigned value

– Consensus value, – expert labs, mean of experts – known reference value from certified reference materials

• σ = SD (or in our case Median Absolute Deviation MAD)

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 19

20

Z ≤ Satisfa tory

Z>

3

Un

sati

sfa

cto

ry

<Z≤

Que

stio

nale

<Z≤

Que

stio

nale

Z>

3

Un

sati

sfa

cto

ry

Example Report

Colour-coded z-scores (legend part of Summary Statistics)

Only participant number

shown - anonymity

Example data only – not real results

Methods used summarised

Youden Plot

Example • Developed by BIRA

– Developed in R

– Easy to interpret • Points along the diagonal– both samples

askew – systematic error: e.g. non-calibrated instrument.

• Away from the diagonal – one sample fine, the other askew – random error: e.g. filter paper for one sample leaked.

• Z-scores

– < 2 – acceptable

– ≥ 2, < 3 – monitor

– ≥ 3 – not acceptable

SYSTEMATIC ERROR

RANDOM ERROR

FAQs -

• Rounds 1&2 still open (sign up!).

• It s heap. Cost is urre tl AU$ for rou ds, i ludi g shippi g ithi Australia.

• No requirement for a big lab, dedicated quality techs or expensive

analytical equipment

• Easy to interpret reports, actionable data.

• The more the merrier (and the better the statistics)

www.biralab.com

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 23

Questions??

• Guide to Proficiency Testing Australia

• Tho pso et al. 00 The i ter atio al har o ized proto ol for the profi ie testi g of a al ti al he istr la oratories Pure Appl. Chem.,

Vol. 78, No. 1, pp. 145–196.

• Interwinery Analysis Group

• BAPS

• ASBC Check Sample

• NATA proficiency testing accreditation

• ISO/IEC 17043:2010 - Conformity assessment -- General requirements for

proficiency testing

• ISO 13528:2005 Statisitical methods for use in Proficiency Testing

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 24

Thanks to Brick Lane Brewing Co for their support of this work

Thanks to the BIRA committee members for their input (especially Thomas, Claire, Daniel)

Thanks!

Asia Pacific Convention 2018 | Wellington, New Zealand Institute of Brewing and Distilling Asia Pacific Section 25

IBD PPT –

Final Slide