precision metrology 1078-01
TRANSCRIPT
-
7/25/2019 Precision Metrology 1078-01
1/54
SCOPE OF ACCREDITATION TO ISO/IEC 17025:2005& ANSI/NCSL Z540-1-1994
PRECISION METROLOGY, INC.9
7350 North Teutonia AvenueMilwaukee, WI 53209
Carol Shipley Phone: 414 351 7420
CALIBRATION
Valid To: July 31, 2016 Certificate Number: 1078.01
In recognition of the successful completion of the A2LA evaluation process, accreditation is granted tothis laboratory to perform the following calibrations1, 9:
I. Acoustical Quantities
Parameter/Equipment Range CMC2, 8() Comments
Sound Level (0 to 140) dB
(0 to 140) dB
0.16 dB + 0.032 %
0.32 dB
Piston phone
Sound level calibrator
Sound Level3 (0 to 140) dB 0.6 dB Sound level calibrator
II. Chemical
Parameter/Equipment Range CMC2, 4() Comments
Conductivity Meters 10 S/cm100 S/cm1000 S/cm1400 S/cm
0.68 S/cm + 0.6R1.2 S/cm + 0.6R4.5 S/cm + 0.6R6.9 S/cm + 0.6R
Conductivity solutions
-
7/25/2019 Precision Metrology 1078-01
2/54
III. Dimensional
Parameter/Equipment Range CMC2, 4() Comments
Angle Blocks 0to 90 5 + 10 parts in 106 Measuring machine
Angle Plates3 Up to 30 in (120 + 6H) in Gage amp. probe, indi-square and test
indicator
Arbor Straightness Up to 24 in (35 + 9L) in Gage amp w/probe
Bench Center
Center Parallelism
Base Flatness & BaseParallelism
Up to 24 in
Up to 24 in
(68 + 4L) in
(62 +2.5L) in
Gage amp w/probe
Benchmics3 Up to 2 in 30 in Gage blocks
Bore Gages3 Up to 12 in 10 in/in + 0.6R Gage blocks
Box Parallels
ParallelismSquareness
5 in 10 in 10 in (12 + 20H) in(18 + 20H) in
Gage amp w/probe,cylindrical square
Calipers3(Including
Intertest, Oditest, SnapJaw and other ODCalipers)
Resolution 0.0001 in Up to 2 in(2 to 12) in
78 in70 in + 4 in/in
Gage blocks
-
7/25/2019 Precision Metrology 1078-01
3/54
Parameter/Equipment Range CMC
2, 4
() Comments
Chamfer Gages/HoleGages3
Up to 12 in 20 in + 0.6R Cylindrical rings
Clinometers andInclinometers
Up to 360 16 + 0.6R Sine bar/gage blocks
CMMs3
Length Error (EL)
Linear
Volumetric
Length Error (E150)
Repeatability Range
(R0)
Length
Volume
Probing Analysis
Bi-directional
Up to 52 in
Up to 52 in
----
Up to 114 in on each
axis
Up to 36 in for
shortest axis
---
38 in + 5.2 in/in
58 in + 8.1 in/in
38 in + 5.2 in/in
---
38 in + 5.2 in/in
82 in + 13 in/in
2 m
(40 + 10L) in
ASME
B89.4.1.10360.2
ASME B89.4.1.2000
Ball bar
Test probe
Coating ThicknessShims/Precision Shims
25 in to 0.5 in 19 in + 82 in/in Heidenhain-CertoCT 6001
Concentricity Gage N/A 130 in + 8 in/in Indicator, bench center
Cylindrical Plugs3 Up to 4 in(4 to 20) in
4 in + 5 in/in4 in + 5.4 in/in
ULM
-
7/25/2019 Precision Metrology 1078-01
4/54
Parameter/Equipment Range CMC2, 4() Comments
Cylindrical Squares Up to 12 in (13 + 9H) in Gage amp w/probe;His the height of thecylinder from the basein inches.
External Spline Gages3
Measurement Over Pins
Circular ToothThickness
Major Diameter
Up to 8 in (170 + 28D) in
(110 + 16L) in
(28 + 12D) in
UMM
Optical comparator
UMM
Feeler/Thickness Gage3 Up to 0.2 in 36 in UMM
Fixed Gaging
Screw Pitch Gages, DrillGages, Taper Gages,
Center Gages, Sheet andWire Gages, Angle Gages
---
---
(110 + 16L) in (lengthmeasures)
54
Measuringmachine/opticalcomparator
Gage Amplifier andProbe(s)3
Single ProbeDual Probe
------
8.9 in2.1 in
Gage blocksGage blocks
Gage Balls Up to 4 in (32 + 1.5D) in Universal measuringmachine (UMM)
Gage Blocks Up to 4 in(4 20) i
(3.1 + 3.2L) in(3 3 3 2L) i
Gage blocks and
-
7/25/2019 Precision Metrology 1078-01
5/54
Parameter/Equipment Range CMC2, 4() Comments
Height Measures3 Up to 24 in 13 in/in + 94 in Height gage
Indicators3
Dial
Test
Up to 8 in
Up to 0.060 in
10 in/in + 0.6R
0.0001 in
Indicator calibrator
Indicator Calibrator3 ---- 12 in + 0.6R Gage blocks, ampw/probe
Length Standards3
CROBLOX
Ceramic
Steel
Up to 4 in(4 to 20) in
Up to 4 in(4 to 20) in
Up to 4 in(4 to 20) in
(20 to 120) in
4 in + 2.2 in/in4 in + 2.4 in/in
4 in + 2.7 in/in4 in + 3 in/in
4 in + 5 in/in4 in + 5.4 in/in
3.7 in/in + 14 in
ULM, gage blocks
ULM
Levels (Machinist)3 Up to 96 in 0.00019 in/ft Sine plate
Linear Measure3 (Up to 24) in 94 in + 13 in/in Height gage
Linear Measure (Up to 30) in (330 + 5.8L) in CMM
Linear Measurement ofMachine Tools3
Up to 20 ft 1.5 in/in + 0.58 in Laser
-
7/25/2019 Precision Metrology 1078-01
6/54
Parameter/Equipment Range CMC2, 4, 8() Comments
Linear Scales/Reticlesand Stage Micrometers
2D
1D
Up to 12 in
Up to 30 in
(115 + 15L) in
9.9 in + 1.5 in/in
Vision system
Gitterperiodeninterferometer
Micrometers3
InsideDepthTri-BoresOutsideGroove
Up to 294 inUp to 12 inUp to 6 inUp to 42 inUp to 4 in
10 in/in + 0.6R10 in/in + 0.6R10 in/in + 0.6R10 in/in + 0.6R18 in/in + 0.6R
Gage blocks
Microscopes3
ReticuleMagnification
Up to 25 mmUp to 1000x
23 m2.4 %
Stage micrometer
Optical Comparator3
LinearAngleMagnification
Up to 12 in0 to 36010x, 20x, 50x,62.5x, 100x, 200x
0.0002 in20.00012 in
Glass scaleCalibration sphereCalibration sphere
Optical Flats Up to 8 in 2.8 in Reference flat andmonochromatic light
source
Parallels3
Steel 1.5 in 6 in (10 + 6L) in Gage amp w/probe;Listhe distance between
-
7/25/2019 Precision Metrology 1078-01
7/54
Parameter/Equipment Range CMC2, 4() Comments
Pitch MicrometerStandards
Length
Angle
Up to 12 in
Up to 60
(22 + 6L) in
54
UMM
Vision system
Precision Diameter Tapes Up to 48 in(48 to 96) in(96 to 780) in
200 in + 4 in/in560 in + 1.4 in/in0.0013 in + 6 in/in
Setting discs, ULM
Profilometers3 120 in Ra 2.8 in Roughness specimen
Protractor Digital and
Mechanical3
0to 180 0.0076 Sine bar and gageblocks
Radius Gage Up to 5 in 0.00015 in Optical comparator
Riser Blocks and Stands Up to 24 in (13 + 8H) in Gage amplifier with
probe
Roundness Testers3
Coning Error
Axial Error
Radial Accuracy
Z-Axis straightness
---
---
---
(0.1 to 12) in
11 in
11 in
11 in
(14 + 12H) in
Test ball and risercylinder
Test ball
Test ball
Cylindrical square; Inthe CMC,His theheight of the cylinderfrom the base in inches
-
7/25/2019 Precision Metrology 1078-01
8/54
Parameter/Equipment Range CMC2, 4() Comments
Sine Plates/Bars3
Flatness/Parallelism &Parallelism Cylinder toBase
Angle CalibrationCenter of Rolls
Up to 20 in
0 to 90Up to 20 in
50 in
0.0004516 in/in
Gage amp w/probe,angle blocks and gageblocks
Snap Gages3 Up to 20 in 88 + 13 L Box parallel with gageamp and probe
Squares3
Master
Granite
Steel
Up to 30 in
Up to 30 in
Up to 30 in
(56 + 5H) in
(42 + 6H) in
(42 + 6H) in
Gage amp and probe
Indi-square and testindicator
Master square and gageblocks
Straightness andStraight Edges3
Up to 60 in (10 + 14L) in Gage amplifier w/probe
Step Gages Up to 50 in 9.7 in + 3.1 in/in Gage amp and probe,gage blocks
Surface Plates3
Flatness
Repeatability
18 in x 24 in to72 in x 144 in
Up to 72 in x 144 in
8Din
34 in
Electronic levelsDis the length of theplate diagonal ininches.
Repeat o meter
-
7/25/2019 Precision Metrology 1078-01
9/54
Parameter/Equipment Range CMC2, 4() Comments
Surface Roughness3 (15 to 150) in Ra 3.4 in Verification ofspecimens per ASMEB46.1 Type C usingprofilometers; Ra is theresolution of theprofilometers inmicroinches.
Tape Measures3 Up to 300 ft (0.026 + 0.6R) in UMM
Tapered Plugs3
Simple Pitch Diameter
Notch Height
Up to 18 in
Up to 18 in
(100 + 12D) in
13 in/in + 94 in
Thread wires/UMM
Height measuringmachine
Tapered Rings3
Simple Pitch Diameter
Taper
Standoff and Thickness
Functional Diameter
Up to 18 in
Up to 18 in
Up to 18 in
Up to 2 in
300 in
400 in
120 in
(300 + 8D) in
NPT master plug,UMM
NPT master plug
NPT master plug andheight measuringmachine
Thickness Gages3 Dial
and Digital
Up to 1 in 19 in + 0.6R Gage blocks
Thread Micrometers3(Screw Thread, Pitch,Point)
Up to 4 in 10 in + 0.6R Pitch standard
-
7/25/2019 Precision Metrology 1078-01
10/54
Parameter/Equipment Range CMC2, 4() Comments
Thread Rings3
Simple Pitch DiameterMinor Diameter
Functional DiameterMinor Diameter
Up to 18 inUp to 2 in
Up to 18 in(2 to 18) in
300 in(70 + 13D) in
(300 + 8D) in(300 + 11D) in
UMM
w/ set plug
Thread Wires Up to 0.5 in (8.5 + 7D) in UMM, gage blocks
Universal MeasuringMachines3(UMMs)
English
Metric
Up to 4 in(4 to 24) in
Up to 100 mm
1.5 in + 1.2 in/in1.3 in + 1.8 in/in
0.04 m + 0.0015 m/mm
Gage blocks
V-Blocks
Parallelism Side
V
Squareness
8 in 8 in 8 in 14 in
20 in
(37 + 11H) in
Gage amplifierw/probe
Cylindrical square
Cylindrical plug
Vision Systems3
X-Y Linear Accuracy
Z-Axis Linear
Accuracy
Up to 24 in
Up to 20 in
(110 + 13L) in
(74 + 11L) in
Glass scale
Gage blocks
Wire Crimpers3
Crimp Height
Crimping Chamber
--- 0.0024 in
(10 + 19D) in
Micrometer
Pin; D is the diameter
-
7/25/2019 Precision Metrology 1078-01
11/54
IV. Electrical DC/Low Frequency
Parameter/Range Frequency CMC2, 7() Comments
AC Current Generate& Measure
(10 to 300) A
(0.3 to 3) mA
(3 to 10) mA
10 Hz20 Hz
30 Hz40 Hz55 Hz100 Hz300 Hz400 Hz500 Hz1 kHz3 kHz5 kHz
10 kHz20 kHz30 kHz
10 Hz20 Hz30 Hz40 Hz55 Hz100 Hz
300 Hz400 Hz500 Hz1 kHz3 kHz5 kHz10 kHz20 kHz30 kHz
10 Hz20 Hz40 Hz400 Hz1 kHz5 kHz10 kHz
0.018 % + 30 pA67 A/A + 30 pA
42 A/A + 30 pA37 A/A + 30 pA37 A/A + 30 pA38 A/A + 30 pA38 A/A + 30 pA38 A/A + 30 pA38 A/A + 30 pA38 A/A + 30 pA46 A/A + 30 pA46 A/A + 30 pA
46 A/A + 30 pA63 A/A + 30 pA80 A/A + 30 pA
0.017 % + 300 pA61 A/A + 300 pA30 A/A + 300 pA30 A/A + 300 pA30 A/A + 300 pA30 A/A + 300 pA
30 A/A + 300 pA30 A/A + 300 pA30 A/A + 300 pA30 A/A + 300 pA30 A/A + 300 pA31 A/A + 300 pA30 A/A + 300 pA45 A/A + 300 pA48 A/A + 300 pA
0.017 % + 1 nA69 A/A + 1 nA38 A/A + 1 nA36 A/A + 1 nA35 A/A + 1 nA36 A/A + 1 nA36 A/A + 1 nA
Fluke 5790A,5700A/EP,
A40, standard resistors
-
7/25/2019 Precision Metrology 1078-01
12/54
Parameter/Range Frequency CMC2, 7() Comments
AC Current Generate& Measure (cont)
(10 to 20) mA
(20 to 30) mA
(30 to 50) mA
(50 to 100) mA
(100 to 200) mA
10 Hz20 Hz40 Hz400 Hz1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz40 Hz400 Hz
1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz40 Hz400 Hz
1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz40 Hz400 Hz
1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz
0.017 % + 2 nA69 A/A + 2 nA38 A/A + 2 nA36 A/A + 2 nA35 A/A + 2 nA36 A/A + 2 nA36 A/A + 2 nA36 A/A + 2 nA64 A/A + 2 nA
0.017 % + 3 nA69 A/A + 3 nA38 A/A + 3 nA36 A/A + 3 nA
35 A/A + 3 nA36 A/A + 3 nA36 A/A + 3 nA36 A/A + 3 nA64 A/A + 3 nA
0.017 % + 5 nA69 A/A + 5 nA38 A/A + 5 nA36 A/A + 5 nA
35 A/A + 5 nA36 A/A + 5 nA36 A/A + 5 nA36 A/A + 5 nA64 A/A + 5 nA
0.017 % + 60 nA70 A/A + 60 nA38 A/A + 60 nA36 A/A + 60 nA
35 A/A + 60 nA36 A/A + 60 nA36 A/A + 60 nA40 A/A + 60 nA68 A/A + 60 nA
0.017 % + 120 nA
Fluke 5790A,5700A/EP,A40, standard resistors
-
7/25/2019 Precision Metrology 1078-01
13/54
Parameter/Range Frequency CMC2, 7() Comments
AC Current Generate &Measure (cont)
(200 to 300) mA
(300 to 500) mA
(0.5 to 1) A
(1 to 2) A
(2 to 3) A
10 Hz20 Hz40 Hz400 Hz1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz40 Hz400 Hz
1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz40 Hz400 Hz
1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz40 Hz400 Hz
1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz
0.017 % + 180 nA72 A/A + 180 nA38 A/A + 180 nA36 A/A + 180 nA35 A/A + 180 nA36 A/A + 180 nA36 A/A + 180 nA40 A/A + 180 nA72 A/A + 180 nA
0.017 % + 300 nA72 A/A + 300 nA38 A/A + 300 nA36 A/A + 300 nA
35 A/A + 300 nA36 A/A + 300 nA36 A/A + 300 nA40 A/A + 300 nA72 A/A + 300 nA
0.018 % + 6 A72 A/A + 6 A42 A/A + 6 A40 A/A + 6 A
39 A/A + 6 A40 A/A + 6 A40 A/A + 6 A52 A/A + 6 A0.011 % + 6 A
0.018 % + 12 A75 A/A + 12 A42 A/A + 12 A40 A/A + 12 A
40 A/A + 12 A40 A/A + 12 A40 A/A + 12 A53 A/A + 12 A0.011 % + 12 A
0.019 % + 18 A
Fluke 5790A,5700A/EP,A40, standard resistors
-
7/25/2019 Precision Metrology 1078-01
14/54
Parameter/Range Frequency CMC2, 7() Comments
AC Current Generate &Measure (cont)
(3 to 5) A
(5 to 10) A
(10 to 20) A**
50 Turn Coils
(Effective CurrentTransfer Ratio)
10 Hz20 Hz40 Hz400 Hz
1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz40 Hz
400 Hz1 kHz5 kHz10 kHz20 kHz50 kHz
10 Hz20 Hz
40 Hz400 Hz1 kHz5 kHz10 kHz20 kHz50 kHz
25 Turn Amps, 50 Hz,
400 Hz500 Turn Amps, 50 Hz,400 Hz800 Turn Amps, 400 Hz1000 Turn Amps, 50 Hz
0.019 % + 30 A85 A/A + 30 A58 A/A + 30 A57 A/A + 30 A
57 A/A + 30 A57 A/A + 30 A57 A/A + 30 A71 A/A + 30 A0.017 % + 30 A
200 A/A + 60 A110 A/A + 60 A73 A/A + 60 A
72 A/A + 60 A72 A/A + 60 A72 A/A + 60 A72 A/A + 60 A90 A/A + 60 A0.013 % + 60 A
0.022 % + 120 A0.013 % + 120 A
86 A/A + 120 A85 A/A + 120 A85 A/A + 120 A85 A/A + 120 A85 A/A + 120 A0.011 % + 120 A0.016 % + 120 A
0.11 % of ratio
0.12 % of ratio
0.14 % of ratio0.14 % of ratio
Fluke 5790A,5700A/EP,A40, standard resistors
** Measure Only
Comparison tostandard coil
-
7/25/2019 Precision Metrology 1078-01
15/54
Parameter/Range Frequency CMC2, 5, 6() Comments
AC Current Generate3
(cont)
(11 to 20.5) A (45 to 100) Hz
100 Hz to 1 kHz
(1 to 5) kHz
0.094 % + 3.9 mA
0.12 % + 3.9 mA
2.4 % + 3.9 mA
Fluke 5520A
AC Current Measure3
(0 to 100) A
(0.1 to 1) mA
(1 to 10) mA
(10 to 100) mA
(0.1 to 1) A
(10 to 20) Hz(20 to 45) Hz(45 to 100) Hz100 Hz to 5 kHz
(10 to 20) Hz
(20 to 45) Hz(45 to 100) Hz100 Hz to 5 kHz(5 to 20) kHz(20 to 50) kHz(50 to 100) kHz
(10 to 20) Hz(20 to 45) Hz(45 to 100) Hz
100 Hz to 5 kHz(5 to 20) kHz(20 to 50) kHz(50 to 100) kHz
(10 to 20) Hz(20 to 45) Hz(45 to 100) Hz100 Hz to 5 kHz(5 to 20) kHz(20 to 50) kHz(50 to 100) kHz
(10 to 20) Hz(20 to 45) Hz(45 to 100) Hz100 Hz to 5 kHz
0.27 % + 20 nA0.11 % + 20 nA0.042 % + 20 nA0.042 % + 20 nA
0.27 % + 0.14 A
0.11 % + 0.14 A0.042 % + 0.14 A0.023 % + 0.14 A0.042 % + 0.14 A0.27 % + 0.27 A0.37 % + 1 A
0.27 % + 1.4 A0.11 % + 1.4 A0.042 % + 1.4 A
0.023 % + 1.4 A0.042 % + 1.4 A0.27 % + 2.7 A0.37 % + 10 A
0.27 % + 14 A0.11 % + 14 A0.042 % + 14 A0.023 % + 14 A0.042 % + 14 A0.27 % + 27 A0.37 % + 100 A
0.27 % + 140 A0.11 % + 140 A0.057 % + 140 A0 070 % + 140 A
Agilent/HP 3458A
-
7/25/2019 Precision Metrology 1078-01
16/54
Parameter/Range Frequency CMC2, 6() Comments
AC Current Measure3(cont)
(1 to 2) A
(2 to 20) A
50 A to 1 kA
10 Hz to 2 kHz(2 to 10) kHz(10 to 30) kHz
10 Hz to 2 kHz(2 to 10) kHz
60 Hz
0.057 % + 190 A0.068 % + 190 A0.25 % + 1.4 A
0.072 % + 1.9 mA0.20 % + 1.9 mA
2.0 %
Fluke 8508
Fluke 8508
Fluke 337
AC Power3 PF = 1
(29 to 330) A(0.33 to 3.3) mA
3.3 mA to 3.3 A
(29 to 330) A(0.33 to 3.3) mA(3.3 to 330) mA330 mA to 3 A
(29 to 330) A(0.33 to 3.3) mA(3.3 to 330) mA
330 mA to 1.1 A(1.1 to 3) A(3 to 11) A(11 to 20.5) A
(29 to 330) A(0.33 to 3.3) mA(3.3 to 33) mA(33 to 330) mA330 mA to 1.1 A
(1.1 to 3) A(3 to 11) A(3 to 20.5) A
(29 to 330) A(0.33 to 3.3) mA(3.3 to 33) mA
(10 to 20) Hz1 mV to 33 V
(20 to 45) Hz1 mV to 33 V
(45 to 100) Hz1 mV to 1020 V
100 Hz to 1 kHz1 mV to 1020 V
(1 to 5) kHz1 mV to 1020 V
0.19 %0.17 %
0.15 %
0.15 %0.11 %0.080 %0.15 %
0.13 %0.088 %0.041 %
0.052 %0.054 %0.065 %0.12 %
0.13 %0.088 %0.041 %0.041 %0.052 %
0.054 %0.095 %0.14 %
0.28 %0.17 %0.070 %
Fluke 5520A
-
7/25/2019 Precision Metrology 1078-01
17/54
Parameter/Range Frequency CMC2, 6, 7() Comments
AC Power3 PF = 1(cont)
(1.1 to 3) A(3 to 11) A(11 to 20.5) A
(29 to 330) A(0.33 to 3.3) mA(3.3 to 33) mA(33 to 330) mA330 mA to 1.1 A(1.1 to 3) A
(29 to 330) A(0.33 to 3.3) mA(3.3 to330) mA
(1 to 5) kHz1 mV to 1020 V
(5 to 10) kHz1 mV to 1020 V
(10 to 30) kHz1 mV to 330 V
0.26 %2.4 %2.4 %
0.67 %0.40 %0.17 %0.18 %2.3 %2.1 %
1.3 %0.79 %0.32 %
Fluke 5520A
AC Voltage Measure& Generate3
(0.6 to 2.2) mV
(2.2 to 7) mV
(7 to 22) mV
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz
(100 to 300) kHz(300 to 500) kHz500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz
(300 to 500) kHz500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz
0.14 % + 1.1 V0.062 % + 1.1 V0.040 % + 1.1 V0.067 % + 1.6 V0.097 % + 2.0 V
0.19 % + 3.2 V0.19 % + 6.3 V0.31 % + 6.3 V
0.068 % + 1.1 V0.032 % + 1.1 V0.021 % + 1.1 V0.034 % + 1.6 V0.049 % + 2.0 V0.096 % + 3.2 V
0.11 % + 6.3 V0.19 % + 6.3 V
0.024 % + 1.1 V0.016 % + 1.1 V0.0096 % + 1.1 V0.018 % + 1.6 V
Fluke 5790A, 5720A
-
7/25/2019 Precision Metrology 1078-01
18/54
Parameter/Range Frequency CMC2, 7() Comments
AC Voltage Measure& Generate3(cont)
(22 to 70) mV
(70 to 220) mV
(220 to 700) mV
(0.7 to 2.2) V
(2.2 to 7) V
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz(300 to 500) kHz500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz
(100 to 300) kHz(300 to 500) kHz500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz
(300 to 500) kHz500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz(300 to 500) kHz
500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz
0.022 % + 1.2 V0.012 % + 1.2 V0.0071 % + 1.2 V0.012 % + 1.6 V0.025 % + 2.0 V0.048 % + 3.2 V0.063 % + 6.3 V0.096 % + 6.3 V
0.017 % + 1.2 V0.0076 % + 1.2 V0.0037 % + 1.2 V0.0058 % + 1.6 V0.013 % + 2.0 V
0.022 % + 3.2 V0.031 % + 6.3 V0.079 % + 6.3 V
0.017 % + 1.2 V0.0061 % + 1.2 V0.0029 % + 1.2 V0.0046 % + 1.6 V0.0063 % + 2.0 V0.017 % + 4.3 V
0.025 % + 6.3 V0.074 % + 6.3 V
0.018 %0.0061 %0.0022 %0.004 %0.0057 %0.015 %0.023 %
0.074 %
0.016 %0.0054 %0.0024 %0.0041 %0.0065 %
Fluke 5790A, 5720A
-
7/25/2019 Precision Metrology 1078-01
19/54
Parameter/Range Frequency CMC2, 7() Comments
AC Voltage Measure& Generate3(cont)
(7 to 22) V
(22 to 70) V
(70 to 220) V*
(220 to 700) V
(700 to 1000) V**
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz(300 to 500) kHz500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz
(100 to 300) kHz(300 to 500) kHz500 kHz to 1 MHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz
(300 to 500) kHz
(10 to 20) Hz(20 to 40) Hz40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz
(10 to 20) Hz(20 to 40) Hz
40 Hz to 20 kHz(20 to 50) kHz(50 to 100) kHz
0.016 %0.0057 %0.0030 %0.0039 %0.0065 %0.018 %0.033 %0.097 %
0.016 %0.0058 %0.0034 %0.0050 %0.0075 %
0.016 %0.032 %0.09 %
0.016 %0.0058 %0.0034 %0.0056 %0.0080 %0.017 %
0.039 %
0.016 %0.0080 %0.0038 %0.011 %0.040 %
0.016 %0.0080 %
0.0036 %0.011 %0.040 %
Fluke 5790A, 5720A
* Subject to 2.2 x 107V-Hz limitation
Fluke 5790A, 5720A,5725A
** Measure only
-
7/25/2019 Precision Metrology 1078-01
20/54
Parameter/Range Frequency CMC2, 5() Comments
AC High Voltage Measure3
(1 to 5) kV(5 to 35) kV(35 to 75) kV
(50 to 60) Hz 0.2 % + 1 V0.4 % + 65 V0.4 % + 49 V
Vitrek 4700A
AC High Voltage Generate3
(1 to 5) kV 60 Hz 0.24 % + 1 V Assoc. res. 3565D
AC Voltage Measure32 MHz
(0 to 10) mV
(10 to 100) mV
(0.1 to 1) V
(1 to 10) V
(1 to 40) Hz40 Hz to 1 kHz(1 to 20) kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz
(1 to 40) Hz40Hz to 1kHz(1 to 20) kHz
(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz300 kHz to 1 MHz(1 to 2) MHz
(1 to 40) Hz40 Hz to 1kHz(1 to 20) kHz(20 to 50) kHz
(50 to 100) kHz(100 to 300) kHz300 kHz to 1 MHz(1 to 2) MHz
(1 to 40) Hz40 Hz to 1 kHz
0.023 % + 2 V0.018 % + 0.74 V0.026 % + 0.74 V0.069 % + 0.74 V0.34 % + 0.74 V2.7 % + 1.4 V
0.033 % + 2.7 V0.014 % + 1.4 V0.019 % + 1.4 V
0.037 % + 1.4 V0.067 % + 1.4 V0.22 % + 6.7 V0.68 % + 6.7 V1.1 % + 6.7 V
0.0047 % + 27 V0.0047 % + 14 V0.0094 %+ 14 V0.020 % + 14 V
0.054 % + 14 V0.20 % + 67 V0.67 % + 67 V1.1 % + 67 V
0.0077 % + 0.27 mV0 0062 % + 0 14 mV
Agilent/HP 3458A
-
7/25/2019 Precision Metrology 1078-01
21/54
Parameter/Range Frequency CMC2, 5, 7, 8() Comments
AC Voltage Measure3(cont) 2 MHz
(10 to 100) V
(100 to 700) V
(1 to 40) Hz40 Hz to 1 kHz(1 to 20) kHz(20 to 50) kHz(50 to 100) kHz(100 to 300) kHz300 kHz to 1 MHz
(1 to 40) Hz40 Hz to 1 kHz(1 to 20) kHz(20 to 50) kHz(50 to 100) kHz
0.025 % + 2.7 mV0.016 % + 1.4 mV0.014 % + 1.4 mV0.025 % + 1.4 mV0.082 % + 1.4 mV0.27 % + 6.7 mV1.1 % + 6.7 mV
0.033 % + 27 mV0.027 % + 14 mV0.041 % + 14 mV0.081 % + 14 mV0.21 % + 14 mV
Agilent/HP 3458A
Capacitance Measure3
1 pF10 pF100 pF to 1 F
(0.7 to 110) mF
10 pf to 0.1 F
1 kHz
DC
12 Hz to 2 MHz
10 F/F13 F/F14 F/F
0.012 %
See Table IV.a
AH2500A
5700A w/ 3458A,charge method
1689M , Quad tech 7600
Capacitance Generate3,Fixed Points
100 pF0.001 F0.002 F(0.01, 0.02, 0.05) F(0.1, 0.5, 1) F
100 Hz, 1 kHz 0.054 %0.068 %0.056 %0.055 %0.055 %
Standard capacitors
-
7/25/2019 Precision Metrology 1078-01
22/54
Table IV.a: CMC (in %) for Capacitance measured with 1689M and 7600 LCR Meter, Fixed Points
10 pF 100 pF 1 nF 10 nF 100 nF 1 F 10 F 100 F 0.001 F 0.01 F 0.1 F12 Hz 3.1 5 1.5 0.79 0.93 0.14 0.11 0.46 4.5
20 Hz 1.9 3 0.63 0.48 0.49 0.11 0.11 0.76
50 Hz 3.8 0.39 0.47 0.25 0.11 0.16 0.058 0.11 0.94
100 Hz 1.4 0.16 0.21 0.15 0.047 0.067 0.047 0.16 1.4
200 Hz 0.71 0.081 0.12 0.095 0.047 0.047 0.047 0.29 2.8
500 Hz 1.9 0.2 0.035 0.042 0.058 0.035 0.035 0.058 0.48 4.4
1 kHz 0.48 0.058 0.024 0.024 0.024 0.024 0.024 0.058 0.48 4.7
2 kHz 0.71 0.081 0.047 0.035 0.035 0.041 0.035 0.17 1.2
5 kHz 0.66 0.083 0.056 0.047 0.047 0.056 0.08 0.32 2.810 kHz 0.42 0.07 0.056 0.055 0.055 0.06 0.11 0.6
20 kHz 0.035 0.024 0.048 0.057 0.058 0.068 0.18 1.3
50 kHz 0.024 0.024 0.066 0.066 0.07 0.11 0.46 4.1
100 kHz 0.024 0.03 0.08 0.081 0.091 0.2 1.2
200 kHz 0.27 0.11 0.11 0.12 0.15 0.46 3.7
500 kHz 0.43 0.4 0.2 0.38 0.38 2.1
1 MHz 0.36 0.35 0.35 0.41 1.1
2 MHz 0.65 0.64 0.66 0.9 3.4
Parameter/Equipment Range CMC2, 7() Comments
DC Power3
33 mV to 1020 V 330 A to 330 mA330 mA to 3 A(3 to 20.5) A
0.068 %0.07 %0.12 %
Fluke 5520A
Parameter/Equipment Range CMC2, 6, 8 () Comments
DC Current Generate3 (10 to 20) A
(20 to 100) A(100 to 300) A
31 A/A + 12 A
27 A/A + 18 A/C0 011 % + 18 A/C
Fluke 5520A,3458A/HFL, andstandard resistor
HP power supply,shunt 3458A/HFL
-
7/25/2019 Precision Metrology 1078-01
23/54
Parameter/Equipment Range CMC2, 5, 6, 8 () Comments
DC Current Measure3 (0 to 100) nA(0.1 to 1) mA(1 to 10) mA
(10 to 100) mA(100 to 1) mA(1 to 10) mA
(10 to 100) mA(0.1 to 1) A(1 to 2) A(2 to 10) A(10 to 20) A
(20 to 100) A(100 to 300) A(300 to 1000) A
30 A/A + 0.27 nA19 A/A + 0.27 nA17 A/A + 0.067 nA
5.2 A/A + 0.01 nA4.9 A/A + 0.1 nA5.1 A/A + 1 nA
5.6 A/A + 60 nA7.3 A/A + 6 A8.9 A/A + 12 A31 A/A + 6 A31 A/A + 12 A
27 A/A + 8.7 A/C0.011 % + 8.7 A/C0.077 % + 8.7 A/C
Agilent/HP 3458A(90day cal)
Agilent/HP3458A/HFLFluke 742 resistor
standards
Fluke 3458A withshunts
DC Voltage Generate,Fixed Points
100 mV
1 V
10 V
19 V
100 V
1000 V
0.82 V/V
0.63 V/V
0.55 V/V
1.1 V/V
0.63 V/V
0.82 V/V
Fluke 732B, 752A,HP 34420A
Fluke 732B, 752A,HP 34420A
Fluke 732B
Fluke 732B, 720A,
752A,HP 34420A
Fluke 732B, 752A,HP 34420A
Fluke 732B, 752A,HP 34420A
DC Voltage Generate (0 to 220) mV220 mV to 2.2 V(2.2 to 11) V(11 to 22) V(22 to 220) V(220 to 1100) V
5.6 V/V + 0.39 V3.2 V/V + 0.63 mV2.4 V/V + 2.4 V2.4 V/V + 3.9 V3.2 V/V + 39 V4.8 V/V + 390 V
Fluke 5700A/EP
-
7/25/2019 Precision Metrology 1078-01
24/54
Parameter/Equipment Range CMC2, 5, 8() Comments
DC Voltage Measure3 Up to 100 mV100 mV to 1 V(1 to 10) V(10 to 100) V(100 to 1000) V
3.0 V/V + 0.21 V2.1 V/V + 0.21 V2.1 V/V + 0.34 V3.5 V/V + 21 V3.7 V/V + 74 V
Agilent/HP 3458A
DC High Voltage
Measure3
(1 to 6) kV
(6 to 35) kV(35 to 100) kV
0.2 % + 1 V
0.4 % + 26 V0.4 % + 140 V
Vitrek 4700A
DC High Voltage Generate3
(1 to 6) kV 0.21 % + 1.1 V Assoc. res. 3565Dmonitored with Vitrek4700A
Edge Characteristics
Amplitude (0 to 100) mV
(0.1 to 1) V
(1 to 10) V(10 to 100) V
3.0 V/V + 0.21 V2.1 V/V + 0.21 V2.1 V/V + 0.34 V3.5 V/V + 21 V
Agilent/HP 3458A/HFL
Electrical Calibration of
RTD IndicatingDevices3
Pt 385, 100
Pt 3926, 100
-200 C to 0 C0 C to 100 C100 C to 300 C300 C to 400 C400 C to 630 C630 C to 800 C
-200 C to 0 C0 C to 100 C100 C to 300 C300 C to 400 C400 C to 630 C
0.05 C0.07 C0.09 C0.1 C0.12 C0.14 C
0.05 C0.07 C0.09 C0.1 C0.26 C
Fluke 5520A
-
7/25/2019 Precision Metrology 1078-01
25/54
Parameter/Equipment Range CMC2() Comments
Electrical Calibration ofRTD IndicatingDevices3(cont)
Pt 385, 200
Pt 385, 500
Pt 385, 1000
Pt 3916, 100
-200 C to 100 C100 C to 260 C260 C to 300 C
300 C to 400 C400 C to 600 C600 C to 630 C
-200 C to -80 C-80C to 100 C100 C to 260 C260 C to 400 C400 C to 600 C
600 C to 630 C
-200 C to 0 C0 C to 100 C100 C to 260 C260 C to 300 C300 C to 600 C600 C to 630 C
-200 C to -190 C-190 C to -80 C-80 C to 0 C0 C to 100 C100 C to 260 C260 C to 300 C300 C to 400 C400 C to 600 C600 C to 630 C
0.04 C0.05 C0.12 C
0.13 C0.14 C0.16 C
0.04 C0.05 C0.06 C0.08 C0.09 C
0.11 C
0.03 C0.04 C0.05 C0.06 C0.07 C0.23 C
0.25 C0.04 C0.05 C0.06 C0.07 C0.08 C0.09 C0.10 C0.23 C
Fluke 5520A
Electrical Calibration ofThermocoupleIndicating Devices3
Type E -250 C to -100 C-100 C to -25 C
0.5 C0 16 C
Fluke 5520A
-
7/25/2019 Precision Metrology 1078-01
26/54
Parameter/Equipment Range CMC2() Comments
Electrical Calibration ofThermocoupleIndicating Devices3(cont)
Type J
Type K
Type S
Type T
-210 C to -100 C-100 C to -30 C-30 C to 150 C150 C to 760 C760 C to 1200 C
-200 C to -100 C-100 C to -25 C-25 C to 120 C120 C to 1000 C1000 C to 1372 C
0 C to 250 C250 C to 1000 C1000 C to 1400 C1400 C to 1767 C
-250 C to -150 C-150 C to 0 C0 C to 120 C120 C to 400 C
0.28 C0.16 C0.14 C0.17 C0.23 C
0.34 C0.18 C0.16 C0.26 C0.40 C
0.47 C0.36 C0.37 C0.46 C
0.63 C0.24 C0.16 C0.14 C
Fluke 5520A
-
7/25/2019 Precision Metrology 1078-01
27/54
Parameter/Range Frequency CMC2, 8() Comments
High FrequencyCapacitance, FixedPoints3
1 pF
10 pF
100 pF
1000 pF
1 kHz1 MHz2 MHz
3 MHz4 MHz5 MHz10 MHz13 MHz
1 kHz(1, 2) MHz3 MHz
4 MHz5 MHz10 MHz13 MHz
1 kHz1 MHz2 MHz3 MHz
4 MHz5 MHz10 MHz13 MHz
1 kHz1 MHz2 MHz3 MHz
4 MHz5 MHz10 MHz13 MHz
73 parts in 1060.01 %0.023 %
0.041 %0.063 %0.088 %0.26 %0.38 %
62 parts in 10662 parts in 10664 parts in 106
67 parts in 106
72 parts in 1060.013 %0.017 %
61 parts in 10662 parts in 10668 parts in 10682 parts in106
0.01 %0.014 %0.034 %0.049 %
63 parts in 10680 parts in 1060.016 %0.028 %
0.046 %0.064 %0.2 %0.29 %
Agilent/HP 16380Aseries capacitors
-
7/25/2019 Precision Metrology 1078-01
28/54
Parameter/Range Frequency CMC2, 8 () Comments
High FrequencyCapacitance, FixedPoints3 (cont)
0.01F
0.1 F
1 F
(0.12, 1, 10, 100) kHz
(0.12, 1, 10, 100) kHz
0.12 kHz(1, 10) kHz100 kHz
63 parts in 106
63 parts in 106
69 parts in 10663 parts in 10685 parts in 106
Agilent/HP 16380Aseries capacitors
Inductance3
Measure (12 Hz to
2 MHz)
Generate FixedPoints
0.1 H to 10 H
100 H(1, 5, 10, 100) mH(1, 2, 5) H
See Table IV.b
0.026 %0.011 %0.011 %
LCR meters
Standard inductors
Table IV.b: Inductance Measure CMC (in %) measured with 1689M and 7600 LCR Meters
Freq. 0.1 H 1 H 10 H 100 H 1 mH 10 mH 100 mH 1 H 5 H 10 H
12 Hz 0.81 1.3 1.1 1.1 1.1
20 Hz 4.8 0.49 0.71 0.58 0.56 0.56
50 Hz 0.97 0.11 0.17 0.26 0.26 0.26
100 Hz 3.6 0.37 0.048 0.069 0.11 0.16 0.16
200 Hz 1.8 0.19 0.048 0.048 0.058 0.11 0.085
500 Hz 4.8 0.49 0.060 0.036 0.036 0.036 0.051 0.036
1 kHz 1.2 0.13 0.024 0.024 0.024 0.024 0.024 0.024
2 kHz 0.90 0.13 0.036 0.042 0.036 0.036 0.048 0.0485 kHz 3.2 0.37 0.084 0.048 0.062 0.062 0.055 0.079 0.096
10 kHz 1.7 0.22 0.079 0.064 0.062 0.064 0.069 0.097 0.26
20 kHz 0.94 0.15 0.074 0.066 0.066 0.067 0.081 0.030 0.048
50 kHz 4.7 0.54 0.12 0.080 0.075 0.075 0.16 0.50 0.058 0.11
100 kHz 3.4 0.42 0.13 0.095 0.091 0.092 0.21 1.9 0.11 0.20
-
7/25/2019 Precision Metrology 1078-01
29/54
Parameter/Range Frequency CMC2 () Comments
Phase Generate
5 Vrms (Voltage Ratio= 1)
50 m to 100 Vrms
(100 to 120) Vrms
(0.1 to 100) Vrms
1 Hz to 1 kHz(1 to 6.25) kHz(6.25 to 50) kHz(50 to 200) kHz
1 Hz to 1 kHz
(1 to 6.25) kHz(6.25 to 50) kHz(50 to 200) kHz
1 Hz to 1 kHz(1 to 6.25) kHz(6.25 to 50) kHz(50 to 200) kHz
1 Hz to 1 kHz(1.001 to 6.25) kHz(6.25 to 50) kHz(50.01 to 100) kHz
6.6 m5.2 m13 m21 m
(6.5 + (0.05 Ratio))m
(11 + (0.1 Ratio))m(19 + (0.15 Ratio))m(41 + (0.4 Ratio))m
(15 + (0.1 Ratio))m(21 + (0.2 Ratio))m(31 + (0.3 Ratio))m(81 + (1 Ratio))m
13 m13 m20 m42 m
Clark-Hess 5500-2Ratio= ratio of thelarger voltage dividedby the smallervoltage
Phase Measure
(0.01 to 0.1) Vrms
(0.1 to 120) Vrms
10 Hz to 10 kHz(10 to 50) kHz
(50 to 100) kHz
10 Hz to 50 kHz(50 to 100) kHz
0.210.36
0.71
0.0540.036
KH 6620 phase meter
-
7/25/2019 Precision Metrology 1078-01
30/54
Parameter/Equipment Range CMC2, 6() Comments
Oscilloscope Calibration3
Squarewave Signal50 at 1 kHz Source
Squarewave Signal1 Mat 1 kHz Source
Leveled Sine WaveFlatness (relative to 50kHz)
Period
Rise Time Generate
Rise Time Measure
(1 to 110) mV110 mV to 2.2 V(2.2 to 11) V(11 to 1100) V
(1 to 110) mV110 mV to 2.2 V(2.2 to 11) V(11 to 1100) V
50 kHz to 100 MHz(100 to 300) MHz(300 to 600) MHz(600 to 1100) MHz
1 ns to 20 ms50 ms100 ms200 ms500 ms1 s2 s5 s
12.8 ps
> 5.4ps
0.27 % + 42 V0.27 % + 130 V0.27 % + 1.2 mV0.27 % + 12 mV
0.15 % + 42 V0.15 % + 130 V0.15 % + 1.2 mV0.15 % + 12 mV
1.5 % + 110 V2 % + 110 V4 % + 110 V5 % + 110 V
2.0 parts in 10659 parts in 10697 parts in 106180 parts in 106410 parts in 106800 parts in 1061600 parts in 1063900 parts in 106
3.9 ps
5.4 ps
Fluke 5520A/SC1100scope option
Tektronix 067-1338-00
Tektronix TDS8000with 80E03
Resistance Generate,Fixed Points3
100
1 m 10 m 0.1
1 10 100
29 /
14 /9.5 /6.5 /
4.6 /4.2 /2 3 /
Norma 80
L&N 4222BL&N 4223B
L&N resistors
Fluke 742A, standard
-
7/25/2019 Precision Metrology 1078-01
31/54
Parameter/Equipment Range CMC2, 8 () Comments
Resistance Generate,Fixed Points3(cont)
10 G100 G1 T10 T
0.13 %0.22 %0.34 %1.2 %
IET high resistancestandard
Resistance Measure,
DC
100
100 to1 m(1 to 10) m(10 to100) m(0.1 to 1)
(0 to 2) (2 to 20) (20 to 200) 200 to 2 k(2 to 20) k
(20 to 200) k200 kW to 2 M(2 to 20) M(20 to 200) M200 Mto 2 G(2 to 20) G
100 Mto 1 G(1 to 10) G(10 to 100) G100 Gto 1 T(1 to 10) T(10 to 100) T
85 /+ 0.48 n
12 /+ 4.8 n9.2 /+ 4.8 n5.4 /+ 48 n2.6 /+ 0.48
3.4 /+ 4.4 2.1 /+ 15 2.6 /+ 33 2.5 /+ 0.33 m1.5 /+ 3.3 m
2.3 /+ 33 mW4.4 /+ 1.1 4.4 /+ 11 8.3 /+ 110 28 /+ 110 k0.21 % + 1.1 M
0.081 %0.12 %0.24 %
0.35 %0.59 %1.2 %
Stable current source,
3458A/HFL, standardresistors
Fluke 8508A andstandard resistors
6500A
Resistance Measure,AC
10 Hz to 2 MHz See Table IV.c LCR meters
-
7/25/2019 Precision Metrology 1078-01
32/54
Table IV.c: AC Resistance Measure CMC (in %) measured with LCR Meters
0.1 1 10 100 1 k 10 k 100 k 200 k 500 k 1 M12 Hz 0.60 0.11 0.11 0.11 0.11 0.11 1.720 Hz 0.60 0.11 0.11 0.11 0.11 0.11 0.85 2.8 4.650 Hz 3.1 0.31 0.058 0.058 0.058 0.058 0.058 0.38 1.3 2.0
100 Hz 2.3 0.23 0.047 0.047 0.047 0.047 0.047 0.23 0.70 1.2200 Hz 2.3 0.23 0.047 0.047 0.047 0.047 0.047 0.15 0.44 0.70
500 Hz 1.5 0.16 0.035 0.035 0.035 0.035 0.035 0.10 0.29 0.441 kHz 0.77 0.09 0.024 0.024 0.024 0.024 0.024 0.082 0.24 0.35
2 kHz 1.0 0.15 0.035 0.035 0.035 0.035 0.047 0.075 0.21 0.315 kHz 0.90 0.14 0.047 0.047 0.047 0.047 0.063 0.072 0.20 0.30
10 kHz 1.0 0.14 0.062 0.055 0.055 0.055 0.063 0.073 0.21 0.3020 kHz 1.1 0.16 0.066 0.057 0.057 0.057 0.067 0.077 0.22 0.3350 kHz 1.4 0.19 0.077 0.066 0.065 0.066 0.31 0.37 1.1 1.7
100 kHz 1.9 0.26 0.10 0.081 0.079 0.081 0.40 0.47 1.5 2.2200 kHz 3.0 0.39 0.14 0.11 0.11 0.12 0.56 0.68 2.2 3.4500 kHz 0.79 0.26 0.20 0.20 0.26 1.1 1.4 4.21 MHz 1.5 0.45 0.35 0.35 0.70 1.9 2.42 MHz 2.8 0.85 0.65 0.64 1.4 3.5 4.5
-
7/25/2019 Precision Metrology 1078-01
33/54
V. Electrical RF/Microwave
Parameter/Range Frequency CMC2, 8() Comments
Amplitude Modulation3
AM Accuracy:
(0.15 to 10) MHz
5 % to 40 % AM Depth
40 % to 99 % AM Depth
(10 to 1300) MHz
5 % to 40 % AM Depth
40 % to 99 % AM Depth
(1.3 to 26.5) GHz
5 % to 40 % AM Depth
40 % to 99 % AM Depth
AM Distortion:
150 kHz to 1.3 GHz
(1.3 to 26.5) GHz
AM Flatness:(0.01 to 26.5) GHz
20 % to 80 % AM Depth
Rate:50 Hz to 10 kHz
Rate: 20 Hz to 10 kHz
Rate 50 Hz to 10 kHz
Rate: 20 Hz to 10 kHz
Rate: 50 Hz to 50 kHz
Rate: 20 Hz to 100 kHz
Rate: 50 Hz to 50 kHz
Rate: 20 Hz to 100 kHz
Rate: 50 Hz to 50 kHz
Rate: 20 Hz to 100 kHz
Rate: 50 Hz to 50 kHz
Rate: 20 Hz to 100 kHz
< 50 % AM Depth
50 % to 95 % AM Depth
< 50 % AM Depth
50 % to 95 % AM Depth
Rate: 90 Hz to 10 kHz
2.3 % + 0.012 % AM
3.5 % + 0.012 % AM
2.3% + 0.12 % AM
3.5 % + 0.12 % AM
1.2 % + 0.012 % AM
3.5 % + 0.012 % AM
1.2 % + 0.12 % AM
3.5 % + 0.12 % AM
1.2 % + 0.012 % AM
3.5 % + 0.012 % AM
1.2 % + 0.12 % AM
3.5 % + 0.12 % AM
1.1 dB
2.1 dB
1.1 dB
2.1 dB
0.31 %
Measuring
receiver
Measuring
receiver, down
converter and
local oscillator
Measuring
receiver and
distortion analyzer
Measuring
receiver
Di t ti A
-
7/25/2019 Precision Metrology 1078-01
34/54
Parameter/Range Frequency CMC2, 8() Comments
Frequency Modulation3
FM Accuracy(0.15 to 10) MHz< 4 kHz FM peak Dev.< 40 kHz FM peak Dev.
(10 to 1300) MHz< 400 kHz FM pk Dev.
(1.3 to 26.5) GHz< 400 kHz FM pk Dev.
FM Distortion
400 kHz to 10 MHzRate: 20 Hz to 10 kHz
10 MHz to 26.5 GHzRate: 20 Hz to 100 kHz
Residual FMCarrier Frequency ()
< 100 MHz
(100 to 1300) MHz(1.3 to 6.2) GHz(6.2 to 12.4) GHz(12.4 to 18) GHz
Rate:
20 Hz to 10 kHz20 Hz to 10 kHz
50 Hz to 100 kHz20 Hz to 200 kHz
50 Hz to 100 kHz20 Hz to 200 kHz
< 10 kHz FM Deviation
< 100 kHz FM Deviation
Bandwidth:50 Hz to 3 kHz
50 Hz to 3 kHz50 Hz to 3 kHz50 Hz to 3 kHz50 Hz to 3 kHz
2.4 % + 1.2 Hz2.4 % + 12 Hz
1.3 % + 120 Hz5.8 % + 240 Hz
1.3 % + 120 Hz5.8 % + 100 Hz
0.15 %
0.15 %
1 Hz (rms)
0.4 Hz + (6 x 10
-9
) 17 Hz33 Hz49 Hz
Measuring receiver
Measuring receiverand distortionanalyzer
Measuring receiver
Phase Modulation3
(0.15 to 10) MHz< 40 Radians (peak)
< 400 Radians (peak)
(10 to 1300) MHz< 4 Radians (peak)< 40 Radians (peak)< 400 Radians (peak)
Rate: 200 Hz to 10 kHz
Rate: 200 Hz to 20 kHz
4.7 % + 0.012 rad
4.7 % + 0.12 rad
5.2 % + 0.0012 rad3.5 % + 0.012 rad3.5 % + 0.12 rad
Measuring receiver
-
7/25/2019 Precision Metrology 1078-01
35/54
Parameter/Range Frequency CMC2, 8() Comments
Range Calibration (PowerMeters)3
Zero SetInstrument Accuracy 3 W
10 W30 W(100, 300) W
(1, 2) mW(10, 30, 100) mW
0.25 % + 6.2 nW1.6 %0.46 %0.28 %0.26 %
0.28 %0.30 %
Range calibrator,Agilent 11683A orAgilent 8477A
RF Power Level3Generate
Absolute
Relative
(Power SensorCalibration Factors)
Relative
(Power SensorCalibration Factors)
0 dBm @ 50 MHz
+20 to - 20 dBm0.1 MHz to 26.5 GHz
-20 to -127 dBM0.9 MHz to 1.3 GHz
-20 to -100 dBm(1.3 to 26.5) GHz
100 kHz
300 kHz1 MHz3 MHz10 MHz30 MHz100 MHz300 MHz500 MHz
1 GHz
2 GHz3 GHz4 GHz5 GHz6 GHz7 GHz8 GHz
0.11 dB +M
0.12 dB +M
0.18 dB +M
0.18 dB +M
0.84 % + M
0.53 % + M0.47 % +M0.47 % +M0.45 % +M0.45 % +M0.55 % +M0.62 % +M0.62 % +M
0.67 % +M
0.78 % +M0.95 % +M1.1 % +M1.2 % +M1.2 % +M1.2 % +M1 2 % + M
Signal generator,power splitter,
attenuator, standardsensors, and powermeter
Signal generator,
power splitter,attenuator, standardsensors, and powermeter. Referencedto, 1 mW
UUT Mismatchuncertainty (M)
Signal generator,
power splitter,attenuator, standardsensors, and powermeter. Referencedto, 1 mW
UUT Mismatch
-
7/25/2019 Precision Metrology 1078-01
36/54
Parameter/Range Frequency CMC2, 8() Comments
RF Power Level3Generate (cont)
Relative(Power SensorCalibration Factors)
16 GHz17 GHz18 GHz19 GHz20 GHz21 GHz
22 GHz23 GHz24 GHz25 GHz26 GHz26.5 GHz
2.2 % +M1.8 % +M2.9 % +M4.1 % +M5.2 % +M4.7 % +M
3.5 % +M2.8 % +M3.1 % +M3.9 % +M3.9 % +M3.9 % +M
Signal generator,power splitter,attenuator, standardsensors, and powermeter. Referencedto, 1 mW
UUT Mismatchuncertainty (M)
RF Power Level3 Measure
Absolute 0 dBm @ 50 MHz
+45 to 20 dBm0.1 MHz to 4.2 GHz
+20 to - 20 dBm0.1 MHz to 26.5 GHz
-20 to -127 dBM0.9 MHz to 1.3 GHz
-20 to -100 dBm(1.3 to 26.5) GHz
0.04 dB +M
0.06 dB +M
0.05 dB +M
0.15 dB +M
0.15 dB +M
8478B, 432A
8482B, powermeter
8482A, 8481A,8485A, powermeter
11722A, 8902A
11792A, 8902A11793A, E8244A
UUT Mismatchuncertainty (M)
-
7/25/2019 Precision Metrology 1078-01
37/54
Parameter/Range Frequency CMC2, 8() Comments
RF Power Level3Measure, Relative (cont)
0 dB
-10 dB-20 dB-30 dB-40 dB-50 dB-60 dB-70 dB-80 dB-90 dB-100 dB
-110 dB-120 dB
-127 dB
+ 10 to -100 dBm0.9 MHz to 26.5 GHz
-100 to -127dBm0.9 MHz to 1.3 GHz
0.01 dB +M
0.04 dB +M0.04 dB +M0.04 dB +M0.08 dB +M0.08 dB +M0.08 dB +M0.08 dB +M0.10 dB +M0.10 dB +M0.10 dB +M
0.10 dB +M0.12 dB +M
0.19 dB +M
11722A, 11792A8902A, 11793A,E8244A
11722A, 8902A
UUT Mismatchuncertainty (M)
Power Sensors, FixedPoints3
100 kHz300 kHz1 MHz(10, 30, 50) MHz100 MHz(0.3, 0.5, 1) MHz2 GHz
3 GHz4 GHz5 GHz6 GHz7 GHz8 GHz9 GHz10 GHz11 GHz(12, 12.4) GHz
13 GHz14 GHz15 GHz16 GHz17 GHz18 GHz19 GHz
2.8 %0.98 %0.93 %0.92 %0.77 %0.83 %0.85 %
0.88 %0.87 %1.1 %1.1 %1.1 %1.1 %1.3 %1.4 %1.7 %1.4 %
1.3 %1.4 %1.7 %1.7 %1.4 %2.1 %3.2 %
Signal generator,power splitter,attenuator, standardsensors, and powermeter referenced to50MHz, 1W
-
7/25/2019 Precision Metrology 1078-01
38/54
Parameter/Equipment Range CMC2() Comments
S-Parameters3 ReflectionS11/22 Magnitude andPhase
3.5 mm (0.01 to 1.0) lin(0 to 0.01) lin(0.01 to 0.1) lin(0.01 to 0.5) lin
(0.5 to 1) lin
Magnitude(0 to 1) lin
Phase(0 to 0.2) lin
(0.2 to 1) lin
7 mm Magnitude
(0 to 1) lin
Phase(0 to 0.2) lin
(0.2 to 1) lin
N-Type Magnitude(0 to 1) lin
Phase
(1.5 to 26.5) GHz
300 kHz to 1.3 GHz(1.3 to 3.0) GHz(3.0 to 6.0) GHz
300 kHz to 1.3 GHz(1.3 to 3.0) GHz
(3.0 to 6.0) GHz
300 kHz to 1.3 GHz(1.3 to 3.0) GHz(3.0 to 6.0) GHz
300 kHz to 1.3 GHz(1.3 to 3.0) GHz(3.0 to 6.0) GHz
300 kHz to 1.3 GHz(1.3 to 3.0) GHz(3.0 to 6.0) GHz
300 kHz to 1.3 GHz(1.3 to 3.0) GHz(3.0 to 6.0) GHz
100 kHz to 0.3 GHz(0.3 to 1.3) GHz(1.3 to 3.0) GHz(3.0 to 6.0) GHz
(0.012 to 0.040) lin(7.4 to 180) deg(0.095 to 20) deg(0.54 to 2.3) deg
(0.50 to 1.1) deg
(0.005 to 0.015) lin(0.008 to 0.002) lin(0.015 to 0.032) lin
(180 to 2) deg(180 to 3) deg
(180 to 4.5) deg
(2 to 1) deg(3 to 1.5) deg(4.5 to 2) deg
(0.001 to 0.006) lin(0.002 to 0.01) lin(0.005 to 0.018) lin
(180 to 1) deg(180 to 1.5) deg(180 to 2) deg
(1 to 0.5) deg(1.5 to 0.5) deg(2 to 1) deg
0.0085 lin(0.005 to 0.018) lin(0.005 to 0.025) lin(0.01 to 0.05) lin
Network analyzerwith S-parametertest set, calibrationkit, and cables
-
7/25/2019 Precision Metrology 1078-01
39/54
Parameter/Range Frequency CMC2
() Comments
S-Parameters3Transmission S12/S21Magnitude andPhase
3.5 mm(0 to 20) dB
(20 to 40) dB
(10 to -90) dB
7 mm(10 to -90) dB
N-Type(10 to -90) dB
(1.5 to 26.5) GHz
(1.5 to 26.5) GHz
300 kHz to 1.3 GHz
(1.3 to 3.0) GHz
(3.0 to 6.0) GHz
300 kHz to 1.3 GHz
(1.3 to 3.0) GHz
(3.0 to 6.0) GHz
300 kHz to 1.3 GHz
(1.3 to 3.0) GHz
(3.0 to 6.0) GHz
(0.11 to 0.29) dB
(1.2 to 12) deg
(0.13 to 0.68) dB(1.5 to 19) deg
(0.3 to 6.0) dB(0.25 to 80) deg
(0.04 to 6.0) dB(0.35 to 90) deg
(0.06 to 9.0) dB(0.67 to 180) deg
(0.03 to 5.0) dB(0.26 to 80) deg
(0.03 to 5.0) dB(0.26 to 90) deg
(0.05 to 10) dB(0.54 to 180) deg
(0.03 to 6.0) dB(0.2 to 80) deg
(0.04 to 6.0) dB(0.44 to 90) deg
(0.1 to 9.0) dB(0.67 to 90) deg
Network analyzer
with S-parametertest set, calibrationkit, and cablesSystem verificationwith verification kit
-
7/25/2019 Precision Metrology 1078-01
40/54
VI. Fluid Quantities
Parameter/Equipment Range CMC2, 4, 8() Comments
Dynamic Viscosity3 (0 to 100 000) cP 0.83 % Standard viscosityfluids, PRT
Flow3 2 sccm to 10 slm
(10 to 1000) slm
0.26 %
0.51 %
Mass flow system
Hydrometers
BaumeLighter than Water
(80.5 to 57)API (81 to 57)
(57 to 10)API (57 to 10)
Heavier than Water
(0.72 to 21)API (10 to -11)
(0.72 to 67)
Specific Gravity:
1.00
(0.66 to 0.75)
(0.75 to 1)
(1 to 1.17)
(1 to 1.85)
0.0058 % + 0.6R
0.0072 % + 0.6R
0.006 % + 0.6R
0.0058 % + 0.6R
0.0058 % + 0.6R
Distilled water
Petroleum basedsolutions
Alcohol solutions
Sulfuric acid and/orglycerine solutions
Sulfuric acidsolutions
Kinematic Viscosity3 Zahn cups (1 to 5)Shell cups (1 to 6)
0.84 s + 0.24 %0.84 s + 0.24 %
Temperature bath,viscosity oils,
stopwatch
Volumetric Calibration (0 to 200) mL200 mL to 1 L(1 to 25) L
0.24 % + 130 nL0.24 % + 9 L0.24 % + 900 L
Gravimetric method
-
7/25/2019 Precision Metrology 1078-01
41/54
VII. Foundry-Industry Specific Calibrations
Parameter/Equipment Range CMC2, 4, 8() Comments
AFS Clay Tester3 --- 0.42 s Stopwatch
Cone Jolt ToughnessTester3
Cam MeasurementRise In CamCone Weight
0.030 in0.030 in(470 to 480) g
180 in200 in0.14 g
Micrometer feelergagescale
Friability Tester3 60 s 0.42 s Stopwatch
Hot Tensile Tester3
LoadTemperature
Gap Measure
130 psi450 F
0.030 in
0.38 psi0.7 F
200 in
Dead weightTemperaturemeasuring device
Feeler gage
Melt Point Apparatus3 (200 to 220) F 2.8 F Temperaturemeasuring device
Mold Strength Tester3 (0 to 50) psi 0.2 % + 0.6R Balances andweights
Moisture Teller3 (0 to 300) F 2 F Temperaturecalibrator
Permmeter3 (0 to 200) perms 6.2 perms Perm standard
Sand Rammer3(Compactability Tester)
--- 0.0066 in + 0.6R Impact rings
-
7/25/2019 Precision Metrology 1078-01
42/54
-
7/25/2019 Precision Metrology 1078-01
43/54
Parameter/Equipment Range CMC2, 4, 8() Comments
Cable Tensiometers/Wire Tension Meters
(0 to 1000) lb 1.1 lb + 0.6R Deadweights
Deadweight Testers
Piston and CylinderArea
Mass
Performance Test(Cross-Float)
(0 to 10 000) psi (12 + 3D) in
0.2 g
0.04 %
UMM
Class F weights andscales
Pressure calibrator withtransducer
Durometers3
Indenter Shape andExtension:
Durometer Spring
Types A, B, C, D, E,O and DO
Pressure foot orificediameter
Indentor extensionlength
Cone angle
Tip radius
Indenter thickness
Up to 45 N
3.2 m
3.2 m
1 arcmin
3.2 m
3.2 m
36 mN
ASTM D2240
Vision system
Durometer calibrator
Durometer Calibrator Scales A and D 0.2 % Scale and CMM
-
7/25/2019 Precision Metrology 1078-01
44/54
Parameter/Equipment Range CMC2, 4, 8() Comments
Dynamometer andForce Rings3
(0 to 10 000) lbf 0.11 % + 0.6R Load cells
Force Gages (0 to 500) lbf(200 to 10 000) lbf
0.0024 % + 0.007 lbf0.6 % + 1.2 lbf
DeadweightsLoad cell
Force Gages3 (0 to 250) lbf 0.12 % + 0.6R Deadweights
Indirect Verification ofBrinell HardnessTesters at TestCondition(s)3
(3000, 1500, 500) Kg (125 to 260) HBW 0.008d
ASTM E10
dis the mean of the n
mean test diameter inmillimeters.
Indirect Verification ofLeeb Hardness Testers
Up to 800 HLD 7.4 HLD Standard test block
Indirect Verification of
MicroindentationHardness Testers(Knoop and Vickers)3
Repeatability under
forces P(gf):
1 P< 500
100 HK 940100 HV 940
20 HK + 1.2 %13 HV + 3 %
ASTM E384
-
7/25/2019 Precision Metrology 1078-01
45/54
Parameter/Equipment Range CMC2() Comments
Indirect Verification ofRockwell Hardness andRockwell SuperficialHardness Testers3
HRA:(22 to 63) HRA(64 to 73) HRA(74 to 83) HRA
HRBW:(0 to 40) HRB(41 to 60) HRB
(61 to 80) HRB
HRC:(25 to 35) HRC(40 to 55) HRC(60 to 70) HRC
HRE:(57 to 63) HRE(64 to 81) HRE(82 to 99) HRE
HRF:(57 to 63) HRF(64 to 80) HRF(81 to 97) HRF
HR15N:(72 to 78) HR15N(80 to 88) HR15N(90 to 94) HR15N
HR30N:(46 to 55) HR30N(60 to 73) HR30N(78 to 86) HR30N
HR45N:(25 to 37) HR45N(43 to 61) HR45N(67 to 77) HR45N
HR15T:(61 to 67) HR15T(70 to 77) HR15T(80 to 91) HR15T
HR30T:(15 to 28) HR30T
0.7 HRA0.7 HRA0.7 HRA
0.7 HRB0.7 HRB
0.7 HRB
0.7 HRC0.7 HRC0.7 HRC
0.7 HRE0.7 HRE0.7 HRE
0.7 HRF0.7 HRF0.7 HRF
0.7 HR15N0.7 HR15N0.7 HR15N
0.7 HR30N0.7 HR30N0.7 HR30N
0.7 HR45N0.7 HR45N0.7 HR45N
0.7 HR15T0.7 HR15T0.7 HR15T
0.7 HR30T
-
7/25/2019 Precision Metrology 1078-01
46/54
Parameter/Equipment Range CMC2, 4, 8() Comments
Load Cells (0 to 500) lbf
(500 to 10 000) lbf
0.034 % + 0.00044 lbf
0.05 % + 0.024 lbf
Dead weights,comparison to masterload cell
Load Cells3 (0 to 10 000) lbf 0.1 % + 0.002 lbf Dead weights,comparison to master
load cell
Manometers (0 to 24) in H2O 0.00017 in H2O Hook gage
Mass 1 mg to 50 g(50 to 220) g(220 to 400) g
400 g to 1.2 kg(1.2 to 8.2) kg(8.2 to 30) kg
17 g35 g0.14 mg
3.7 mg12 mg0.11 g
By comparison
Mass3 (2 to 30) kg 0.2 g By comparison
Pressure/Vacuum
Gauges
3
(-15 to 1000) psi
Pneumatic(1000 to 5000) psi
(5000 to 10 000) psi
0.005 psi + 0.012 % fs
0.32 psi + 0.026 % fs
1.8 psi + 0.026 % fs
Pressure calibrator
Pressure transducers
Refractometers3 (0 to 100) % 0.02 % + 0.6R Standard solutions
Scales3 (1 to 50) mg51 mg to 10 g(11 to 50) g(51 to 300) g(301 to 1000) g(1 to 5) kg
0.016 mg + 0.58R0.03 mg + 0.58R0.054 mg + 0.58R0.24 mg + 0.58R3.8 mg + 0.58R11 mg + 0 58R
Verification with massstandards
-
7/25/2019 Precision Metrology 1078-01
47/54
Parameter/Equipment Range CMC2, 4, 8() Comments
Scales3(cont) Up to 0.5 lb(0.5 to 1) lb(1 to 2) lb(2 to 5) lb(5 to 10) lb(10 to 20) lb(20 to 50) lb
(50 to 100) lb(100 to 500) lb(500 to 1000) lb
0.000 0015 oz + 0.58R0.000 0030 oz + 0.58R0.000 011 oz + 0.58R0.000 14 oz + 0.58R0.000 34 oz + 0.58R0.000 68 oz + 0.58R0.000 93 oz + 0.58R
0.0043 oz + 0.58R0.0086 oz + 0.58R0.043 oz + 0.58R
Verification withmass standards
Scales ProportionalTesting3
Up to 750 lb(750 to 1500) lb(1500 to 3000) lb(3000 to 4500) lb
(4500 to 6000) lb
0.22 lb + 0.6R0.28 lb + 0.6R0.36 lb + 0.6R0.42 lb + 0.6R
0.48 lb + 0.6R
NIST Class Fweights
Spring Testers3
Force Scale
Deflection Accuracy
(0 to 1000) lbf
Up to 20 in
0.02 % + 0.6R
26 in + 0.6R
Weights, load cells;Ris the resolution ofthe force scale of thespring tester.
Gage blocks;Risthe resolution of thedeflection scale.
Tachometers3
Non-Contact
Contact
Up to 180 000 rpm
Up to 5000 rpm(5000 to 15 000) rpm
0.8R
0.024 % + 1.4 rpm0.032 % + 0.84 rpm
Function generator,LED
Tachometercalibrator
Tensile Testers3 (0 to 30 000) lbf 0.4 % + 0.08 % fs Load cells
-
7/25/2019 Precision Metrology 1078-01
48/54
Parameter/Equipment Range CMC2, 4, 8() Comments
Torque Calibration ofTorque Meters andSensors
(0.1 to 2000) ftlbf 0.14 % Weights
Torque Watches3 (0.5 to 215) inozf 0.08 % + 0.52 inozf Torque calibrator
Ultrasonic ThicknessTesters3
(0 to 10) in (800 + 13L) in Standard thicknessspecimen
IX. Optical Quantities
Parameter/Equipment Range CMC
2
() Comments
Gloss Meters3 20
60, 85
0.73 GU
0.54GU
Gloss standards; GUrepresents gloss units.
X. Thermodynamics
Parameter/Equipment Range CMC2, 4() Comments
Dew Point -20 C to 60 C -0.44 C + 0.6R Thunder Scientifichumidity chamber
Industrial PlatinumResistanceThermometers
-30 C to 75 C75 C to 200 C200 C to 250 C250 C to 300 C
0.024 C0.032 C0.041 C0.047 C
SPRT, temperaturebath, resistance bridge
-
7/25/2019 Precision Metrology 1078-01
49/54
Parameter/Equipment Range CMC2, 4() Comments
Liquid-in-GlassThermometers andTemperature MeasuringDevices
-30 C to150 C150 C to 250 C250 C to 300 C
0.02 C + 0.6R0.03 C + 0.6R0.04 C + 0.6R
Constant temperaturebath, SPRT
Relative Humidity Measure
Hygrometer
Psychrometer
(10 to 95) % RH
5 C to 35 C
0.7 % RH + 0.6R
0.1 C + 0.84R
Humidity generator,Thunder Scientific2500ST
Comparison to SPRT intemp bath
Temperature Measuring Equipment3
-10 C to 50 C
50 C to 650 C
0.04 C + 0.6R
0.045 C + 0.6R
Dry-wells
Thermocouples andThermocouple Wire
-80 C to 650 C 0.42 C Comparison to SPRTwith Agilent 3458A
XI. Time & Frequency
Parameter/Equipment Range CMC2 () Comments
Frequency MeasuringEquipment
10 MHz
0.01 Hz to 200 kHz
200 kHz to 26.5 GHz
3 parts in 1011
3 parts in 1011+ 30 nHz
3 parts in 1011
GPS
Function generatorwith ext. timebase
Signal generator withext timebase
-
7/25/2019 Precision Metrology 1078-01
50/54
Parameter/Equipment Range CMC2() Comments
Stopwatches & Timers3 10 s to 72 hr
(2 to 960) s
28 ms + 0.6 s/s
0.037 s/day
Function generator,frequency counter
Timometer
______________________________________
1This laboratory offers commercial calibration service and field calibration service.
2Calibration and Measurement Capability Uncertainty (CMC) is the smallest uncertainty of measurementthat a laboratory can achieve within its scope of accreditation when performing more or less routinecalibrations of nearly ideal measurement standards or nearly ideal measuring equipment. CMCsrepresent expanded uncertainties expressed at approximately the 95 % level of confidence, usuallyusing a coverage factor of k = 2. The actual measurement uncertainty of a specific calibrationperformed by the laboratory may be greater than the CMC due to the behavior of the customers deviceand to influences from the circumstances of the specific calibration.
3Field calibration service is available for this calibration and this laboratory meets A2LAR104 GeneralRequirements: Accreditation of Field Testing and Field Calibration Laboratoriesfor these calibrations.Please note the actual measurement uncertainties achievable on a customer's site can normally beexpected to be larger than the CMC found on the A2LA Scope. Allowance must be made for aspectssuch as the environment at the place of calibration and for other possible adverse effects such as thosecaused by transportation of the calibration equipment. The usual allowance for the actual uncertaintyintroduced by the item being calibrated, (e.g. resolution) must also be considered and this, on its own,could result in the actual measurement uncertainty achievable on a customers site being larger than theCMC.
4In the statement of CMC,Lis the numerical value of the nominal length in inches;Ris the resolution ofthe unit under test; D is the diameter in inches; H is the height of the unit under test (except wherenoted); and fs represents full scale.Rais the numerical value of the nominal roughness of the surfacemeasured in micrometer roughness, except where noted.
5CMC for calibrations performed in the laboratory with the Agilent/HP 3458A/HFL is based upon 90-day specifications. CMC for calibrations performed field with the Agilent/HP 3458A is based upon 1-year specifications. The measurands stated are generated with the Agilent/HP 3458A. This capability issuitable for the calibration of the devices intended to measure the stated measurand in the rangesindicated. CMC are expressed as either a specific value that covers the full range or as a fraction of thereading plus a fixed floor specification. Unless otherwise noted, percentages are defined as percent ofreading.
6 CMC f lib ti f d ith th Fl k 5520A i b d 1 ifi ti CMC f
7 CMC f lib ti f d ith th Fl k 5790A i b d 1 ifi ti Th
-
7/25/2019 Precision Metrology 1078-01
51/54
7 CMC for calibrations performed with the Fluke 5790A is based upon 1-year specifications. Themeasurands stated are generated with the Fluke 5700 series of instruments. This capability is suitablefor the calibration of the devices intended to measure the stated measurand in the ranges indicated.CMCs are expressed as either a specific value that covers the full range or as a fraction of the readingplus a fixed floor specification. Unless otherwise noted, percentages are defined as percent of reading.
8In the statement of CMC, the value is defined as the percentage of reading, unless otherwise noted.
9This accreditation covers calibrations performed at the main laboratory listed above, and the followingsatellite laboratory located at 2591 S. Oakwood Road, Oshkosh, WI 54904.
Satellite Location:
PRECISION METROLOGY, INC.2951 S. Oakwood RdOshkosh, WI 54904
Carol Shipley Phone: 414 351 7420
Valid To: July 31, 2016 Certificate Number: 1078.01
I. Dimensional
Parameter/Equipment Range CMC2, 4() Comments
Bore Gages3 Up to 12 in 10in/in + 0.6R Gage blocks
Calipers3
Resolution 0.0001 in
Resolution 0.0005 in
Resolution 0.001 in
Up to 2 in(2 to 12) in
Up to 24 in(24 to 60) in
Up to 30 in(30 to 80) in
78 in70 in + 4 in/in
450 in340 in + 3.7 in/in
840 in700 in + 3.4 in/in
Gage blocks
Chamfer Gages/HoleGages3
Up to 12 in (20 + 0.6R) in Cylindrical rings
-
7/25/2019 Precision Metrology 1078-01
52/54
Parameter/Equipment Range CMC2, 4() Comments
Height Gage Up to 36 in 15 in/in + 0.6R Reference stack
Intertest/OditestCalipers
Up to 4 in 54 in + 0.6R Gage blocks
Micrometers3
DepthGroove
Up to 12 inUp to 4 in
20 in/in + 0.6R18 in/in + 0.6R
Gage blocks
Micrometer ODMicrometers
Up to 42 in 12 in/in + 0.6R Gage blocks
Protractor Digital and
Mechanical3
0to 180 0.0076 Angle blocks
Test Indicator Up to 0.5 in 0.0001 in Indicator calibrator
Thickness Gages3 Dialand Digital
Up to 1 in 19 in + 0.6R Gage blocks
Tri-Bore Micrometers Up to 3 in 15 in/in + 0.6R Cylindrical rings
II. Mechanical
Parameter/Equipment Range CMC2, 4, 5() Comments
Torque Wrenches Up to 1000 inlbfUp to 250 ftlbf
0.6 % + 0.6R0.6 % + 0.1 ftlbf
Torque calibrator
-
7/25/2019 Precision Metrology 1078-01
53/54
2Calibration and Measurement Capability Uncertainty (CMC) is the smallest uncertainty of measurementthat a laboratory can achieve within its scope of accreditation when performing more or less routinecalibrations of nearly ideal measurement standards or nearly ideal measuring equipment. CMCsrepresent expanded uncertainties expressed at approximately the 95 % level of confidence, usuallyusing a coverage factor of k = 2. The actual measurement uncertainty of a specific calibrationperformed by the laboratory may be greater than the CMC due to the behavior of the customers deviceand to influences from the circumstances of the specific calibration.
3Field calibration service is available for this calibration and this laboratory meets A2LAR104 GeneralRequirements: Accreditation of Field Testing and Field Calibration Laboratoriesfor these calibrations.Please note the actual measurement uncertainties achievable on a customer's site can normally beexpected to be larger than the CMC found on the A2LA Scope. Allowance must be made for aspectssuch as the environment at the place of calibration and for other possible adverse effects such as thosecaused by transportation of the calibration equipment. The usual allowance for the actual uncertaintyintroduced by the item being calibrated, (e.g. resolution) must also be considered and this, on its own,could result in the actual measurement uncertainty achievable on a customers site being larger than theCMC.
4In the statement of CMC,Ris the resolution of the unit under test.
5In the statement of CMC, the value is defined as the percentage of reading, unless otherwise noted.
-
7/25/2019 Precision Metrology 1078-01
54/54
A2LA has accredited
PRECISION METROLOGY, INC.Milwaukee, WI
for technical competence in the field of
CalibrationThis laboratory is accredited in accordance with the recognized International Standard ISO/IEC 17025:2005 General Requirements for
the Competence of Testing and Calibration Laboratories. This laboratory also meets the requirements of ANSI/NCSLI Z540-1-1994 and any
additional program requirements in the field of calibration. This accreditation demonstrates technical competence for a defined scope and theoperation of a laboratory quality management system(refer to joint ISO-ILAC-IAF Communiqu dated 8 January 2009).
Presented this 23rd
day of September 2014.
_______________________
President & CEO
For the Accreditation Council
Certificate Number 1078.01Valid to July 31, 2016
For the calibrations to which this accreditation applies, please refer to the laboratorys Calibration Scope of Accreditation.