leantest key: test coverage analysis powered by traceability

If there is no solution, there is no problem

LeanTest key: Test coverage analysis powered by traceability

Christophe [email protected]

ASTER Technologies

1

IEEE 11th InternationalBoard Test Workshop

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation

• Test repeatability

• Real contribution - AOI

Conclusion


Targets: Key objectives

Our targets are to provide tools that:Create an effective environment to continually improve the delivered quality of manufacturing processes.Assist in reducing costs of assembly, test, rework, scrap and warranty.Help improve line utilization and reduce cycle time.Allow manufacturers to better prioritize the deployment of constrained resources.Allow manufacturers to benchmark their DPMO rates to others in the industry.

… regardless of board complexity.2

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Test coverage and traceability

Good products must be defect-free and cheap. How to detect or prevent all faults on the product

so that only good products are shipped?

Test coverage is a key metric as it will be the quality warranty and the main driving factor for LeanTest.

This paper describes how traceability tools should be used in order to improve test coverage understanding.

3

Defect Prevention

Defect Detection

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


4

• Insufficient

• Excess

• Cold Solder

• Marginal Joints

• Voids

• Polarity (PCAP)

• Missing

• Gross Shorts

• Lifted Leads

• Bent Leads

• Extra Part

• Bridging

• Tombstone

• Misaligned

• Polarity

• Shorts

• Open

• Inverted

• Wrong Part

• Dead Part

• Bad Part

• In-System Programming

• Functionally Bad

• Short/Open on PCB

AOI

In-Circuit

X-RaySolder

Material

• At-speed memory tests

• At-speed interconnect

• Fault Insertion

• Gate level diagnosis

JTAG

(unpowered)

(unpowered)Placement

Defect Universe

Identify the faults that can occur.

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Defect Universe

Typical manufacturing defects:

We need to group defects into categories, to understand what defects can be captured by a particular test strategy.

5

Missingcomponents

Misalignment Wrong value

Incorrect Polarity

Open Circuits

Short circuitsInsufficient

solder

Broken components

Tombstone

Material(Supply chain) Placement Solder

Excessive solder

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


The ability to detect defects can be expressed with a number: coverage.

Each defect category fits with its test coverage:

MPSF [1] PPVSF PCOLA/SOQ /FAM

Material Value Correct

Live

Placement Presence Presence

Alignment

Polarity Orientation

Solder Solder Short

Open

Quality

Function Function Feature

At-Speed

Measure

Test coverage

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Test coverage by defect category

For each category (Material, Placement, Solder) of defects (D), we associate the corresponding coverage (C).

The test efficiency is based on a coverage balanced by the defects opportunities.

7

DM + DP + DS + DF

DM CM + DP CP + DS CS + DF CFEffectiveness =

We need a better coverage where there are

more defect opportunities!DPMO

Coverage

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Test coverage by defect category

Each test technique brings a certain ability to detect the defects defined within ‘defect universe’.

No single solution is capable of detecting all the defects.

Good coverage = combination of tests.

8

M

S FM

P S

FM

P

S

M P S F

F

P

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Case 1: Faulty boards at system level

Electronic plants, in charge of board integration, often discover a significant amount of defective boards at system test.

How is it possible to get failures at system level if we only buy good boards? The defect appears during packing and transportation

(vibration, extreme temperatures, moisture). The defect is a dynamic problem which is revealed by

the integration of the board in the complete system. The reality is usually more simple…

9

AOI ICT BST FT

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



If the board is failing at system test, it is usually because the escape rate (or split) is higher than expected.

There are only two possibilities: The combined coverage is lower than optimal. The DPMO figures are higher than expected.

10

Test

FPY

FOR

Pass

Fail

Good

Bad

Good

Bad

Slip

False reject

Productsshipped

Productsrepaired

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



The auditing conclusions were: Wrong or inadequate coverage metrics are produced:

Example: confusion between accessibility and testability; coverage by component only - without incorporating solder joint figures ; Over optimistic report (marketing driven report),

Wrong DPMO figures due to limited traceability or incorrect root cause analysis (Example: confusion between fault message and root cause/defect).

11

Coverage estimation

Coverage measurement

Selected strategies

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Case 2: DPMO estimation

Going beyond solving surface issues.

12

Solder

Placement

Material

Insufficient solder

wrong value

Misalignment

Tombstone

Polarity

Open

Short

Missing components

Test coverage

Broken leads

DPMO

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



The weighted coverage (with DPMO) is a key factor to estimate the production model.

We need an accurate value for DPMO if we want realistic production models.

13

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



Production model in a test line

14

AOI

ICT

FT

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



Basic analysis uses average numbers coming from iNemi or the PPM-Monitoring.com web site. It does not reflect the reality, but it is much better than considering all defects as equally probable!

The best approach is to use the traceability database in order to extract a table including parameters such as partnumber, shape, mounting technology, pitch, number of pins, function/class, DPMO per category (MPSF).

15

Assembly machines

Test, inspection & other machines

RepairDatabase

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



Define data collection methods around existing IPC standards. IPC 9261 In-Process DPMO and Estimated Yield. IPC 7912 Calculation of DPMO and

Manufacturing Indices for PCBAs.

Define data stratification and classification methods.

Combine the data into a single database: DPMO for Material (Part number). DPMO for Placement (Package type). DPMO for Soldering (Reflow & Wave).

It requires good cooperation between test and quality services.

16

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



Range and standard deviation for any DPMO statistic.

Compare actual yield to estimated yield: By test step. Full test line.

Correlation of test coverage/strategy to DPMO rates.

17

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion



During test coverage analysis, TestWay uses various algorithms to estimate the DPMO. Same Part Number, Same shape for placement DPMO, Same pitch and number of pins.

With an accurate DPMO representation, it is possible to: Estimate the yield and the escape rate. Two key

factors used to select the best Contract Manufacturer or EMS - DPMO figures per EMS site.

Identify the real overlap for test/inspection optimization. DfT becomes one of the principal contributors in cost reduction.

18

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Case 3: Test Repeatability

During first production run, we selected a set of boards where a SPC analysis has been conducted in GR&R context.

Gage R&R (Gage Repeatability and Reproducibility) is the amount of measurement variation introduced by a measurement system,which consists of the measuring instrument itself and the individuals using the instrument. A Gage R&R study is a critical step in manufacturing Six Sigma projects, and it quantifies : Repeatability – variation from the measurement instrument. Reproducibility – variation from the individuals using the instrument.

19

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Case 3: Test Repeatability

Quality and traceability analysis helps to compute the classic Cp, CpK … and CmC.

CmC means “Calibration and Measurement Capability”. CmC = Tolerance / k (k = 6 for critical components).

A test which is not repeatable cannot claim to qualify a component. So CmC is used to weight the Correctness coverage.

In addition, the Failure Mode and Effects Analysis (FMEA) gives the criticity per component which limit the oversized test tolerance.

20

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Case 3: Test Repeatality

Passive measurements Correct value: Value is tested at 100%. Minor deviation: Value is tested at 95%. Medium deviation : Value is tested at 50%. Major deviation: Value is not tested. Incorrect value: Component is not tested.

21

95%50%

0%

For more accuracy:Compare CAD value

and tolerances against minimum and

maximum tested values

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Case 4: Real contribution of AOI/AXI

AOI and AXI are inspection techniques which are checking for deviations.

When deviation is big enough, it should become a defect.

When a test line includes an inspection machine and an electrical tester (ICT, BST), it is difficult to agree on test contribution.

22

AXI BST FT

95% 47% 53%

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Case 4: Real contribution of AOI/AXI

With a traceability system that collects defects/repair information in real time, we are able to record that a fault has been detected and how it has been diagnosed (ie: Root cause analysis).

We compare the defects that have been detected with ICT and FT against the defects detected by AOI in order to adjust real coverage.

23Diagnosis/

Repair

Database

FT

ICT

AOI

Test Coverage Analysis

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


Conclusion

Continual reassessment of capability metrics. Improved accuracy of quality estimations. Enhanced defect detection rate by increasing

the understanding of test coverage. Reduced escape rate (bad boards to the

customer).

24Zero-defect road…

Definitions

• Key Objectives

• Defect Universe

• Test Coverage

• Test efficiency

Case Studies

• Faulty boards

• DPMO estimation



Conclusion


25

leantest key: test coverage analysis powered by traceability

Documents