Multiple Classifier Combination for Character Recognition: Revisiting the Majority Voting System and its

Variations

M.C. Fairhurst

University of Kent

UK

A. F. R. Rahman and H. Alam

BCL Technologies Inc.

USA

Basic Problem Statement

Given a number of experts working on the same problem, is group decision superior to individual decisions?

Ghosts from the Past…

• Jean-Charles de Borda (1781)

• N. C. de Condorcet (1785)• Laplace (1795)• Issac Todhunter (1865) •

•CC. L. Dodgson (Lewis Carrol) (1873) • M. W. Crofton (1885)• E. J. Nanson (1907)• Francis Galton (1907)

Is Democracy the answer?

• Infinite Number of Experts

• Each Expert Should be Competent

How Does It Relate to Character Recognition?

Each Expert has its:

• Strengths and Weaknesses

• Peculiarities

• Fresh Approach to Feature Extraction

• Fresh Approach to Classification

• But NOT 100% Correct!

Practical Resource Constraints

Unfortunately, We Have Limited

• Number of Experts

• Number of Training Samples

• Feature Size

• Classification Time

• Memory Size

Solution

• Clever Algorithms to Exploit Experts– Complimentary Information– Redundancy: Check and Balance– Simultaneous Use of Arbitrary Features and

Classification Routines

Question?– Recent trend is towards complicated decision combination

schemes– Exhaustive Classifier Selection– Theoretical analysis in place of empirical methods

How sophisticated (read “complex”)

algorithms do we really need?

Majority Voting Philosophy

• Should the decision agreed by the majority of the experts be accepted without giving due credit to the competence of the experts?

---- OR ----• Should the decision delivered by the most

competent expert be accepted without giving any importance to the majority consensus?

[1] Simple Majority Voting

Classifier Classifier Classifier

ClassificationDecision

ClassificationDecision

ClassificationDecision

Decision Fusion : Counting Individual Votes to Support aDecision

Final Decision

21

12n

n

k

Decision accepted if at least k of the experts agree, where

If n is even,

If n is odd.

[2] Weighted Majority Voting

OverallWeight of

theClassifier

Classifier

OverallWeight of

theClassifier

Classifier

OverallWeight of

theClassifier

Classifier

ClassificationDecision

ClassificationDecision

ClassificationDecision

Decision Fusion: Counting Weighted Votes for Individual Decisions to Support a FinalDecision

Final Decision

[2] Weighted Majority Voting (Contd.)

So if decision to assign the unknown pattern to the class is denoted by with , being the number of classes, then the final combined decision supporting assignment to the class takes the form of:

The final decision is therefore:

thk thiikd mi 1 m

cmid

thi

nk

ikkcomi dd

,...,2,1

*

comd

comimi

com dd ,..,2,1max

[3] Class-wise Weighted Majority Voting

Class-basedWeight of

theClassifier

Classifier

Class-basedWeight of

theClassifier

Classifier

Class-basedWeight of

theClassifier

Classifier

ClassificationDecision

ClassificationDecision

ClassificationDecision

Decision Fusion: Counting Class-based Weighted Votes for Individual Class Decisions toSupport a Final Decision

Final Decision

[4] Restricted Majority Voting (Top Choice)

OverallClassifierWeight

ClassifierOverall

ClassifierWeight

ClassifierOverall

ClassifierWeight

Classifier

Best Classifier Selection

Decision Fusion: Selection of Best Decision Delivered by BestClassifier

Final Decision

[4] Restricted Majority Voting

(Generalized)

[5] Class-wise Best Decision Selection

Class-basedClassifierWeight

ClassifierClass-based

ClassifierWeight

ClassifierClass-based

ClassifierWeight

Classifier

Best Class-basedClassifier Selection

Decision Fusion: Selection of Best Class-based DecisionDelivered by Best Class-wise Classifier

Final Decision

[6] Enhanced Majority Voting

[7] Ranked Majority Voting

• Not only the top choice, but ranked list of other classes• Takes account of the negative votes cast by the experts against a particular

decision. • Each expert not only supplies the top choice (class) decision, but also supplies the

ranking of all the other choices considered.• The idea is to translate this ranking into ``scores" which would be comparable

across all the decisions by all the experts.

[7] Ranked Majority Voting: Continued

(Class Set Reordering)

• Highest Rank: Take the highest assigned rank

• Borda Count: Sum of the number of classes ranked below it by each classifier.

• Regression Method

Selection of a Database

• NIST Handwritten Characters

• Collected Off-line

• Total 34 Classes (0-9, A-Z, no Distinction between 0/O and I/1)

• Total Samples of Over 34,000 characters

• Size Normalized to 32X32

Performance of the Classifiers

Expert Accepted Recog. Error Rej.

FWS 97.35 78.76 18.59 2.65

MPC 97.62 85.78 11.84 2.38

BWS 95.50 72.31 23.19 4.50

MLP 95.13 82.31 12.82 4.87

Performance of the CombinationCombination Method Accepted Recog. Error Rej.

Simple 96.59 90.59 6.00 3.41

Weighted 96.85 90.64 6.21 3.15

Class-wise Weighted 96.86 90.70 6.16 3.14

Restricted Top Choice 95.68 88.97 6.71 4.32

Class-wise Best Decision 96.76 89.64 6.79 3.24

Restricted Generalized 96.54 90.63 5.91 3.46

Enhanced (ENOCORE) 97.14 90.91 6.23 2.86

Ranked (Borda) 96.99 90.77 6.22 3.01

Committee 95.98 89.63 6.35 4.02

Regression 97.68 90.68 6.83 2.32

Comparative Study

Method Accepted Recogn. Error Reject

BKSM 96.20 90.84 5.36 3.80

Sum Rule

96.40 90.21 6.19 3.60

GA 96.36 92.39 3.97 3.64

Best of MVS

97.14 90.91 6.23 2.86

Conclusions• Majority Voting Solutions can be very versatile

and adaptive• Different variations may be adopted for different

problem domains• The Majority Voting configuration is generic• Majority Voting Systems may be as applicable to

any task domains with equal effectiveness as other complicated solutions