clustering-based, fully automated mixed-bag jigsaw puzzle ... · clustering-based, fully automated...

Clustering-Based, Fully Automated Mixed-BagJigsaw Puzzle Solving

Zayd Hammoudeh Chris Pollett

Department of Computer ScienceSan José State University

San José, CA USA

17th International Conference on Computer Analysis of Images and PatternsAugust 22-24, 2017

20

Clustering-Based,Fully Automated

Mixed-Bag JigsawPuzzle Solving

Hammoudeh & Pollett

1Introduction

Mixed-Bag SolverSegmentation

Stitching

Hierarchical Clustering

Quantifying QualityDirect Accuracy

Experimental ResultsInput Puzzle Count

Solver Comparison

References

Dept. of Computer ScienceSan José State University

IntroductionJigsaw Puzzles

I First jigsaw puzzle introduced in the 1760s

I First computational jigsaw puzzle solver introducedin 1964 [4]

I Solving a jigsaw puzzle is NP-complete [1, 3].

I Example Applications: DNA fragment reassembly,shredded document reconstruction, and speechdescrambling

I Generally, the ground-truth source is unknown.

20



Hammoudeh & Pollett

2Introduction


Stitching




Solver Comparison

References


IntroductionMixed-Bag Puzzles

Jig Swap Puzzles: Variant of the traditional jigsaw puzzle

I All pieces are equal-sized squares.

I Piece rotation, puzzle dimensions, and ground-truth inputcontents are all unknown.

“Mixed-Bag”: Simultaneous solving of multiple jig swap puzzles

I The number of inputs may be unknown.

Randomized Solver Input – 2,017 Pieces

Solver Output #1 Solver Output #2 Solver Output #3805 Pieces 540 Pieces 672 Pieces

20



Hammoudeh & Pollett

3Introduction


Stitching




Solver Comparison

References


Summary of Key Contributions

I Primary Contribution: Novel mixed-bag puzzle solverthat outperforms the current state of the art [6] by:

I Requiring no external “oracle” information

I Generating superior reconstructed outputs

I Supporting more simultaneous inputs

I Additional Contribution: Define the first metrics thatquantify the quality of outputs from a multi-puzzle solver

Our Contribution:

The Mixed-Bag Solver

20



Hammoudeh & Pollett

Introduction

4Mixed-Bag SolverSegmentation

Stitching




Solver Comparison

References


Mixed-Bag SolverOverview

Basis of the Mixed-Bag Solver: Human puzzle solvingstrategy to:

I Correctly assemble small puzzle regions (i.e., segments)I Iteratively merge smaller regions to form larger ones

Simplified Algorithm Flow:

...

Mixed Bag

Final Assembly

Hierarchical Segment

ClusteringSegmentation Stitching

20



Hammoudeh & Pollett

Introduction

Mixed-Bag Solver5Segmentation

Stitching




Solver Comparison

References


SegmentationMixed-Bag Solver Stage #1

I Segment: Partial puzzle assembly where this is a highdegree of confidence pieces are placed correctly

I Each piece is assigned to at most one segment.

I Role of Segmentation: Provide structure to the set ofpuzzle pieces by partitioning them into disjoint segments

20



Hammoudeh & Pollett

Introduction


Stitching




Solver Comparison

References


SegmentationAlgorithm Overview

I Iterative process consisting of one or more rounds.

I In each round, any pieces not already assigned to asegment pieces are assembled into a single puzzle.

I This assembly is then segmented based on inter-piecesimilarity (i.e., the “best buddies” principle).

I Segments of sufficient size are saved for use in laterMixed-Bag Solver stages.

I Segmentation terminates when an assembly has nosegments whose size exceeds a minimum threshold(e.g., 7).

20



Hammoudeh & Pollett

Introduction


Stitching




Solver Comparison

References


SegmentationFirst-Round Example

Ground-TruthInputs Solver Output Segmented Output

20



Hammoudeh & Pollett

Introduction


8Stitching




Solver Comparison

References


StitchingMixed-Bag Solver Stage #2

I Role of Stitching: Quantify the extent that any pair ofsegments is related

I Mini-Assembly: Places a pre-defined, fixed number(e.g., 100) of pieces

I Stitching Piece: A piece near the boundary of a segmentthat is used as the seed of a single mini-assembly

I Segment Overlap: Inter-segment affinity score based onthe composition of a segment’s mini-assembly

20



Hammoudeh & Pollett

Introduction


9Stitching




Solver Comparison

References


StitchingExample – Single Input Image

Ground Truth Segmenter OutputStitching

PiecesMini-Assembly

Stitching piece selected from upper-right corner of the top segment

20



Hammoudeh & Pollett

Introduction


Stitching

10Hierarchical Clustering



Solver Comparison

References


Hierarchical Segment ClusteringMixed-Bag Solver Stage #3

I A single ground-truth image may be comprised of multiplesegments.

I Role of Hierarchical Clustering: Estimate the number ofinputs by grouping together all segments from the sameground-truth image.

I Single-Link Clustering: Inter-cluster similarity equals thesimilarity of their most similar respective members

20



Hammoudeh & Pollett

Introduction


Stitching

11Hierarchical Clustering



Solver Comparison

References


Terminating the SolverBuilding the final outputs

I The solver continues merging segment clusters until oneof two criteria is satisfied:

I Only a single segment cluster remains

I Maximum similarity between any segment clusters is belowa predefined threshold

I Final Assembly: Builds the final solver outputs are builtusing the cluster membership results

Quantifying Solver Performance

20



Hammoudeh & Pollett

Introduction


Stitching


12Quantifying QualityDirect Accuracy


Solver Comparison

References


Quantifying Solver Performance

I Metrics quantify the quality of the solver outputs as thereconstructions may not be reconstructions.

I Two Primary Quality Metrics: Range [0,1]I Direct Accuracy

I Neighbor Accuracy (not discussed in this presentation)

I Disadvantages of Current Metrics: Neither account forissues unique to mixed-bag puzzles including:

I Pieces from one input misplaced in multiple output puzzles

I Pieces from multiple inputs in the same output

20



Hammoudeh & Pollett

Introduction


Stitching


Quantifying Quality13Direct Accuracy


Solver Comparison

References


Direct AccuracyOverview of the Current Standard

Standard Direct Accuracy: Fraction of pieces, c placed in thesame location in both the ground-truth and solved puzzlesversus the total number of pieces, n

Formal Definition:

DA =cn

(1)

20



Hammoudeh & Pollett

Introduction


Stitching


Quantifying Quality14Direct Accuracy


Solver Comparison

References


Direct AccuracyShiftable Enhanced Direct Accuracy Score (SEDAS)

SEDAS: A new quality metric with two primary improvementsover standard direct accuracy:

SEDASPi = maxl∈L

(maxSj∈S

ci,j,l

ni +∑

k 6=i(mk,j)

)(2)

I Mixed-Bag Support: For input, Pi ∈ P, and output, Sj ∈ S,penalize for missing pieces (via ni ) and additional pieces(via

∑k 6=i mk,j )

I Shiftable Reference: Shift the direct accuracy referencecoordinate, l within a set of possible puzzle piece locations, L,(l ∈ L), in order to maximize the overall score

Experimental Results

20



Hammoudeh & Pollett

Introduction


Stitching



15Experimental ResultsInput Puzzle Count

Solver Comparison

References


Overview of the Experiments

I Standard Jig Swap Puzzle Experiment Conditions:Defined by Cho et al. (CVPR 2010) [2] and followedby [7, 6, 9, 5]

I Procedure: Randomly select, without replacement, aspecified number of images (between 2 and 5) from the805 piece, 20 image data set [8]

I Two Primary Experiments:

I Estimation of the Ground-Truth Input CountI Comparison of Overall Reconstruction Quality

I Baseline: Current State of the Art - Paikin & Tal(CVPR ’15) [6]

I Our Competitive Disdvantage: Paikin & Tal’s algorithm hadto be provided the number of input puzzles.

20



Hammoudeh & Pollett

Introduction


Stitching



Experimental Results16Input Puzzle Count

Solver Comparison

References


Estimating the Ground-Truth Input CountMultiple Input Puzzles

0 1 2 30

20

40

60

80 75

16

72

4448

4 4

50 50

0 0

60

20 20

0

Size of Input Puzzle Count Error

Freq

uenc

y(%

)

Mixed-Bag Solver’s Input Puzzle Count Error Frequency

2 Puzzles 3 Puzzles 4 Puzzles 5 Puzzles

Puzzle Count Error: Difference between the actual numberof input puzzles and the Mixed-Bag Solver’s estimate

Overall Accuracy: 65%I Iterations with Error Greater than One: 8%I Performance consistent across input count

Over-Rejection of Cluster Mergers: Neverunderestimated the number of input puzzles

20



Hammoudeh & Pollett

Introduction


Stitching




17Solver Comparison

References


Comparison of Reconstruction QualityPerformance on Multiple Inputs

I Goal: Compare the quality of the outputs from theMixed-Bag Solver (MBS) and Paikin & Tal’s algorithm

I Note: Our Mixed-Bag Solver’s performance when itcorrectly estimated the puzzle count is also shown.

I This is an approximate representation of the performancehad there been optimal hierarchical clustering.

20



Hammoudeh & Pollett

Introduction


Stitching




18Solver Comparison

References


Comparison of Reconstruction QualityShiftable Enhanced Direct Accuracy Score (SEDAS)

2 3 4 50

0.2

0.4

0.6

0.8

1

Number of Input Puzzles

SE

DA

S

Effect of the Number of Input Puzzles on SEDAS

MBS Correct Puzzle Count MBS All Paikin & Tal

20



Hammoudeh & Pollett

Introduction


Stitching




19Solver Comparison

References


Performance on Multiple Input PuzzlesResults Summary

I Summary: Our Mixed-Bag Solver significantlyoutperforms the state of the art, Paikin & Tal.

I This is despite their algorithm having a competitiveadvantage by being supplied the number of input puzzles.

I Puzzle Input Count: Our approach shows no significantperformance decrease with additional input puzzles.

I Effect of Clustering Errors: Performance only decreasedslightly when incorrectly estimating the input puzzle count

I Many of the extra puzzles were relatively insignificant insize.

20



Hammoudeh & Pollett

Introduction


Stitching




Solver Comparison

20References


List of References I

[1] Tom Altman. Solving the jigsaw puzzle problem in linear time. Applied Artificial Intelligence, 3(4):453–462, January 1990.ISSN 0883-9514.

[2] Taeg Sang Cho, Shai Avidan, and William T. Freeman. A probabilistic image jigsaw puzzle solver. In Proceedings of the2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’10, pages 183–190. IEEEComputer Society, 2010.

[3] Erik D. Demaine and Martin L. Demaine. Jigsaw puzzles, edge matching, and polyomino packing: Connections andcomplexity. Graphs and Combinatorics, 23 (Supplement):195–208, June 2007.

[4] H. Freeman and L. Garder. Apictorial jigsaw puzzles: The computer solution of a problem in pattern recognition. IEEETransactions on Electronic Computers, 13:118–127, 1964.

[5] Andrew C. Gallagher. Jigsaw puzzles with pieces of unknown orientation. In Proceedings of the 2012 IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), CVPR ’12, pages 382–389. IEEE Computer Society, 2012.

[6] Genady Paikin and Ayellet Tal. Solving multiple square jigsaw puzzles with missing pieces. In Proceedings of the 2015IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’15. IEEE Computer Society, 2015.

[7] Dolev Pomeranz, Michal Shemesh, and Ohad Ben-Shahar. A fully automated greedy square jigsaw puzzle solver. InProceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’11, pages 9–16.IEEE Computer Society, 2011.

[8] Dolev Pomeranz, Michal Shemesh, and Ohad Ben-Shahar. Computational jigsaw puzzle solving.https://www.cs.bgu.ac.il/~icvl/icvl_projects/automatic-jigsaw-puzzle-solving/, 2011. (Accessed on05/01/2016).

[9] Dror Sholomon, Omid David, and Nathan S. Netanyahu. A genetic algorithm-based solver for very large jigsaw puzzles.In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), CVPR ’13, pages1767–1774. IEEE Computer Society, 2013.

https://www.cs.bgu.ac.il/~icvl/icvl_projects/automatic-jigsaw-puzzle-solving/

clustering-based, fully automated mixed-bag jigsaw puzzle ... · clustering-based, fully automated...

Documents