The 2012 Biomedical Engineering International Conference (BMEiCON-20I2)

A New Content-Based Medical Image Retrieval System Based on Wavelet Transform and

Multidimensional Wald-Wolfowitz Runs Test

Mr. Phatsarun Nakaram School of Electronics Engineering, Faculty of Engineering

King Mongkut's Institute of Technology Ladkrabang Ladkrabang, Bangkok, Thailand 10520

phatsarun@gmail.com

Abstract-Recently, one of the authors proposed a new

similarity measure, called weighted multidimensional Wald and

Wolfowitz (MWW) runs test, for the content-based color image

retrieval system. The algorithm outperforms conventional

similarity measures for comparing two color images. In this

paper, we propose a new content-based medical image retrieval

system based on discrete wavelet transform (DWT) symlet and

the weighted MWW runs test. The DWT is used to extracted

texture features of the medical images. The weighted MWW runs

test is used to compare distributions of texture features of two

medical images. Our experiments were performed on 1,000 medical images from image retrieval in medical applications

(IRMA). The experimental results show promisingly efficient to

retrieve the medical images.

Index Terms - wavelet transform, k-mean clustering

algorithm, Wald and Wolfowitz runs test

LINTRODUCTION

Nowadays in the medical domain, a number of medical images such as X-ray, MRI and ultrasound images have been creating everyday [1]. These images have been found useful in many medical areas such as identification of the presence of disease, evaluation of the extent of the disease, characteri­zation of the patterns of the disease, narrowing the differential diagnosis, as a guide to the site of biopsy, and assessing the clinical course of the disease and response to therapy [2]. Then an efficient and flexible tool for automatically searching images in the entire medical image database has been becoming real demand. The picture archiving and communications system (P ACS) is a general and widely used tool to store, retrieve, and transmit the images in the DICOM (Digital Imaging and Communication in Medicine) [3] format. However, the PACS provides only simple textual search using patient's name and identification, date, image modality, and physician's name.

A content-based image retrieval (CBIR) system has been developed to retrieve images which are relevant to a query image, using information derived from the images themselves. There are several CBIR systems that have been developed and provide satisfactory retrieval performance such as WebSeek [4], QBIC [5], MIT's photobook [6], and etc. The CBIR system consists of two main modules: the feature extraction module, in which visual features are extracted from each

978-1-4673-4892-8/12/$31.00 ©20 12 IEEE

Dr. Thurdsak Leauhatong School of Electronics Engineering, Faculty of Engineering

King Mongkut's Institute of Technology Ladkrabang Ladkrabang, Bangkok, Thailand 10520

thurdsak@hotmail.com

image stored in the database, and the similarity measurement module, in which a distance or similarity between the query image and each image in the database is computed, by making use of the extracted features.

The visual features commonly used in the CBIR system are colors [7, 8], textures [9, 10], edges [11, 12], and shapes [13,14]. In the medical X-ray images, the texture features play an important role for obtaining information of identification for disease diagnosis [15] and information of various pathologies [16]. The textures can be regarded as a function of intensity variations with repeated structures or patterns. Recently, many works on texture analysis concentrate on two-dimensional wavelet transform which was inspired by the multichannel filtering mechanism in neurophysiology. Selection of wavelet basis function is important issue for analyzing the texture information. The selected wavelet basis function should have the desired function properties, including support in time and frequency domain, symmetry, and shift invariance. The support of a wavelet quantifies its localization in spatial and frequency domain. The symmetric linear-phase filter is also important for avoiding dephasing. A non-symmetric filter will result in shift variance of the outputs and this should be avoided in texture analysis. It is well-known that the Symlets a compactly supported wavelets with the least asymmetry and the highest number of vanishing moment for a given support width. As a result, it is a good reason to select the Symlets for analyzing the textures.

In order to compare two images, a feature vector of texture should be constructed to represent the content of each image. In the CBIR system, the most popular feature vector is a multidimensional histogram. Statistically, the histogram of the texture describes the overall texture content of the image. There are a number of similarity measures to compare two

histograms such as histogram intersection [17], X2 test[ 18],

Kullback-Leibler divergence[19], and Jeffrey divergence[20]. The main problem of using the histogram is that the multidimensional space of the texture feature is quantized into a fixed number of bins, usually of a predefined size. Often only a small fraction of the bins in the histogram contain significant information. To overcome this problem, many similarity measures such as earth mover's distance [21] and multivariate Wald-Wolfowitz (MWW) runs test [22] were proposed

Recently, an extended of MWW runs test, called weighted MWW runs test [23], was proposed. The weighted MWW runs test outperforms all of the conventional similarity measures with respect to the precision and the computational time.

In this paper, we proposed a new content-based medical image retrieval system based on the wavelet transform and the weighted MWW runs. The organization of the paper is following. In section II, the fundamental theories of the wavelet transform and the weighted MWW runs test are described. Section III presents the proposed algorithm. The experiments are given in section IV. Finally, conclusions and future work are discussed in section V.

II. FUNDAMENTAL THEORY

A. The Discrete Wavelet Transform (DWT) Theory

. .

HU

HHI

Fig. 1: An example of the 2D DWT of an image.

The 2-D DWT represents an image in terms of a set of

shifted and dilated wavelet functions {I,LIHf" , I,LIw ,I,LIHH } and

scaling functions ¢LL that form an orthonormal basis for

L2 (912 ) . Given a J -scale DWT, an image I (x, Y) of

N x N is decomposed as:

NJ-l I(x,y) = I UJ,k/jJ��,i (x,y) (I) k,i=O

where ¢;,�,i (x,y) == TJ/2 ¢( TJ X - k, TJ Y - i) , l,LI�k,i (x, y) == Tf/2I,L1B ( Ti X - k, Ti Y - i) , and BE B = {HL,LH,HH}, and Nf = N / 2J. In this paper

HL, LH, HH are called wavelet or DWT subbands,

Ui,k,i = f f I ( x, y ) ¢�t,idxdy denotes a scaling coefficient

in scale j, and W7,k,i = f f I ( x, y) l,L17,k,idxdy denotes the

(x, y t wavelet coefficient in scale j and subband B. An

example of the 2D DWT of an image is shown in Fig. l.

B. The Graph Theory and the Minimal Spanning Tree

Let V = {Vi I Vi E 91d}�=1' d;:::: 2, be a set of finite

points in d -dimension vector space 9td• A graph on V is a

set of pair G = (V, E) such that E c {( Vi' vJ

I( Vi' VI ) E V} :J=l

' Elements of V and E are called

vertexes and edges of G respectively. Two vertexes Vi and

VI of G are adjacent or neighbors, if (Vi' Vj) is an edge of

G. The degree Di of a vertex Vi is the number of edges

incident to the vertex. A sub graph of G = (V, E) is a graph

H = (V',E') such that V' c Vand E' c E. A walk of

length k in a graph G is a non-empty alternating sequence

voeo vlel",vkek_l of vertexes and edges in G such that

ei = (Vi' Vi+J for all 1< k. A path in G is a walk whose all

vertexes are different. A path voeo vlel",vkek_l is a cycle, if

Vo = vk• A non-empty graph G is called connected, if any

two of its vertexes are linked by a path in G. A tree is a

connected graph on V without cycle. A spanning tree on V is

a tree which contains all vertexes of V. Finally, a minimal

spanning tree [24] (MST) r on V is a spanning tree such that

The sum of the length of edges of r is minimal; that is:

(2)

where r' is any spanning tree on V, and lei = IIVi -Vi II is the

length of an edge e = (Vi' Vi )

C. The multivariate generalization of the Wald and Wolfowitz runs Test

Suppose xPx2, ... ,xm are independent and identically

distribution (Ud.) random points in 9td with common

distribution function j, and YPY2, ... ,Ym are i.i.d. random

points in 9td with common distribution function g. Friedman

and Rafsky [25] defined a multidimensional Wald-Wolfowitz

(MWW) runs test [26] to assess the null hypothesis

Ho : f = g as follows.

Let r be the MST on the union of two sets X = {Xi tl and Y = {Yi tl' V = Xu Y. We call an edge which joins

vertexes from different sets as an inter set (IS) edge. Friedman

and Rafsky defmed the MWW runs test, R, as the number of

disjoint trees which result from removing all IS edges. Friedman and Rafsky made a conjecture that the null

hypothesis Ho becomes stronger as R becomes higher.

Since r is a cyclic, if any edge of r is removed, then r is divided into two disjoint trees (see [25] Theorem 2). As a

result, R equals the number of IS edges of T plus I. Since

the total number of vertexes of V is m + n, the number of

edges of the MST r is m + n -1. Number these m + n-1 edges of r arbitrarily. Then R is defined as:

m+n-1

R= L>i+1 (3) i-I

where Zi = 1, if the ith edge is the IS edge, and Zi = 0 otherwise.

An example of the MWW runs test is shown in Fig. 2. Fig.

2(a) shows the MST on V = Xu Y where + symbols and •

symbols indicate the vertexes of X and Y respectively. An

IS edge is an edge which links a + symbol to a . symbol

and there are 8 IS edges in the MST on V. Fig. 2(b) shows

9 resultant disjoint trees. Then R = 9.

(a) The MST on V =XuY (b) the resultant disjoint trees.

Fig. 2: An example of the MWW runs test.

Let N = m + n, and Ci be the number of edge pairs

sharing the ith vertex of the MST, and Di the degree of the

ith vertex. Then Ci =.!.. Di (Di -1). Therefore, the total 2

number of edge pairs can be computed as follows:

(4)

The mean and the variance of R can be computed as follows [8]:

E[R] = 2�n +1 (5)

Var[RIC]= 2mn { 2mn-N N(N -1) N (6)

+(

C-X+2 )[N(N-1)-4mn+2]}

N -2 N-3

It has been shown that the quantity:

asymptotically approaches the standard normal distribution [25].

(7)

W can be used as similarity measure in a way that the

bigger W is, the more similar the two distributions are.

III. THE PROPOSED SIMILARITY MEASURE

The proposed similarity measure consists of three steps: 1) the wavelet texture extraction, 2) the wavelet texture quantization, and 3) the weighted MWW runs test. The detail of each step can be described as follows.

D. The Wavelet Texture Extraction HL LH HH W;,k,t' W;,k,t' and W;,k,i can be considered as the

magnitude of local intensity variations in horizontal, vertical,

and diagonal directions in scale j respectively. U;,k,i can be

considered as the local average of intensity in scale j. Then

some of their combinations can be most efficient in representing texture information. The wavelet texture used in this paper can be described as follows

M J�l l-------f

Fig. 3: Construction of the texture image.

Let o.� be an image of wavelet coefficient, where

0.7 (k,i) = W;'k,i' and let MJ be an image of scaling

coefficient, where M/(k,i)=u/,k,;' An image of texture

information in direction of B subband, <DB, can be

constructed as shown in Fig. 3. Firstly, compute o.��p o.��2' o.7�3' and MJ�I using Symlet wavelet. Next, resize o.7�2 and o.��3 to the same dimension of o.��l' Then construct

<DB by summing o.7�1' the resized o.7�2 and o.7�3' and

MJ=I'

E. The Wavelet Texture Quantization

Let wand h be the width and height of a given image

respectively, and let <D = {( <DHL (x, y), <DLH (x, y),

<D"" (x,y)1 X � [0",., ; -1].y � [0, ... , � -I]} donotc

a dataset of wavelet texture features of the image.

Let M be the number of clusters, and let Cp 0 0 . , C M be

the centroids of M clusters which are obtained by applying

the k-means clustering algorithm to <D where the distance

metric is the Euclidean distance. Next, let S; be a Voronoi cell

of C;, which consists of all wavelet texture features which are

closer to C; than t other centroids; that is:

S; = {¢ E <D111¢-C;II:s; II¢-C,II, j E [1, ... ,M]}

(8)

where Ilql denotes the Euclidean distance. Let Wei be the

number of the wavelet texture features which belong to the

V oronoi cell with the centroid C;. Let We; be called the

weight of C;. F. The Weighted MWW Runs Test

Let <Dl and <D 2 be datasets of wavelet texture features of

the image II and 12 respectively. From the theoretical point

of view, an MWW runs test of the MST on <Dl u<D2 can be

an effective similarity measure. However in practical

applications, creating the MST on <DI U <D2 for counting the

number of IS edges is an impossible task because the number of vertexes is enormous. Then a method to approximate the

number of IS edges is needed. A new approximation method, called the weighted MWW runs test, was proposed by Leauhatong et.al [27]. The weighted MWW runs test can be described as follows.

Let F={(F;,Wr, )}:1 and G={(G;,WC;, )}:I be two

datasets of centroids and weights quantized from <DI and <D2 respectively. Let Sf; and SG, be the Voronoi cells defmed by

Eq. (8) of F; and G; respectively.

Fig. 4: An Example of G1 and its neighbor with their Voronoi cells.

Consider a centroid G1 0 the MST on F U G and its

neighbors G3, F;, and F; with their V oronoi cells SGI' SC3' Sf;' and Sf; as shown in Fig. 4.

The number of IS edges in the region SCI U SC3 U Sf3

uSr:. can be approximated by Eq. (5) as follows. ,

According to Eq. (5), it can be easily proven that the mean

of the number of IS edges in the region Sq U SCi3 U SF, 2mn uS r:. is --- , where m and n are the number of wavelet

, m+n texture features in the region Sf3 U Sf; and SCI U SC3 respectively. In this case, and

n = WCI + WC3' Then the approximated number of IS edges,

R ( G1), in the region SCI U SC3 U Sf3 u Sf; around G1 is

defmed as:

(9)

The method defmed above can be used to approximate the number of IS edges around the other centroids

F; ,oo.,FM, G1 , 0 0 ' , GM• The summation of the all approximated IS edges:

(10) i=l i=l

is called the weighted MWW runs test, and can be expected to

be a effective similarity measure between <Dl and <D2. In the case that the weighted MWW runs test is used for

similarity measure, the rank between the query image and the retrieval image is defined as follows:

Rank ( <D Q' <D fi ) < Rank ( <D Q ' <D fJ ) if R ( <D Q ' <D ii ) � R ( <D Q' <D ii ) .

IV. EXPERIMENTS

In this experiment, the medical images from the Image Retrieval for Medical Application (IRMA) of Aachen University of technology are used. The database consists of 20 categories, and each category consists of 50 images with various sizes. The categories in the experiments are chosen comprehensively X-ray images which used for diagnosis in medical field. The example images are shown in figure 5, and the categories are shown in table 1. The block diagram of experiment process has shown in figure 6.

TABLE 1 THE GROUP IN DATABASE

group Irma code group Irma code

1 1121-4aO-414-700 11 1121-120-943-700

2 1121-4aO-914-700 12 1121-120-961-700

3 1121-115-700-400 13 1121-210-330-700

4 1121-120-200-700 14 1121-220-230-700

5 1121-120-310-700 15 1121-220-310-700

6 1121-120-330-700 16 1121-320-941-700

7 1121-120-421-700 17 1121-430-213-700

8 1121-120-516-700 18 1123-110-500-000

9 1121-120-800-700 19 1123-211-500-000

10 1121-120-922-700 20 1124-310-620-625

The experiment details are following as: • The images in database must be random to use for

query image. Each category is random 3 images,

totals 60 images.

• Database image and query image are extracted

wavelet texture features by symlet wavelet transform

and then they are rebuilt the new wavelet texture

feature from those wavelet texture features. The

details of computation have shown in section 3D.

• We will be combined the new three wavelet texture

features (HH subband HL subband and LH subband)

same R G B pattern.

• Then the wavelet texture features are computed to

reduce quantity information by K-mean clustering

algorithm. The details of computation have shown in

section 3E.

• We will be weighted centroids and be weighted

values, obtained from cluster of k-mean clustering

computation in database image and query image.

Then they are calculated with MWW runs test to

compare similarity. The details of computation have

shown in section 3F.

• The performance of retrieval image in the experiment

are calculated with precision value as :

relevant images Precision =

retrieved images (11)

The figure 7 has shown graph relation of precision and number K-mean. Since the computation time of K - mean clustering and the MWW runs test will be direct varied with number cluster. Therefore the figure 7 has shown relation of precision and computation time. The good system of content­based image retrieval must be given result at highly precision in retrieval image and used less computation times. In figure 7, we can be considered that the appropriate number clustering equals 40 clusters since although the cluster number will increase but the precision increase trivial. The best wavelet texture feature to compute is symlet4 after that the performance of retrieval image precision equals 5.36667.

Fig. 6: Block diagram of proposed algorithm.

� Cluster Cluster Cluster Cluster Cluster Wavele 20 40 60 80 100

sym2 0.294 0.546 0.424667 0.441 0.534528

sym3 0.249667 0.483 0.403667 0.506333 0.482961

sym4 0.301 0.506333 0.441 0.483 0.536667

sym5 0.233333 0.49 0.485333 0.445667 0.5250004

sym6 0.245 0.455 0.485333 0.455 0.5110004

sym7 0.247333 0.452667 0.476 0.473667 0.473667

sym8 0.205333 0.422333 0.422333 0.443333 0.4900003

sym9 0.278833 0.427 0.445667 0.457333 0.459667

6 - sym2

S - sym3

4

Precision 3 - sym4

2 - symS

1 - sym6

0 - sym7

20 40 60 80 100 - sym8

Number Cluster - sym9

Fig. 7: Relation between precision and number cluster

V. CONCLUSION

In this paper, we have presented a new content-based medical image retrieval system based on wavelet transform and MWW runs test. The advantage of the proposed method is to use with many resolutions and many kinds of medical images. Figure 7 demonstrate that the best accuracy of retrieval was symlet 4 which had about 0.53. Several kinds of image experiment have generate highly precision. In the future work, we will solve some kind of medical image which have low precision. We will not consider with black local in background of medical image. It should be increase precision.

ACKNOWLEDGMENT The authors would like to thank Prof. Dr. Thomas

M.Deserno Lehmann for medical dataset. The main image dataset used in this study is courtesy of the IRMA Group, Aachen, Germany, http://irma-project.org.

REFERENCES

[1] Wei Xiong, Bo Qiu, Qi Tian, Changsheng Xu, Sim Heng Ong, Kelvin Foong (2005), "Content Based Medical Image Retrieval

using Dynamically optimised Regional Features,. ICIP (3) 2005: 1232-1235

[2] Dash, 1.K.; Gupta, R.D.; Mukhopadhyay, S.; Khandelwal, N.; Bhattacharya, P.; Garg, M.; , "Content-based image retrieval for interstitial lung diseases," Signal Processing, Computing and

Control (ISPCC), 2012 IEEE International Conference on , vol., no., pp.I-4, 15-17 March 2012

[3] P. Mildenberger, M. Eichelberg, and E. Martin, "Introduction to the DlCOM standard." European radiology, vol. 12, no. 4, p. 920, 2002.

[4] Smith, J.R.; Shih-Fu Chang; , "Visually searching the Web for content," MultiMedia, IEEE , vol.4, no.3, pp.12-20, Jul-Sep 1997.

[5] Flickner, M.; Sawhney, H.; Niblack, W.; Ashley, 1.; Qian Huang; Dom, B.; Gorkani, M.; Hafner, 1.; Lee, D.; Petkovic, D.; Steele,

D.; Yanker, P.;, "Query by image and video content: the QBIC system," Computer , vol.28, no.9, pp.23-32, Sep 1995.

[6] A Pentland, R. W. Picard, and S. Sclaroff, "Photobook: Content­Based Manipulation of Image Databases," In SPIE Storage and Retrieval Image and Video Databases, no. 2185, pp. 34-47, 1994.

[7] Kok, C.W.; Hui, Y.; Nguyen, T.Q.; , "Medical image pseudo coloring by wavelet fusion," Engineering in Medicine and Biology Society, 1996. Bridging Disciplines for Biomedicine.

Proceedings of the 18th Annual International Conference of the IEEE , vol.2, no., pp.648-649 vol.2, 31 Oct-3 Nov 1996.

[8] Tai Xiao-ying; Wang Li-dong; , "Medical Image Retrieval Based on Color-Texture Algorithm and GTI Model," Bioinformatics and Biomedical Engineering, 2008. ICBBE 2008. The 2nd International Conference on , vol., no., pp.2574-2578, 16-18 May 2008.

[9] Gang Zhang; Zong-Min Ma; , "Texture feature extraction and

description using gabor wavelet in content-based medical image retrieval," Wavelet Analysis and Pattern Recognition, 2007.

ICWAPR '07. International Conference on , vol. I , no., pp.169-173.2-4 Nov. 2007.

[10] Mavromatis, S.; Boi, 1.M.; Sequeira, 1.; , "Medical image segmentation using texture directional features," Engineering in Medicine and Biology Society, 2001. Proceedings of the 23rd

Annual International Conference of the IEEE , vol.3, no., pp. 2673- 2676 vol.3. 2001.

[II] Wang, A; Zhaowei Gu; Wang Bo; Gongjian Shen; , "HMT Model and B-Spine Wavelet Based Intelligent Medical Image Edge Extraction Algorithm," Intelligent Networks and Intelligent Systems, 2008. ICINIS '08. First International Conference on , vol., no., pp.552-555, 1-3 Nov. 2008.

[12] Wang, A; Xinhua Zhang; Yu Chen; lie Wu; , "B Spline

Wavelet and SVM Threshold Based Medical Image Edge Extraction," Industrial Electronics, 2007. ISIE 2007. IEEE International Symposium on , vol., no., pp.1628-1632, 4-7 June 2007.

[13] Fu Li-dong; Zhang Yi-fei; , "Medical Image Retrieval and Classification Based on Morphological Shape Feature,"

Intelligent Networks and Intelligent Systems (ICINIS), 2010

3rd International Conference on , vol., no., pp.116-119, 1-3 Nov. 2010.

[14] Gang Zhang; Ma, Z.M.; Qiang Tong; Ying He; Tienan Zhao; , "Shape Feature Extraction Using Fourier Descriptors with Brightness in Content-Based Medical Image Retrieval,"

Intelligent Informati on Hiding and Multimedia Signal Processing, 2008. IIHMSP '08 International Conference on , vol., no., pp.71-74, 15-17 Aug. 2008.

[IS] J. Sklansky, "Medical radiographic image processing and pattern recog nition," in Proc. Sth Int. Conf. Pattern Recognit., 1980, pp. 374-382.

[16] R. W. Conners, C. A. Harlow, and S. 1. Dwyer, "Radiographic image analysis: Past and present," in Proc. 6th Int. Conf. Pattern Recognit., 1982, pp. 11S2-l169.

[17] Banu, M.S.; Nallapemmal, K.; , "Analysis of color feature

extraction techniques for pathology image retrieval system," Computational Intelligence and Computing Research (ICCIC), 2010 IEEE International Conference on , vol., no., pp.1-7, 28-29 Dec. 2010.

[18] Liu Yunning; Wang Huan; Guo Xiuhua; Liang Zhigang; He Qian; , "A Comparison between Inside and Outside Texture Features Extracting from Pulmonary Nodules of CT Images," Natural Computation, 2009. ICNC '09. Fifth International Conference on , vol.6, no., pp.183-186, 14-16 Aug. 2009.

[19] Weisenfeld, N.L.; Wartleld, S.K.;, "Normalization of joint image-intensity statistics in MRI using the Kullback-Leibler divergence," Biomedical Imaging: Nano to Macro, 2004. IEEE

International Symposium on , vol., no., pp. 101- 104 Vol. 1, IS-18 April 2004.

[20] Bertalya; Prihandoko; Kerami, D.; Kusuma, T.M.; , "Classification of X-Ray Images Using Grid Approach," Signal Image Technology and Internet Based Systems, 2008. SITTS '08. IEEE International Conference on , vol., no., pp.314-319, Nov. 30 2008-Dec. 3 2008.

[21] Dempere-Marco, L.; Xiao-Peng Hu; Ellis, S.M.; Hansell, D.M.;

Guang-Zhong Yang;, "Analysis of visual search patterns with EMD metric in normalized anatomical space," Medical Imaging, IEEE Transactions on , vol.2S, no.8, pp.1 011-1 021, Aug. 2006.

[22] Linde, Y.; Buzo, A.; Gray, R.; , "An Algorithm for Vector Quantizer Design," Communications, IEEE Transactions on ,

vol.28, no.l, pp. 84- 9S, Jan 1980.

[23] Leauhatong, T.; Hamamoto, K.; Atsuta, K.; Kondo, S.; , "A New Similarity Measure for Content-Based Image Retrieval Using

the Multidimensional Generalization of the Wald-Wolfowitz Runs Test," Communications and Information Technologies, 2008. ISClT 2008. International Symposium on , vol., no., pp.81-86, 21-23 Oct. 2008.

[24] Mandic, L.; Grgic, S.; Grgic, M.;, "Comparison of Color Difference Equations," Multimedia Signal Processing and Communications, 48th International Symposium ELMAR-2006 focused on , vol., no., pp.1 07 -110, June 2006.

[2S] J.H. Friedman and L.c. Rafsky, "Multivariate generalizations of

the Wolfowitz and Smirov two-sample tests", Annals of Statistics, vol. 7, no. 4, pp. 697-717, Jul. 1979.

[26] A. Wald and J. Wolfowitz, "On a test whether two samples are from the same population", Ann. Math. Statist., vol. 11, pp. 147-162, 1940.

[27] T. Leauhatong, K. Hamamoto, K. Atsuta, S. Kondo, "A New Content-based Image Retrieval Using the Multidimensional

Generalization of Wald-Wolfowitz Runs Test," IEE.T Trans. Electronics, Information and Systems, vol. 129, no. I, 2009