A Computer Vision System

for Diagnosing Scoliosis using moiré images

M. Batouche, R. Benlamri, M. K. Kholladi

Institut d'Informatique, University of Constantine

Route de Ain El-Bey, Constantine 25000, Algeria

Fax. (213) 4 69 09 16

ABSTRACT

For young people, scoliosis deformities are an evolving process which must be detected and

treated as early as possible. The moiré technique is simple, inexpensive, not aggressive and

especially convenient for detecting spinal deformations. Doctors make their diagnosis by

analysing the symmetry of fringes obtained by such techniques. In this paper, we present a

computer vision system for help diagnosing spinal deformations using noisy moiré images of the

human back. The approach adopted in this paper consists in extracting fringe contours from

moiré images, then localizing some anatomical features (the spinal column, lumbar hollow and

shoulder-blades) which are crucial for 3D surface generation that is carried out using Mota's

relaxation operator. Finally, rules furnished by doctors are used to derive the kind of the spinal

deformation and to yield the diagnosis. The proposed system has been tested on a set of noisy

moiré images, and the experimental results have shown its robustness and reliability for the

recognition of most scoliosis deformities.

Keywords : Vision system, scoliosis, moiré images, 3D surface generation, medical

diagnosis, discrete relaxation, image segmentation.

1 Introduction

For young people, scoliosis (spinal deformity) is a major public health problem. This is due to the

fact that a spinal deformity is an evolving process which must be detected and treated as early as

possible. The impetus for the research reported here stems from the enormous quantity of X-rays

which must be taken and examined at schools, nurseries and similar places in screening for

deformities. Furtheremore, recent medical research has shown that pubscent girls, diagnosed as

scoliotic and monitored in this way, are at a 10 times greater risk of breast cancer in later life[1].

A method that is inexpensive, not aggressive and capable of assessing the results of treatments

would be a major contribution. The moiré technique is a method of producing a contour line

image of an object without contact. It is simple: needing a camera, a light source and a grid

(figure.1).

Conventionally, the diagnosis of scoliosis is done manually. Doctors make the diagnosis from a

photograph of a child's back obtained by the moiré technique. The work is repetitive and tedious,

as many photographs must be interpreted. A computer vision system for aiding this process

would be very useful.

In this paper, we present a computer vision system for help decision support in the recognition of

spinal deformations from moiré images of the human back. The objective is to realise an

autonomous system which takes as input an image of a back and yields the diagnosis, recognising

the kind of spinal deformation, if any. Furthermore, the system is intended to detect suspicious

cases of spinal deformities which are not easily diagnosed by a clinical examination. We aim to

produce a preliminary diagnostic aid which allows to separate suspicious cases from other cases,

recognising that replacing a doctor is not a realistic goal. The analysis proceeds in four steps: a

segmentation process based on the Infinite size Symmetric Exponential Filter (ISEF) [2] which is

used to locate fringes; anatomical features are then localised (the centres of shoulder-blades, the

lumbar hollow and the spinal column) and measurements of them performed. Mota's discrete

relaxation operator is then used for 3D surface generation from the noisy 2D contour lines

representing the fringes. Finally, the diagnosis is made.

The remainder of the paper is organised as follows. The following section describes the principle

of moiré topography. A detailed description of the moiré image segmentation process is then

given followed by a section that describes the feature extraction and 3D surface reconstruction

processes. The interpretation and diagnosis stages and a set of experimental results are given in

the penultimate section. Finally, conclusions are drawn from the work.

2 Moiré Topography

Projecting a grating pattern onto an object's surface is an easy and efficient way to characterize

the 3D information of the object. When a grating pattern is projected onto an object's surface,

the grating pattern is perturbed according to the topography of the object. The perturbed grating

pattern carries the 3D information. Similarly, the moiré interference of a grid with its shadow

cast onto a surface of an object (figure1) create 2D moiré fringes which carry 3D information.

The moiré technique [3,4,5,6] produces images with alternate bright and dark fringes. Moiré

fringes are level set contours (equidistance contours) which look like contours of cross sections

(figure 2). The generation of these contours can be understood as the phase modulation and

subsequent demodulation of a carrier signal when the source and the observer are at a finite

distance from the grid (figure 1). The intensity of a point M(x, y, z) viewed by the observer

(figure 1) can be approximated by (from [7 ]):

I x yC d z

p hobserved ( , ) cos.

.

22 (1)

Then the difference of the height between the points p1 and p2 as shown in figure1 is:

z = z - z = (n - n ) ph

d = N

ph

d (2)2 1 2 1

Where p is the period of the sinusoidal grid and N = (n2 - n1) is the number of fringes between the

points p2 and p1 (or the difference in the fringe number at the two points).

The grey level images used in this study (figure 2) are digitised using a CCD TV-camera of

resolution 512 * 512 eight bit pixels and recorded on a SUN-SPARC workstation.

3. Moiré Image Segmentation

The segmentation of a moiré image is required in order to extract fringe contours which consist

of a list of linked edges. These are used as the main primitive in the interpretation stage. Most

edge based methods are based on numerical differentiation. However, numerical differentiation

of images is an ill-posed problem [8,9]. This is due to the fact that differential operators are

sensitive to noise. Preprocessing such as smoothing is often used to reduce the noise. However,

even after smoothing, an essential difficulty remains due to the incompatible requirements of

noise insensitivity and edge localisation. To overcome this difficulty, the problem of extracting

fringe contours from low signal-to-noise ratio moiré images is approached with the use of the

Infinite size Symmetric Exponential Filter (ISEF) which is an optimal smoothing filter based on a

one step model (a step edge and the white noise) and a multi-edge model [2,10]. This optimal

smothing filter is a symmetric exponential filter of an infinitely large window size. The first and

second directional derivative operators can be realised by a recursive algorithm and calculated

simultaneously as shown below [2]:

Ix(x,y) =

x(I(x,y) f(x,y)) = I(x,y)

xf(x,y)

Ix(x,y) = I(x,y) f1(y) f2(y) (f2(x) - f1(x))

and

Iy(x,y) =

y (I(x,y) f(x,y)) = I(x,y)

yf(x,y)

Iy(x,y) = I(x,y) f1(x) f2(x) (f2(y) - f1(y))

where :

I(x,y) is the input grey level image.

f(x,y) is the optimum ISEF filter extented to the two dimensions.

function f1and f2 are realised by simple recursive algorithms [2].

Finally, the Gradient vector at each point of the image is approximated by:

I(x,y) = (Ix(x,y), Iy(x,y))

The derivative images are obtained by applying in pipeline fashion a smoothing process

followed by a diffentiation as shown in the figure 3.

Experimental results have shown that edges detected by differentiation of optimal exponential

are less noisy and very well localised (figure 4). However, this model of processing require a long

processing time due to the amount of data to be treated and the number of operations required to

compute each output edge-pixel. A parallel implementation of a such segmentation algorithm is

described in previous work [11,12]. The proposed parallel system exploits both spatial and

temporal parallelism inherent in this type of processing. The temporal characteristic suggests the

use of a pipeline consisting of two stages. The first stage performs the smoothing operation and

then communicates the resulting data to the second pipe which carries out differentiation. The

spatial parallelism is due to the fact that each line (column) of the input image can be processed

separately and this implies that each line (column) is assigned to a processing element. This

enables an overlapping of a sequence of input images.

Once the image edge map is produced, edges are linked into contours using an edge chaining

algorithm [13]. Every contour is a chain of pixel coordinates which is oriented using the gradient

information such that the darker region is always at the right of the oriented contour. These

oriented chains are approximated into chains of segments (figure 4) using Wall and Danielson's

algorithm for the polygonal approximation [14]. Then, they are connected in order to obtain

reliable fringe contours [15].

4. Three-dimensional Surface Generation

The fringes created by the moiré interference of a grid with its shadow cast onto a surface of an

object provide relief information. The relief of a back from moiré images may be reconstructed

by estimating each fringe's altitude. The fundamental assumption is that points belonging to the

same fringe have the same altitude, and the altitude difference between neighbouring (parallel)

fringes is relatively constant.

A disadvantage of the moiré method is that the direction of depth change between fringes (i.e.

increasing or decreasing depth) is lost in the operation of contouring. Additional information is

needed to eliminate this ambiguity. A priori knowledge is used to generate the 3D surface by first

localising relevant anatomical features (figure 5) such as the centres of shoulder-blades, the

lumbar hollow and the spinal column.

To find the shoulder-blade centres we must use a priori knowledge. Although moiré images of

backs can be quite different from one person to another, they have many common visual

characteristics. We can note that fringes in the shoulder-blade region appear as concentric curves.

The shoulder-blade centre can be characterised as the centre of the inner fringe in this area.

Therefore, to find the innermost fringe, first, fringes located at the neighbourhood of shoulder-

blades are approximated by least-squares ellipses, and then, a search is performed to determine

the innermost ellipse [16].

To localise the lumbar hollow, we know that the spinal column traverses this region and the

lumbar fringes are vertical. Therefore, the centre of the lumbar hollow can be characterised as

the intersection point of the spinal column line and the straight line which approximates the

maxima and minima of the vertical curve sections (figure 6) [17].

The reconstruction of the spine itself is also crucial, since it allows us to check whether the

fringes are symmetric or not. To make the diagnosis, the human back is divided into three major

regions as follows:

1. Dorsal fringes, i.e. those that appear above the shoulder-blade centres.

2. Lumbar fringes, i.e. fringes which are "vertical" and are in the proximity of the

lumbar hollow.

3. Dorsal lumbar fringes, i.e., those that are between the shoulder-blade centres and the

lumbar hollow. These fringes appear as "horizontal" curves.

To measure the inclination of fringes in each region, we select the central maxima and minima of

the curves contained in each region, and approximate these points with a straight line (regression

line), as shown in figure 7. Finally, To reconstruct the spine, we use a spline curve which

approximates all the maxima and minima (figure 7). However, in the lowermost region of the

image, we cannot use the same approach. To check for fringes' inclinations, the curves in this

region are approximated by a least-squares line which provides more stable information about the

spinal deformation [17]

Having obtained reliable fringes, one proceeds to the three dimensional surface generation of the

back. The difficulty, of course, is that a grey level image is a 2D projection of the 3D scene.

However, in this case, fortunately, the fringes provide shape information. By using the hypothesis

that points belonging to the same fringe are at the same altitude, and that the altitude variation

between neighbouring fringes is relatively constant, one tries to obtain the depth relationship

between neighbouring fringes. The remaining problem is that the direction of depth change

between fringes may not be uniquely determined. To eliminate this ambiguity, we use a priori

knowledge such as the relevant features.

The problem of estimating each fringe level is similar to the consistent labeling and constraint-

satisfaction problems [18,19,20]. One way to resolve it is to assign each contour fringe a set of

possible labels which correspond to possible relative fringe levels. Then, one uses a discrete

relaxation operator [20,21] to eliminate incompatible labels by examining pairs of parallel

(neighbouring) contour fringes. A label of a fringe is discarded if and only if there is no

supporting label in at least one neighbouring fringe. This relaxation operator provides an efficient

way of reducing ambiguity in the case of ambiguous interpretations but requires that there is no

ambiguity in the segmentation process, i.e. that there is no oversegmentation (each contour fringe

is at only one altitude). It means that the ordinary discrete relaxation operator is very sensitive to

errors in the segmentation process.

One way to overcome the above mentioned problem is to split each ambiguous contour fringe

into two or more atomic contour fringes so that it retains no ambiguity in the "imperfect"

segmentation. Then we can use tree search methods to choose the best

segmentation/interpretation. However, search methods are sequential and their run-time

complexity is still exponential. A more convenient and efficient way is to use Mota's discrete

relaxation operator.

Mota and Velasco have introduced a new discrete relaxation operator [22] which deals with

ambiguous ("imperfect") segmentation. This new operator considers all possible segmentations

resulting from an ambiguous segmentation simultaneously in only one relaxation process. The

ambiguity in the segmentation process is modeled assuming that every segment (contour fringe)

in any segmentation is formed as the union of a finite set of atomic segments. The output of the

relaxation process is a collection of segmentations and interpretations with no conflict between

interpretations given to related contour fringes. Among these segmentations, we select the one

which contains less atomic contour fringes to have more significant features.

Estimation of the relative fringe level proceeds in three steps: first determining all hypotheses

which eliminate the ambiguity in the initial segmenation by propagating relief information from

the relevant points, then constructing the neighbours of each atomic segment (contour fringe) in

each possible segmentation and finally, applying Mota's relaxation operator to find the best

segmentation/interpretation.

To determine the neighbouring of each contour in every segmentation, we use parallelism

relationships and a priori khowledge. Then we construct the constraints network by propagating

the relief information from the relevant points according to the object's topology. It is obvious

that points in the shoulder-blade region are higher than points belonging to lumbar hollow region

and so on.

Mota's relaxation operator can be defined as follows: a label of a contour fringe is discarded if

and only if there is no segmentation contained in the ambiguous segmentation which has a

supporting label. This operator is applied many times, iteratively until no application is possible.

The repeated application of this operator converges. The proof is based on the fact that the initial

labeling is a finite set of labels and each application of this operator reduces the number of labels

until a stable state is reached [22].

For example, assume that we have the ambiguous segmentation A shown in figure 8:

A = {{a},{b},{c},{d},{b,c},{c,d}}

where

R = {{a},{b},{c},{d}} is the set of atomic contour fringes.

and

{b,c},{c,d}are the ambiguities as shown in figure 8.

Let us also assume that we have the following constraints :

a >1 b, a > 1 c, a > 1 cd and a > 1 bc

b > 1 d and b > 1 cd

bc > 1 d

and

f(a) = f(b) = f(c) = f(d) = f(bc) = f(cd) = {1,2,3}.

where :

x > 1 y x = y + 1

and f represents the initial labelling.

The neighbouring of each contour (figure 8) in every segmentation of (S1, S2, S3) is described in

table I, and the iterative application of Mota's relaxation operator is given in table II. The results

in table II show that the best and most significant segmentation/interpretation that contains less

atomic contour fringes is S2.

S2 = {a, {b,c},d}

with f(a) = 3, f({b,c}) = 2 and f(d) = 1.

In the example shown above f was initialised with {1, 2, 3}. However, in a moiré image, there is

no prior knowledge about the number of fringes. Therefore, one needs to determine the highest

fringe level (which will correspond to one of the shoulder-blades) which is given the largest

label. In this way, the relaxation process will surely converge towards the most consistent

segmentation/interpretation (figure 9).

Mota's relaxation operator is a parallel operator in the sense that it can be applied simultaneously

to every contour fringe. It is also clear that if the initial segmentation is not ambiguous, Mota's

relaxation operator reduces to ordinary discrete relaxation operator.

Once the relative elevation of each fringe (contour line) is determined using Mota’s relaxation

operator, the relative heights are computed using formula (2) given in section 2. These computed

heights are used to generate the three-dimensional relief of the back by interpolating the 3D

contour fringes by cubic-splines (figures 10-11). It should be noted that the computed heights are

found to agree with the actual heights to the nearest half of a millimeter. This error was

quantified by comparing mechanically measured heights and optically measured heights

(computed heights) on many sample objects. After the 3D reconstruction of the human back, we

can make measurements of depth differences on both sides of the spine at different regions. The

rules furnished by doctors are used to derive the kind of spinal deformation and to yield the

diagnosis.

5. Interpretation and Diagnosis

The doctor's analysis is based on the symmetry of fringes on both sides of the spine in different

regions. Particularly, rules furnished by doctors are used to derive three classes of spinal

deformations. These are obtained by projecting the spinal column onto three different planes

(frontal, sagital and horizontal).

The projection of the spinal column onto the frontal plane enables the detection of the two kinds

of scoliosis (figure 12): with simple or with double curvature. Similarly, the projection of the

spinal column onto the sagital plane (figure 13) enables the detection of three types of

deformities: hypercyphosis, hyperlordosis and the planar back. The presence of either

hypercyphosis and hyperlordosis is characterized by a high density of fringes at respectively the

cervical and the lumbar regions; a planar back is characterized by few fringes at the dorsal and

dorso-lumbar regions. These two classes of deformations are easily identified by comparing the

angle of curvature of the spinal column to certain threshold values as shown in figure.14.

The projection of the spinal column on a set of horizontal planes enables the detection of

gibosities. A gibosity is a partial torsion of the spinal column (figure 15), leading to rotation of

some spines. On a moiré image, this is characterized by the depth difference on either side of the

spine at different heights (figure 16). The expertise provided by doctors [23] is used to derive the

kind of gibosity and to yield a diagnosis.

A knowledge-based system is used for diagnosing gibosities. It takes as input the measurements

(D, DL, L) as shown in figure.16, which are used to compute the high (H) and low (L) depth

differences as given below:

H = D DL and L = DL L

As far as the diagnosis reliability is concerned, it could be noted that the depth error due to

optically measured heights (computed heights) which corresponds approximately to a half

fringe elevation has no significant effect on the above computed values, and thus, on the

system diagnosis. These two parameters (H, L) are used to classify gibosities in the

following way:

Class A:

Depth-

differences

Range values

H 0

L 0

Diagnostic: Normal back

Class B:

Depth-

differences

Range values Range values

H >0 <0

L >0 <0

Diagnostic: Global gibosity right if

left if

H

L

0

0 with a total amplitude of |H+L|

Class C:

Depth-

differences

Range values Range values

H 0 0

L 0 0

Diagnostic:

Gibosity right if

left if and

high if

low if

( )

( )

H L

H L

H

H

0

0

0

0 with an amplitude of : max(|H|,|L|)

Class D:

Depth-

differences

Range values Range values

H 0 0

L 0 0

Diagnostic:

Double gibosity towards the right if

towards the left if and

high if

at equilibrium if

low if

( )

( )

| | | |

| | | |

| | | |

H L

H L

H L

H L

H L

0

0

with a max amplitude of : max(|H|,|L|)

The above expertise is translated into rules to be used by a knowledge-based system to automate

the detection of gibosities [16]. The proposed decision support system has been implemented in

C language. The system has been extensively tested and the results are acceptable in that most of

them agree with the doctor's diagnoses. Some example results are shown in figure.17.

6. Conclusion

In this paper we have described the implementation of an automated system for help decision

support to detect spinal deformation. The objectif is to produce a system that could be used as a

first aid in schools, nurseries and similar places to detect suspicious cases of spinal deformities

that are not easily diagnosed by a clinical examination. Thus, the main purpose is to separate

suspicious cases from other cases, recognising that replacing a doctor is not a realistic goal.

The paper outlines a new approach for 3D surface generation from noisy and ambiguous

segmentation using Mota and Velasco's discrete relaxation operator. A knowledge based system

written in C is used for diagnosis; it takes as input measurements provided by the low-level

processes and yields an interpretation. The experimental results have shown that the extracted

shape information can be successfully used to diagnose the spinal deformation.

References

[1] P. Curran and D. Groves. "Assessing Spinal Deformities", Image Processing

Magazine, September/October, (1990), pp.14-16.

[2] S. Castan, J. Zhao and J. Shen. "Optimal Filter for Edge Detection Methods and

Results", Proc. of the First European Conf. on Computer Vision, France (1990),

pp.13-17.

[3] S. Takasaki. "Moire Topgraphy", Applied Optics, Vol.9, No.6, (1970),

pp. 1467-1472.

[4] L. Rayleigh. "On the Manifacture and Theory of Diffraction-Gratings", Journal of

Science, Vol. 4, No 310, (1974), pp.81-93.

[5] M. Idesawa, T. Yatagai and T. Soma. "Scanning Moiré Methods and Automatic

Measurement of 3D shapes", Applied Optics, Vol.16, No. 8, (1977),

pp.2152-2162.

[6] M. Idesawa. "3D Model Reconstruction and Processing for CAE", Proc. of the 8th

Int. Conf. on Pattern Recognition, France, (1986), pp.220-225.

[7] M. D. Meadows, W. O. Johnson and J. B. Allen. "Generation of Surface Contours

by Moiré Patterns", Applied Optics, Vol.9, No.4, (1970), pp942-947.

[8] T. Poggio, V. Torre and C. Koch. "Computational Vision and Regularization Theory",

Nature, (1985), pp. 314-319.

[9] V. Torre and T. Poggio. "On Edge Detection, IEEE Trans. on PAMI, Vol.8, No.2,

(1986), pp.147-163.

[10] J. Shen and S. Castan. "Un nouvel Algorithme de Detection de Contours", Proc of

the 5th AFCET Conference. on Pattern Recognition and Atificial Intelligence,

France, (1985), pp. 201-212.

[11] M. Batouche, R. Benlamri, H. Ayaidia and M. Bounekkar, "Implémentation Parallèle

des Méthodes de Détéction de Contours Basés sur le Filtre Optimal ISEF", Proc. of

the Int. Conf. on Signals and Systems (ICSS’94), Vol.1, pp.III.154-III.158, Algiers,

24-26 September 1994.

[12] M. Batouche & R. Benlamri."A Computer Vision System for Diagnosing Scoliosis",

Proc. of the IEEE Conf. on Systems, Man and Cybernetics (SMC’94), Vol.3, pp.2623-

2628, Edition IEEE, ISBN: 0-7803-2129-4, San Antonio, Texas, USA, 2-5 October, 1994.

[13] G. Giraudon. "An Efficient Edge Following Algorithm", Proc. of the 5th Scandinavian

Conf. on Image Analysis, Stockholm, (1987), pp.547-554.

[14] K. Wall and P. Danielson. "A fast Sequential Method for Polygonal Approximation of

Digitized Curves", Computer Vision, Graphics and Image Processing, Vol.28,

(1984), pp.220-227.

[15] M. Batouche, K. Tombre and P. Leduc. "Towards an .Automatic System for Diagnosing

Spinal Deformations.", Proc. of the 7th Scandinavian Conference on Image Analysis,

(Aalborg) Denmark, Vol. 1, pp. 562-569, 1991.

[16] M. Batouche. "Un système de vision pour l’interprétation des clichés moiré rachidiens",

Ph.D Thesis, Institut National Polytechnique de Lorraine, 1993.

[17] M. Batouche. "A knowledge based system. for Diagnosing Spinal Deformations : Moiré

Pattern Analysis and Interpretation.", Proc. of the 11th Int. Conf. on Pattern Recognition,

(Den Haag) Netherlands, Vol. 1, pp. 591-594, 1992.

[18] V. Kumar. "Algorithms for Constraint-Satisfaction Problems: A Survey", Artificial

Intelligenge Magazine, Vol.13, No.1, (1992), pp.32-44.

[19] Y. C. Cheng and S. Y. Lu "The Binary Consistency Checking Scheme and its

Applications

to Seismic Horizon Detection", IEEE Trans. on PAMI, Vol.11, No.4,(1989), pp.439-447.

[20] R. A. Rosenfeld, R A Hummel and S. W. Zuker. "Scene Labeling by Relaxation

Operations", IEEE Trans. on Systems, Man and Cybernetics, Vol.6, (1976), pp.420-433.

[21] R. Mohr and T. C. Henderson. "Arc and Path Consistency Revisited", Artificial

Intelligence, Vol.28, (1986), pp.225-233.

[22] F. A. Mota and F. R. D. Velasco. "A Method for the Analysis of Ambiguous

Segmentations of Images", IEEE Trans. on PAMI, Vol.8, No.6, (1986), pp.755-760.

[23] P. Leduc. "Element de Lecture d'un Cliché Moiré Rachidien", Internal Report, Centre de

Medecine Preventive, Nancy, France, (1988).

S1 S2 S3

a b,c bc b,cd

b a,d -- a,cd

c a -- --

d b bc --

bc -- a,d --

cd -- -- a,b

Table I. The Neighbouring of Fringe Contours in Every Segmentation

a b c d bc cd

f 1,2,3 1,2,3 1,2,3 1,2,3 1,2,3 1,2,3

f 2,3 2 1,2 1,2 2 1,2

2f 3 2 1,2 1 2 1

3f 3 2 2 1 2

4f 3 2 2 1 2

Table II. The Iterative Application of Mota's Relaxation Operator

I(x,y)f1(y) f2(y)

f1(x)

f2(x)

* 2

Ix(x,y)

Ixx(x,y)

Smoothing block

Differential block

+

+

++

_

_

Input

imageDifferential

output

Figure 3. Processing model of ISEF filter [2].