comparison between ophthalmoscopy and fundus photography in determining severity of diabetic...
TRANSCRIPT
Comparison Between Ophthalmoscopy and Fundus Photography in Determining Severity of Diabetic Retinopathy SCOT E. MOSS, MA, RONALD KLEIN, MD, STEPHEN D. KESSLER, OD, KAREN A. RICHIE, BA
Abstract: Diabetic retinopathy was assessed in a population-based study of 2708 diabetic persons in southern Wisconsin. The retinopathy levels as determined by opthalmoscopy and by the grading of stereoscopic fundus photographs were compared in the eyes of 1949 persons. Ophthalmoscopy was performed by an ophthalmologist and a specially trained optometrist and ophthalmic technician. Consultation among the three examiners was permitted. There was exact agreement between ophthalmoscopy and grading for detecting retinopathy (none, nonproliferative, proliferative) 85.7% of the time. The kappa statistic, which corrects for chance agreement, was 0.749. There were no significant differences among the three ophthalmoscopists. Ophthalmoscopy was more likely to disagree with fundus photography grading in eyes with less severe forms of retinopathy and in patients examined early in the study. Other factors found to influence the degree of agreement were age, visual acuity, and duration of diabetes. It is concluded that with proper training ophthalmoscopy can be an acceptable alternative to fundus photography in certain situations. [Key words: diabetic retinopathy, fundus photography, methodology, ophthalmoscopy.] Ophthalmology 92:62-67, 1985
Diabetic retinopathy is the leading cause of legal blindness in the United States.) A number of methods are currently used to detect and follow retinopathy and include ophthalmoscopy, stereo fundus photography and fluorescein angiography. Depending on the objective of detecting the retinopathy (eg. routine examination, screening, clinical trial, etc.) some of these methods are preferred over others. Grading severity of retinopathy from stereo fundus photography is the accepted standard method to evaluate progression in clinical trials and
large epidemiologic studies?3 There is less consensus on which method is preferred in routine examinations by nonophthalmologists. Since ophthalmoscopy is widely employed and less expensive its sensitivity in detecting retinopathy by individual examiners with varied backgrounds is important in planning public health strategies.4
There have been few reports comparing ophthalmoscopy and grading of fundus photographs for detecting severity of retinopathyY We compare these methods in data from a large, geographically defined population of diabetic persons.
From the Department of Ophthalmology, University of Wisconsin Medical School. Madison. Wisconsin
Supported by grant EY03083 (Dr. Klein) from the National Eye Institute.
Reprint requests to Ronald Klein, MD. Department of Ophthalmology, Clinical Science Center, 600 Highland Avenue, Madison, WI 53792.
62
MATERIALS AND METHODS
Case identification methods and descriptions of the population appear in detail in previous reports.6-
8 The
MOSS. et al • OPHTHALMOSCOPY AND FUNDUS PHOTOGRAPHY
Table 1. Comparison of Retinopathy as Determined by Ophthalmoscopy and Grading of Stereoscopic Fundus Photographs
in Right Eyes of Diabetic Persons (Wisconsin. HSA-1 . 1980-82)
No Reti-
Ophthalmoscopy nopathy
No Retinopathy 890 Nonproliferative
Diabetic Retinopathy 43 Proliferative
Diabetic Retinopathy 3
Grading of Stereoscopic Fundus Photographs
Nonproliferative Proliferative Diabetic Diabetic
Retinopathy Retinopathy
181 3
646 32
16 135
participants were examined in a mobile examination van in a location near their residences. The examination consisted of measuring blood pressure; measuring best corrected visual acuity using the Early Treatment Diabetic Retinopathy Study protocol;9 performing a slitlamp examination for chamber depth and the presence of iris neovascularization; measuring intraocular pressure by applanation; dilating the pupils; taking a medical history using a standard questionnaire; examining the lenses for cataracts; performing an ophthalmoscopic examination; taking stereoscopic color fundus photographs of seven standard fields and a nonstereoscopic red reflex photograph for each eye; determining urine glucose, ketone and protein; and determining blood glucose and glycosylated hemoglobin levels from a fingerprick capillary blood sample. The examination generally followed this order although during periods of heavy scheduling the examiners sometimes deviated from the order to make maximum use of equipment.
The protocol for the ophthalmoscopic examination called for direct ophthalmoscopy supplemented by indirect ophthalmoscopy if the examiners felt this was necessary for a complete evaluation of the eye. Whether the right or left eye was examined first was left to the preference of the examiner. Also, these was no prohibition of the examiner consulting with the other examiners in difficult or unusual cases although no record was kept of these occurrences. The examiner was to indicate whether retinopathy was absent, questionably present, definitely present, or ungradable. If retinopathy was definitely present, the examiner was to evaluate the presence or absence of microaneurysms or dot hemorrhages, hard and soft exudates, blot hemorrhages, new vessels at the disc or elsewhere, fibrous proliferans at the disc or elsewhere, and vitreous or preretinal hemorrhage. The presence of intraretinal microvascular abnormalities or venous beading were not determined by the ophthalmoscopist. For purposes of this paper retinopathy as described by the ophthalmoscopist was classified into three categories: no retinopathy; nonproliferative retinopathy based on the presence of microaneurysms or dot hemorrhages, blot hemorrhages, or hard
or soft exudates; and proliferative retinopathy based on the presence of new vessels at the disc or elsewhere, fibrous proliferans at the disc or elsewhere, or vitreous or preretinal hemorrhage.
The stereoscopic fundus photographs were graded at the Fundus Photograph Reading Center using the ETDRS adaptation of the Modified Airlie House Classification of Diabetic Retinopathy. 10 Double gradings of each set of photographs were performed by trained graders who were not involved in the ophthalmoscopic examinations. When the two determinations disagreed, the eye was regraded for general level by another grader. If that grader agreed with either of the first two determinations, that result was accepted. However, if discrepancies remained, the case was referred to the most senior grader for adjudication. One of nine levels of retinopathy was assigned to each eye (Appendix). The graders had no knowledge of the patients' medical histories or of the results of the ophthalmoscopic examinations. For this analysis, groupings of retinopathy into three severity levels (none, nonproliferative or proliferative retinopathy) comparable to those of the ophthalmoscopic examination were used.
The current age was defined as the age at the time of examination. Duration of diabetes was the time period between the age when the diagnosis was first recorded by a physician on the patient's chart or on the hospital record and the age at the time of examination. Cataract was defined as the presence of either nuclear sclerosis or posterior subcapsular cataract as described in a previous report. II
The database software system WISAR, Wisconsin Storage and Retrieval, was used for processing case records and for calculating the Student's t and chisquare statistics. 12 The kappa statistic, which measures the degree of agreement between observers above that expected by chance alone, was used to evaluate the agreement between ophthalmoscopy and stereoscopic fundus photography grading. 13
RESULTS
Of the 2708 diabetic persons examined, 2059 were examined by three principal observers (RK, SDK, KAR). These cases are the subject of this report. The remaining 649 patients were examined by 18 other observers. In addition, individual eyes were excluded for the following reasons: photographs not gradable, eyes enucleated, or no ophthalmoscopy performed (110 right eyes, 120 left eyes). Thus, 1949 right eyes and 1939 left eyes remained for statistical analysis. Parallel analyses for the right and left eyes were performed. As the results were similar, only the analysis for the right eye is presented.
Table 1 compares the retinopathy level from ophthalmoscopy with the level from grading of stereoscopic fundus photographs, hereafter referred to simply as "grading". The overall agreement between the methods is 85.7%. The agreements between methods for the eyes examined by each of the three ophthalmoscopists are
63
OPHTHALMOLOGY • JANUARY 1985 • VOLUME 92 • NUMBER 1
84.5, 85.5, and 86.5%. These are not significantly different statistically. A kappa statistic ofzero indicates agreement equal to that expected by chance, whereas a value of unity indicates complete agreement. The overall kappa statistic for the data in Table I is 0.749 [standard error (SE) 0.018], indicating a high degree of agreement between ophthalmoscopy and grading over that expected by chance. The kappa statistics for the three examiners are 0.716 (SE 0.036), 0.752 (SE 0.035), and 0.764 (SE 0.027). Again, these are not significantly different statistically.
Factors which might affect either ophthalmoscopy or grading and result in disagreements were examined. However, factors causing the ophthalmoscopist to overlook retinopathy detected by grading may differ from factors causing the ophthalmoscopist to see retinopathy not detected by grading. Thus, comparisons of factors were made between subgroups of the study population in which the methods agree or disagree. The subgroups were defined by the type of disagreement, that is, ophthalmoscopy detecting less or more retinopathy than grading, and by the level of retinopathy.
The first subgroup is that part of the study population in which proliferative diabetic retinopathy was found by grading, ie. column three of Table I (N = 170). Proliferative retinopathy was detected by ophthalmoscopy (agreement) in 135 cases (79%) and less retinopathy was detected in 35 cases. The photographs of the eyes in which there was disagreement were reviewed by one of the authors (RK) to establish the reasons for the discrepancies. Twenty-eight were found to be new vessels at the disc or elsewhere which were missed by the ophthalmoscopist, six were cases of fibrous proliferans missed by the ophthalmoscopist, and one case was considered to be a grader overcall of new vessels. Among the 170 eyes with proliferative diabetic retinopathy by grading, 36 had Diabetic Retinopathy Study high-risk characteristics.9 Only one of these was classified as nonproliferative diabetic retinopathy by ophthalmoscopy. Table 2 compares the agreements with the disagreements with respect to a number of factors. Disagreement between ophthalmoscopy and grading is associated with older current age (P < 0.01), better visual acuity (P < 0.05), and examination in the first year of the study (P < 0.005). A suggested association is found with poorer photograph quality (0.05 < P < 0.10).
The next subgroup consists of those subjects in which nonproliferative retinopathy was found by grading and nonproliferative or no retinopathy was found by ophthalmoscopy, ie. Table 1, column two, rows one and two (N = 827). Ophthalmoscopy agreed with grading in 646 cases (78%) and disagreed in 181 cases. Of the 827 eyes, microaneurysms alone were found in 30 I eyes by grading. Nonproliferative retinopathy was detected by ophthalmoscopy (agreement) in 150 cases (50%) and no retinopathy (disagreement) was detected in 151 cases. In the remaining 526 eyes, more severe nonproliferative retinopathy than microaneurysms alone was found by grading. Nonproliferative retinopathy was detected by
64
Table 2. Association of Various Characteristics with Agreement or Disagreement between Ophthalmoscopy and Grading in Persons
with Proliferative Diabetic Retinopathy by Grading
Characteristic Agreement Disagreement P
Number 135 35 Mean current age (SE) 42.1 (1.3) 49.6 (2.8) <0.01 Mean duration of
diabetes (SE) 22.2 (0.7) 20.5 (1.6) <0.50 Mean visual acuity,
letters correct (SE) 35.6* (1.7) 43.2 (2.8) <0.05 Mean diopters (SE) - .52t (0.27) .52:j: (.78) <0.25 Percent male 57.8 45.7 <0.50 Percent with cataracts 60.7 68.6 <0.75 .Percent aphakic 4.4 8.6 <0.75 Percent younger onset 75.6 60.0 <0.25 Percent with direct
ophthalmoscopy only 66.7 77.1 <0.50 Percent with photograph
quality less than fair 44.4 62.9 <0.10 Percent with photography
problems reported by the examiner 51.1 57.1 <0.75
Percent examined in first year of study 40.0 68.6 <0.005
SE = standard error of the mean. * Missing data in one case. tMissing data in 12 cases. :j:Missing data
in two cases.
ophthalmoscopy (agreement) in 496 cases (94%) and no retinopathy (disagreement) was detected in 30 cases (6%). Table 3 examines the same factors as in Table 2. Here an association is found between disagreement and
Table 3. Association of Various Characteristics with Agreement or Disagreement between Ophthalmoscopy and Grading in Persons
with Nonproliferative Retinopathy by Grading and Nonproliferative or No Retinopathy by Ophthalmoscopy
Characteristic Agreement Disagreement P
Number 646 181 Mean current age (SE) 50.5 (0.8) 51.4 (1.7) <0.75 Mean duration of diabetes
(SE) 16.6 (0.3) 11.8 (0.6) <0.001 Mean visual acuity, letters
correct (SE) 50.0 (0.5) 50.8 (0.8) <0.50 Mean diopters (SE) 0.26* (0.13) 0.7 4t (0.22) <0.10 Percent male 48:3 45.3 <0.75 Percent with cataracts 55.9 52.5 <0.50 Percent aphakic 4.5 4.4 <0.90 Percent younger onset 47.5 40.9 <0.25 Percent with direct
ophthalmoscopy only 74.3 79.6 <0.25 Percent with photograph
quality less than fair 39.2 41.4 <0.75 Percent with photography
problems reported by the examiner 50.0 49.7 <0.95
Percent examined in first year of study 58.7 66.9 <0.10
SE = standard error of the mean. * Missing data in three cases. t Missing data in five cases.
MOSS, et al • OPHTHALMOSCOPY AND FUNDUS PHOTOGRAPHY
Table 4. Association of Various Characteristics with Agreement or Disagreement between Ophthalmoscopy and Grading in Persons with Nonproliferative or No Retinopathy by Grading and Equal
or Greater Retinopathy by Ophthalmoscopy.
Characteristic Agreement Disagreement P
Number 1536 62 Mean current age (SE) 50.6 (0.6) 53.7 (2.9) <0.50 Mean duration of
diabetes (SE) 11.3 (0.2) 15.2 (1.2) <0.001 Mean visual acuity,
letters correct (SE) 50.9* (0.3) 49.2 (1.6) <0.50 Mean diopters (SE) 0.32t (O.OB) .51* (.42) <0.75 Percent male 4B.B 51.6 <0.90 Percent with cataracts 54.5 61.3 <0.50 Percent aphakic 3.B 4.B <0.95 Percent younger onset 39.0 35.5 <0.75 Percent with direct
ophthalmoscopy only 75.9 82.3 <0.50 Percent with photograph
quality less than fair 40.6 40.3 <0.95 Percent with photography
problems reported by the examiner 51.4 62.9 <0.25
Percent examined in first year of study 5B.9 67.7 <0.25
SE = standard error of the mean. * Missing data in one case. t Missing data in 16 cases.
shorter duration of diabetes (P < 0.001) and suggested associations are found with more myopia (0.05 < P < 0.10) and examination in the first year of the study (0.05 < P < 0.10).
The final subgroup consists of those cases in which the retinopathy level by ophthalmoscopy is greater than by grading or in which retinopathy by both methods is nonproliferative or absent ie. Table 1, column one and column two, rows two and three (N = 1598). Note that 646 of these cases are included in the previous subgroup as well. Ophthalmoscopy agrees with grading in 1536 (96%) and disagrees in 62 cases. There are 19 cases in which proliferative retinopathy was found on ophthalmoscopy which is not seen by grading which were reviewed by one of the authors (RK). Thirteen of these were cases of intraretinal microvascular abnormalities (IRMA) which were called proliferative retinopathy by the ophthalmoscopist, two cases were undercalls by the graders, two cases were new vessels at the disc explained by an etiology other than diabetes, and two discrepancies could not be explained. Table 4 again displays the factors possibly affecting agreement between methods. The only variable associated with disagreement is longer duration of diabetes (P < 0.001).
DISCUSSION
The three examiners in this study were an ophthalmologist, an optometrist and an opthalmic technician
who had extensive trammg in ophthalmoscopy and fundus photography. When their ratings of retinopathy level by ophthalmoscopy are compared to those obtained by the grading of stereoscopic fundus photographs, there are only minor differences in performance among them. However, as consultation among them occurred during the examinations, individual differences would tend to be lessened. Thus, these results may not reflect their true differences. As a group they have a high degree of agreement with grading of fundus photographs (86%). The diagnosis of proliferative retinopathy was missed by the examiner in 26% of cases in which it was found by grading. However, only one high-risk eye was missed by ophthalmoscopy and in only three of these eyes did the examiner see no retinopathy. In addition, their performance compares favorably to results of other studies.3,5 Sussman et al5 found overall error rates of 30 to 74% for various physician groups from retinal specialists to internists comparing ophthalmoscopic examinations to grading of fundus photographs. This compares to 14% in the present study. Also, they found serious error rates, ie. missing the diagnosis of proliferative retinopathy, of 0 to 70% compared to 26% in the present study. These rates are not strictly comparable, however, as Sussman et al., used more diagnostic categories which would tend to increase the error rate. Palmberg et al3
report prevalence rates of retinopathy of 31.5% as detected by ophthalmoscopy and 42.3% as detected by grading stereoscopic fundus photographs. The corresponding rates in the present study are 44.9% and 52.0%. Although the rates from the present study are higher, the magnitude and direction of their difference is comparable to those of Palmberg et al.
The implicit assumption has been made through much of the present study that the true diagnosis is reflected by the grading of fundus photographs. However, it is noted in the results that a few of the major disagreements between ophthalmoscopy and grading involving proliferative retinopathy are due to grader error. Although the minor errors involving nonproliferative retinopathy were not reviewed, it is likely that additional grader errors exist among them. 14
The retinopathy which is overlooked by the examiners during ophthalmoscopy is generally less severe. According to the detailed fundus photograph grading all but one of the cases of proliferative retinopathy missed by ophthalmoscopy had . less than high risk characteristics. Similarly the nonproliferative retinopathy overlooked by ophthalmoscopy was predominately classified as microaneurysms or less by detailed grading. Fifty percent of the retinopathy graded as microaneurysms alone was not detected by ophthalmoscopy.
The associations of agreement between ophthalmoscopy and grading with other variables are generally in the directions expected. The shorter duration of diabetes in persons where ophthalmoscopy undercalls retinopathy compared to grading and longer duration where ophthalmoscopy overcalls may represent an observer bias. As the examiner has knowledge of the patients' diabetes
65
OPHTHALMOLOGY • JANUARY 1985 • VOLUME 92 • NUMBER 1
history from the questionnaire prior to ophthalmoscopy, he/she may not be expecting to see as much retinopathy in shorter duration cases and, conversely, may be expecting more retinopathy in persons with longer duration. The time effect observed, that is more disagreements than expected in the first year of the study and fewer than expected in the second year, probably reflects a learning process on the part of the observers. The finding of better visual acuity in the cases of disagreement in the first subgroup is not easily explained. However, given the number of statistical tests performed it is not unlikely that a spurious difference could occur by chance. A number of factors which were expected to have some effect on ophthalmoscopy and/or grading in fact had little effect. These include the presence of cataracts or aphakia, type of ophthalmoscopy, photography problems and photograph quality. We conclude that these factors either have no detectable effect or affect ophthalmoscopy and grading equally.
The advantages of ophthalmoscopy are that it does not require an expensive fundus camera and it has negligible ongoing costs. In addition, with ophthalmoscopy it is possible to examine peripheral areas of the fundus outside of the area usually photographed and thus detect retinopathy not included in the seven fields. Its disadvantages are lack of stereopsis with the direct method (which may make the detection of new vessels difficult) and lack of a permanent record of retinopathy. Thus it is difficult to accurately document subtle changes in retinopathy OVer time. Stereoscopic fundus photography has the advantage of providing a permanent record of retinopathy which can be used at a later date to document progression of the retinopathy or response to treatment. It also provides the opportunity for a more detailed grading of retinopathy. The disadvantages of fundus photography are that it is limited to the posterior pole (thus leading to possible undercalling of the retinopathy) and its expense. In addition to the initial cost of the fundus camera there are ongoing costs of film, processing and grading.
In conclusion, ophthalmoscopy appears to be a conditionally acceptable alternative to grading of fundus photographs provided that only a gross classification of diabetic retinopathy as used in this report is required. The conditions are that the examiner be well trained in the procedure and that patients in which retinopathy is detected be referred for further evaluation. In situations where permanent documentation of retinopathy or grading of lesions is required (as in clinical trials or epidemiologic studies) fundus photography is necessary.
ACKNOWLEDGMENTS
The authors are grateful to Drs. Matthew D. Davis, Barbara E. K. Klein, David L. DeMets, Mr. Larry Hubbard and Ms. Jean Espenshade for consultation and criticism at different stages of the project; to Moneen Meuer for project coordination; to Stacy Meuer for data management; to Gloria Boone, Rose
66
Brothers, Magnus Harding, Cheryl Hiner, Yvonne Magli, Mary Peckham, and Anita Temple for detailed grading of the fundus photographs; Anik Ganguly for programming and data management advice; and to Mae Wildt for secretarial assistance.
APPENDIX
DEFINITIONS OF RETINOPATHY LEVELS9
For each eye, the maximum grade in any ofthe seven standard photographic fields was determined for each of the lesions used in defining the "retinopathy levels:"
Levell: No retinopathy. Level 1.5: Retinal hemorrhages only, no microaneu
rysms. Level 2: Microaneurysms (one or more) only. Level 3: Microaneurysms and one or more of the
following: retinal hemorrhages, but total of hemorrhages and microaneurysms less than Standard Photo #2A;13 hard exudates less than Standard Photo #3;13 soft exudates questionably present; intraretinal microvascular abnormalities questionably present; venous beading questionably present, or venous focal narrowing or loops definitely present.
Level 4: Microaneurysms and one or more of the following, but definition of Level 5 not met: hemorrhages and microaneurysms ~ Standard Photo #2A; hard exudates ~ Standard Photo #3; soft exudates definitely present; intraretinal microvascular abnormalities defi-. nitely present; venous beading definitely present; venous reduplication definitely present.
Level 5: In Fields 4 to 7 only, any three of the following: hemorrhages and microaneurysms ~ Standard Photo #2A in at least one field; soft exudates definitely present in ~ two fields; venous beading definitely present in ~ two fields, or intraretinal microvascular abnormalities present in four fields and ~ Standard Photo #8A in ~ two fieldsY
Level 6: Three sublevels: 6.0: fibrous proliferans only; 6.1-6.4: no evidence of 6.0 or 6.5 but scars of photocoagulation either in "scatter" or confluent patches, presumably directed at new vessels; 6.5: new vessels on or within one disc diameter of the disc graded less than Standard Photo # lOA; new vessels elsewhere of any extent or preretinal or vitreous hemorrhage.
Level 7: Diabetic Retinopathy Study high risk characteristics with one or more of the following: new vessels elsewhere greater than one half disc area and preretinal hemorrhage or vitreous hemorrhage; new vessels on or within one disc diameter of the disc graded less than Standard Photo # 1 OA with preretinal or vitreous hemorrhage; new vessels on or within one disc diameter of disc graded ~ Standard Photo # lOA with or without preretinal or vitreous hemorrhage.
Level 8: Eyes which could not be graded for retinopathy level because of vitreous hemorrhages obscuring the retina, phthisis bulbi, or enucleation secondary to a complication of diabetic retinopathy.
MOSS, et al • OPHTHALMOSCOPY AND FUNDUS PHOTOGRAPHY
REFERENCES
1. National Society to Prevent Blindness. Vision Problems in the US; Data Analysis: Definitions, Data Sources, Detailed Data Tables, Analysis, Interpretation. New York: National Society to Prevent Blindness, 1980.
2. Klein BEK, Davis MD, Segal P, et al. Diabetic retinopathy; assessment of severity and progression. Ophthalmology 1984; 91: 1 0-7.
3. Palmberg P, Smith M, Waltman S, et al. The natural history of retinopathy in insulin-dependent juvenile-onset diabetes. Ophthalmology 1981; 88:613-8.
4. Herman, WH, Teutsch SM, Sepe SJ, et al. An approach to the prevention of blindness in diabetes. Diabetes Care 1983; 6:608-. 13.
5. Sussman EJ, Tsiaras WG, Soper KA. Diagnosis of diabetic eye disease. JAMA 1982; 247:3231-4.
6. Klein R, Klein BE, Moss SE, et al. Prevalence of diabetes mellitus in southern Wisconsin. Am J Epidemiol 1984; 119:54-61.
7. Klein R, Klein BEK, Moss SE, et al. The Wisconsin Epidemiologic Study of Diabetic Retinopathy. II. Prevalence and risk of diabetic
retinopathy when age at diagnosis is less than 30 years. Arch Ophthalmol 1984; 102:520-6.
8. Klein R, Klein BEK, Moss SE, et al. The Wisconsin Epidemiologic Study of Diabetic Retinopathy. III. Prevalence and risk of diabetic retinopathy when age at diagnosis is 30 or more years. Arch Ophthalmol 1984; 102:527-32.
9. Early Treatment Diabetic Retinopathy Study. Manual of Operations. Baltimore: ETDRS Coordinating Center, Department of Epidemiology and Preventive Medicine, University of Maryland, 1980; Chapter 12.
10. Early Treatment Diabetic Retinopathy Study. Manual of Operations. Baltimore: ETDRS Coordinating Center, Department of Epidemiology and Preventive Medicine, University of Maryland, 1980; Chapters 5, 18.
11. Klein R, Klein BEK, Moss SE. Visual impairment in diabetes. Ophthalmology 1984; 91: 1-8.
12. Harberg J, Holladay 0, Entine S, et aI. WISAR: Wisconsin Storage and Retrieval System. Madison: University of Wisconsin Clinical Cancer Center, 1979.
13. Fleiss JL, Cohen J, Everitt BS. Large sample standard errors ( kappa and weighted kappa. Psychol Bull 1969; 72:323-7.
14. Davis MD, Hubbard LD, Trautmann J, et al. Methodology for the study and classification of diabetic retinopathy in juvenile onset diabetic'patients included in the KROC study. Diabetes, In press.
67