Lydia P. E. van der Steen, B.Sc.(Math.)J. Joris Hage, M.D., Ph.D.Martijn P. J. Loonen, M.D.Moshe Kon, M.D., Ph.D.

Plast Reconstr Surg. 2004 Jul;114(1):113-20

From the multitude of oral presentations at major medical meetings, the most in-formative and highest-quality studies make it to full publication in peer-reviewedjournals. The rate of publication may be regarded as an indicator of the scientificlevel of the meeting. Study of the publication rates of consecutive annual meetingsallows for the evaluation of the consistency of the scientific level of these meetingsand for comparison with publication rates of other meetings in the same field ofinterest. To grade how useful any publication is to other authors, one can further-more measure how frequently they cite it in their own publications.Finally, the time lag between oral presentation and full publication is of importanceto both its authors and the audience at the meeting. The main objectives of thisstudy were to determine the publication rate of papers of various fields of interestas presented at five consecutive annual meetings of the European Association ofPlastic Surgeons (EURAPS) and the time lag between these presentations and theirpublication. The authors compared their overall findings to those reported for othersurgical specialties. Moreover, they identified and classified the journals in whichthe full publications appeared as an indicator of the scientific value of the meeting.They conclude that a greater than average number of papers presented at the1995 through 1999 annual EURAPS meetings went on to full publication in peer-reviewed journals. Among these journals, Plastic and Reconstructive Surgery wasthe best source for information presented at the meetings.

Although approximately 90 percent of the publications appeared before 3 yearshad passed after a meeting, additional publications may be expected to appeareven more than 6 years after the meeting. Given the high publication rate and thehigh average normalized impact factor of the journals in which the presentationsappeared, the five studied EURAPS meetings overall had high scientific value.

40

From the multitude of oral presentations at major medical meetings, the most in-formative and highest-quality studies make it to full publication in peer-reviewedjournals.1 Given that the quality of the presentation is associated with the likelihoodof such publication,2,3 the rate of publication may be regarded as an indicator ofthe scientific level of the meeting.4,5 Study of the publication rates of any society’sconsecutive annual meetings allows for the evaluation of the consistency of the sci-entific level of these meetings. Large differences in the publication rates from oneyear to another may suggest fluctuations in the quality of oral presentations andthat alterations in the criteria for selection of abstracts may be necessary.2Moreover, differentiation of publication rates of presentations in subspecializedfields of interest may indicate which of these fields is less well represented duringthe meeting, as compared with other subspecialized fields.2,3,5 Finally, if the publi-cation rate of a certain society’s meeting falls far below the typical publication rateof other meetings in the same field of interest, the meeting in general can bejudged as less informative.1 Still, full publication is only an intermediate outcomethat may be satisfying to the authors but may not necessarily be of use to others.To grade how useful any publication is to other authors, we can measure how fre-quently they cite it in their own publications.6 Apart from the rate and grade of pub-lication, the meeting’s typical time lag between oral presentation and fullpublication is of importance to both its authors and their audience at the meeting.2The rate, grade, and date of full publication of papers presented at scientific meet-ings have been studied for various medical specialties7,8 but never for plastic sur-gery. For the reasons mentioned above, the main objectives of our study were todetermine the publication rate of papers of various fields of interest as presentedat five consecutive annual meetings of the European Association of Plastic Sur-geons (EURAPS) and the time lag between these presentations and their publica-tion. We compared our overall findings with those reported for other surgicalspecialties. Moreover, we identified and classified the journals in which the fullpublications appeared as an indicator of the scientific value of the meeting.

All 306 papers presented at the annual scientific meetings of the EURAPS from1995 through 1999 and included in the respective abstract books were enteredinto a database. Each presentation was categorized in one of the three categories“clinical study,” “basic research,” and “aesthetic surgery” discriminated by the EU-RAPS’s scientific committee, and this category was noted in the database alongwith the date of presentation (Table I). In February of 2003, we searched PubMedto identify all presentations that had been published in full in a peer-reviewed jour-nal.9 Because of the ability to search from 1966 through 2002,2 the use of PubMedmade it possible to find presentations published even before the respective meet-ings. Apart from PubMed, all volumes of the European Journal of Plastic Surgeryup to December 31, 2002, were searched for matching publications. Although thisjournal is not indexed in PubMed, we included the European Journal of Plastic Sur-gery because it is the official organ of EURAPS and of six European national plastic

41

surgical societies.The search machine of the European Journal of Plastic Surgery website was usedto trace any publication from January of 1998 through 2002,10 whereas the indexof the bound annual volumes of its printed issues were searched for any presen-tations published until 1998. The searches began with the first author’s name but,if needed, all the authors’ names were used until a matching title was found. If amatch was not found, PubMed and the European Journal of Plastic Surgery weresearched for keywords from the title or abstract of the presentation.1 Whenever apublication was retrieved the authors, title, and abstract of the oral presentationwere compared with those of the full publication to establish whether they matchedin accordance with the stringent criteria used for previous studies.2,5,8,11All matching publications were entered into the database along with the date ofpublication and the journal name. Although cases of dual publication were noted,any second appearance of a matching publication was disregarded in our analysis.Finally, each publication was categorized by its publishing journal to allow for thegrading of the overall scientific value of the meeting. For this, we entered the im-pact factor of the publishing journal in the year of publication as calculated by theInstitute for Scientific Information (Philadelphia, Pa.).12This impact factor is the ratio of the number of citations to the journal in questionfor the year in question to the number of citable articles in that journal in the past2 years and hence, it is a measure of the average frequency of citations for the ar-ticles in the journal in question.12,13 For example, if there were 142 references (ci-tations) made in all journals indexed by the Institute for Scientific Information in2000 to a certain journal, and if that journal had published 100 citable articles in1998 and 1999, then the impact factor for 2000 of that journal would have been1.420.

Rate of PublicationOn the basis of the outcome of our search, overall rates of publication were cal-culated for each of the respective EURAPS meetings. These publication rates werecompared with each other by the chi-square test and with those reported for othersurgical disciplines in studies that used MEDLINE or PubMed to identify subsequentfull publication.2,3,11,14 –19

Because the number of presentations studied in these reports varied extensively,we weighted the reported publication rate by the square root of the total numberof presentations at each of the meetings reported on to obtain a weighted mean

42

publication rate.8,20 The rates of publication of presentations in each of the threecategories were determined and tested by the chi-square method to determinewhether these categories acted as predictors of likelihood of full publication.21 Weaccepted a confidence interval of 95 percent for the chi-square tests, and p valuesless than 0.05 were accepted as indicative of statistical significance.

Date of PublicationFor those papers that were published, the mean and median time lag betweenpresentation and publication were determined. Papers published before their pres-entation at a EURAPS meeting were given a negative value for this time lag. Wecompared the observed cumulative rate of publication over time with those re-ported for other surgical specialties.2,3,11,15–17,19

Grade of PublicationBecause the Institute for Scientific Information impact factor of any given journalfluctuates over the years and because presentations from any one meeting arepublished in various years, we normalized the scientific value of a journal by cal-culation of the ratio of its impact factor in the year of publication to the impact fac-tor of Plastic and Reconstructive Surgery in the same year.22–24 In this way, it ispossible to compare the average quality of published presentations of each meet-ing to that of the other meetings.22,23 We chose Plastic and Reconstructive Surgeryas the definitive standard for this because it has an impact factor that is continu-ously and clearly above that of other journals related to plastic surgery.12

Rate of PublicationOf the 306 studied presentations, 195 were published in PubMed-indexed journalsor the European Journal of Plastic Surgery until December 31, 2002, giving anoverall publication rate of 63.7 percent (Table II). Six presentations were duallypublished. The publication rates for each of the five annual meetings did not sig-nificantly differ (p =0 .49). The fraction of presentations in each of the three cat-egories fluctuated over the years and there was a nonsignificant trend thatpresentations on basic research, in general, were more likely to be followed by fullpublication than those on aesthetic surgery (p = 0.06) or clinical study (p = 0.15)(Table II). The overall publication rate observed in our study compared favorably

43

with those reported for other individual surgical meetings and with the weightedmean rate for all of these meetings (Table III).

Date of PublicationThe mean delay from oral presentation to full publication over the years was 14months (SD, 23.9 months; median, 13 months), with 40 of the 306 presentations(13 percent) having already been published in full before the actual meeting (TableIV).5 Even though 174 of the 195 publications (89 percent) had appeared in printat 3 years after the respective meetings, the delay of full publication of the 1995and 1996 presentations was found to amount up to 89 and 72 months, respec-

tively.Grade of PublicationThe 195 publications to date appeared in 52 different journals, of which 16 wereplastic surgery, microsurgery, aesthetic surgery, craniofacial surgery, or hand sur-gery journals. Most appeared in Plastic and Reconstructive Surgery (n = 72), fol-lowed by the British Journal of Plastic Surgery (n = 17), European Journal of PlasticSurgery (n =16), and Annals of Plastic Surgery (n =11) (Table V). The impact fac-tors of the 52 publishing journals over the years varied from 0.157 to 31.473 (TableVI). The mean impact factor of the publishing journals proved susceptible to out-liers, whereas the median impact factor for each year was equal or close to thatof our definitive standard, Plastic and Reconstructive Surgery. Hence, the median

44

of the so-called normalized ratio of the publishing journal and Plastic and Recon-structive Surgery was 1.0 for the 1996, 1998, and 1999 meetings, 0.4 for the 1995meeting, and 0.7 for the 1997 meeting.

Presentation at a meeting is an accepted first step toward the dissemination ofnew scientific work, but the abstracts of these meetings are often published onlyin proceedings that are exclusively available to those who attended the meeting.5These abstracts are therefore of little use to the scientific community at large andthe ultimate goal of any research project should be to take the next step, in whichthe manuscript is published in a peer-reviewed journal.1 Moreover, such publicationis considered an acknowledgment of the study’s validity and represents the bench-mark of a researcher’s accomplishments,1,3,16 whereas failure to disseminate the re-sults of well-designed and well-conducted trials ought to be considered unethical.20

Rate of PublicationBefore we discuss the implications of our findings, some potential methodologicallimitations of our study should be considered. They include the possibility of omit-ting papers published in journals that are not indexed in PubMed other than Eu-ropean Journal of Plastic Surgery and our inclusion of publications in the latterjournal. By relying solely on the indexes of PubMed and European Journal of PlasticSurgery, it is possible that we underestimated the true publication rate.3 This did,however, allow us to compare our observations with those reported by others usingsimilar methodology.Our overall publication rate compares favourably with those reported in other sur-

45

gical disciplines (Table III), and it is significantly higher than the weighted meanrate of 41 percent to 45 percent calculated in reviews of more than 45 reports onfull publication of abstracts related to all biomedical sciences.7,8 Even when thesearch results from European Journal of Plastic Surgery are excluded, the overallpublication rate for the five consecutive EURPAS meetings is still 59 percent. Ob-viously, EURAPS succeeds at fulfilling its mission to furnish an annual forum for aselection of the best scientific works presented at national societies, and this maybe expected to result in a higher publication rate than that of the national meet-ings.Still, one wonders what happened to the 111 presentations that have not beenpublished to date. The reasons for failure to publish are multifactorial.8 Eventhough “lack of time or low priority” has been indicated by most authors, 11,20,25 “in-completeness of study” or “problem in study design,”20 “lack to resubmit after initialrejection of the manuscript,”4 and even “lack of economic reward”21 have beencited as reasons for such failure.8 Another factor of influence may be the publica-tion bias against studies with small sample sizes and studies with negative or non-significant findings.3,8,20,25Because we, like others,8,11 observed a higher publication rate for “basic research”presentations, we surmise that basic researchers have more incentive to completetheir work15 and their studies are less likely to meet with publication bias.11 Anotherfactor that may influence rate of subsequent publication is the amount of data re-quired by societies for abstract submission. The review process for accepting apaper for presentation at a meeting is often markedly different from that for amanuscript being considered for publication in an indexed journal,26 and it is pos-sible that more rigorous requirements lead to greater likelihood of eventual fullpublication.21 As such, the high publication rate we observed not only reflects thedrive and academic determination of plastic surgeons to complete and publish theirstudies but, in addition, it validates the quality of the EURAPS annual meetingsand the selection process for the abstracts by its consecutive program committees.4,27,28

Only one of three submitted abstracts is accepted for podium presentation, and at-tending a EURAPS meeting implies being exposed to a large number of studiesthat most likely will be published in written form over the next few years.1 Giventhat nearly twothirds of the presentations in our study are expected to be followedby full publication and because our selection criteria for inclusion of these publi-cations were as stringent as those reported by other authors,8 we conclude thatthe overall scientific value of the presentations at the EURAPS annual meetingsmay be judged as more than adequate.2,3 The nearly identical publication ratesover the 5-year period suggests that there is ample quality research to be pre-sented at these meetings.16

Date of PublicationInitially, we expected the vast majority of publications to appear in the first 3 yearsafter presentation.7,8,17 Hence, we chose to study abstracts presented up to the1999 EURAPS meeting to ensure that a reasonable amount of time would have

46

elapsed to allow such publication. From 1996 to 1997, however, we noticed amarked drop in publication rate from 0.72 to 0.58 (Table II) and we attribute thisto insufficient time having passed between the meeting and the evaluation periodto cover the time lag between presentation and publication.16Because only two presentations from the 1995 and 1996 meetings were publishedmore than 6 years after the meeting, we accept the mean of the publication ratesof these two meetings (81 of 117, or 0.69) as indicative for the years to follow.3Hence, the total number of presentations at the 1997, 1998, and 1999 meetingsto be published at any time after those meetings are expected to be 42, 42, and48, respectively. Therefore, we expect seven more presentations from the 1997meeting to be published in the future and three and eight more of the 1998 and1999 presentations, respectively. Considering the relatively large number of “basicresearch” papers at the 1998 meeting, this may even be an underestimation forthat year.

Early PublicationBecause some authors prefer to postpone public presentation of their work untilit is “publication-ready,”3 we reckoned with publications in the year before the re-spective meetings. Still, the premeeting publication rate of 13 percent in our studyis higher than the premeeting publication rates of 1 percent to 9 percent observedfor other surgical meetings (Table IV).2,3,11,15–19 This may explain why we foundabout half of all published presentations in our study to have appeared before thefirst year after the respective meetings had passed, whereas the weighted meanpublication rate after the first year for the other surgical meetings was approxi-mately one-third of the final publication rate (Table III).We believe that the 11 presentations we found to have appeared in print morethan 1 year before the meeting ought to be regarded as redundant. Even thoughno attempt was made to question these authors as to why they chose to presentat the EURAPS at such a late date,16 it is widely understood that presentations ata meeting hardly serve a purpose once the full paper has been published.5

Grade of PublicationThe publications ultimately appeared in a wide variety of journals, but it is clearthat Plastic and Reconstructive Surgery is the best source for information presentedat EURAPS meetings (Table V). This reflects the interest of both the EURAPS andPlastic and Reconstructive Surgery in presenting the latest plastic surgical re-search.16 That we found 152 of the 194 publications in the 16 journals related toplastic surgery confirms that most research presented at a specialty meeting issubsequently published in a specialty journal.5,11 An innovation in our study was theuse of Institute for Scientific Information impact factors to grade the overall sci-entific value of the published presentations. The use of this impact factor as agrading instrument for scientific output has been fiercely condemned,13,24,29 buteven its opponents accept it as a reliable instrument to compare journals within agiven research field or discipline.23,24,29 Merely counting the number of citations ofa published paper without consideration of the journal in which it was published

47

would not allow for a standardized comparison.There is a significant association between a journal’s impact factor and its qual-ity,30,31 and articles of higher methodological quality are published in journals whosearticles are cited more frequently.30 The journal’s impact factor was recently foundto be the only variable of influence on the citations per year of an article in thatjournal. Other characteristics such as number of studied subjects, human versusanimal subjects or objects, binary variables representing presence or lack of an ex-plicit hypothesis, control group, blinding, prospective or retrospective methodology,and the subjective newsworthiness and quality scores either had no influence orwere almost completely subsumed by the impact factor of the publishing journal.6The impact factor is a predictor of the number of articles from that journal that willachieve classic status in the future.32 Provided the aggregation level of the studiedmaterial is sufficiently high, the impact factor is well applicable as a bibliometric in-dicator of research performance in biomedical sciences.23 The joint scientific outputof five consecutive annual EURAPS meetings may be considered an adequate ag-gregation level. Still, just as a journal’s impact factor is an indicator of the journal’sscientific value rather than of the value of an individual article in that journal, theaverage of the normalized factors we used is an indicator of the meeting’s scientificvalue rather than the value of the individual presentations.The bibliometric data on the presentations awarded as best papers at the consec-utive EURAPS annual meetings illustrate that full publication in a peer-reviewedjournal may not be used as the definitive standard for the scientific value of indi-vidual presentation (Table VII).28 Whereas two of the 10 awarded presentationshave not been published to date, the (normalized) impact factors for five of theeight remaining awarded presentations did not exceed the mean (normalized) im-pact factor of publications from the same meeting (Table VI). Obviously, authorshave reasons other than the impact factor for choosing the journal in which theypublish their work.32,33

A greater than average number of papers presented at the 1995 through 1999annual EURAPS meetings went on to full publication in peer-reviewed journals.Among these journals, Plastic and Reconstructive Surgery was the best source forinformation presented at the meetings. Although approximately 90 percent of thepublications appeared before 3 years had passed after a meeting, further publica-

48

tions may be expected to appear even more than 6 years after the meeting. Giventhe high publication rate and the high average normalized impact factor of thejournals in which the presentations appeared, the five studied EURAPS meetingsoverall had high scientific value.

49

1. Wang, J.C., Yoo, S., and Delamarter, R.B. The publication rates of presenta-tions at major spine speciality society meetings (NASS, SRS, ISSLS). Spine24: 425, 1999.

2. Daluiski, A., Kuhns, C.A., Jackson, K.R., and Liebermanm, J.R. Publicationrate of abstracts presented at the annual meeting of the Orthopaedic Re-search Society. J Orthop Res 16: 645, 1998.

3. Bhandari, M., Devereaux, P.J., Guyatt, G.H., et al. An observational study oforthopaedic abstracts and subsequent full-text publications. J Bone JointSurg 84A: 615, 2002.

4. Helfet, D.L. The scientific sessions at the OTA annual meeting - Editorial.J Orthop Trauma 12: 456, 1998.

5. Castillo, J., Garcia-Guasch, R., and Cifuentes, I. Fate of abstracts from theParis 1995 European Society of Anaesthesiologists meeting. Eur J Anaesthe-siol 19: 888, 2002.

6. Callaham, M., Wears, R.L., and Weber, E. Journal prestige, publication bias,and other characteristics associated with citation of published studies inpeer-reviewed journals. JAMA 287: 2847, 2002.

7. von Elm, B.J., Poglia, G., Walder, B. and Tramèr, M.R. Fate of abstracts sub-mitted to biomedical meetings: A systemic review.http://www.amaassn.org/public/peer/prc_program2001.htm (accessed April2003).

8. Scherer, R.W. and Langenberg, P. Full publication of results initially pre-sented in abstracts (Cochrane methodology review). In: The Cochrane Li-brary, Issue1, 2003. Oxford: Update Software, 2003.

9. http://www.ncbi.nim.nih.gov/PubMed/ (accessed February 2003).10. http://link.springer.de/link/service/journals/00238/index.h tm (accessed

February 2003).11. Hamlet, W.P., Fletcher, A., and Meals, R.A. Publication patterns of papers

presented at the annual meeting of the American Academy of OrthopaedicSurgeons. J Bone Joint Surg 79A: 1138, 1997.

12. Dellon, A.L., Scally, A.L., and Kuhlemeier, K.V. Impact of the journal Plasticand Reconstructive Surgery. Plast Reconstr Surg 90: 717, 1992.

13. Hecht, F., Hecht, B.K., and Sandberg, A.A. The journal "Impact Factor": Amisnamed, misleading, misused measure. Cancer Genet Cytogenet 104: 77,1998.

14. Dudley, H.A.F. Surgical research: Master and servant. Am J Surg 135: 458,1978.

15. Gavazza, J.B., Foulkes, G.D., and Meals, R.A. Publication pattern of paperspresented at the American Society of Surgery of the Hand annual meeting.J Hand Surg 21A: 742, 1996.

16. Nguyen, V., Tornetta III, P., and Bkaric, M. Publication rates for the scientificsessions of the OTA. J Orthop Trauma 12: 457, 1998.

50

17. Murrey, D.B., Wright, R.W., Seiler, J.G.III, Day, T.E., and Schwartz, H.S. Pub-lication rates of abstracts presented at the 1993 annual academy meeting.Clin Orthop 359: 247, 1999.

18. Bowrey, D.J., Morris-Stiff, G.J., Clark, G.W.B., Carey, P.D., and Mansel, R.E.Peer-reviewed publication following presentation at a regional surgicalmeeting. Med Education 33: 212, 1999.

19. Jackson, K.R., Daluiski, A., and Kay, R.M. Publication of abstracts submittedto the annual meeting of the Pediatric Orthopaedic Society of North Amer-ica. J Ped Orthop 20: 2, 2000.

20. Scherer, R.W., Dickersin, K., and Langenberg, P. Full publication of resultsinitially presented in abstracts - A meta-analysis. JAMA 272: 158, 1994.

21. Marx, W.F., Cloft, H.J., Do, H.M., and Kallmes, D.F. The fate of neuroradio-logic abstracts presented at national meetings in 1993: Rate of subsequentpublication in peer-reviewed, indexed journals. Am J Neuroradiol 20: 1173,1999.

22. Shubert, A. and Braun, T. Cross-field normalization of scientometric indica-tors. Scientometrics 36: 311, 1996.

23. van Raan, A. J. F. Advanced bibliometric methods as quantitative core ofpeer review based evaluation and foresight exercises. Scientometrics 36:379, 1996.

24. Seglen, P. O. Why the impact factor of journals should not be used for eval-uating research. Br. Med. J. 314: 498, 1997.

25. Callaham, M. L., Wears, R. L., Weber, E. J., Barton, C., and Young, G. Posi-tive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. J.A.M.A. 280: 254, 1998.

26. Rubin, H. R., Redelmeier, D. A., Wu, A. W., and Steinberg, E. P. How reliableis peer review of scientific abstracts? Looking back at the 1991 AnnualMeeting of the Society of General Internal Medicine. J. Gen. Intern. Med. 8:255, 1993.

27. van der Steen, L. P. E., Hage, J. J., Kon, M., and Mazzola, R. Reliability of astructured method of selecting abstracts for a plastic surgical scientificmeeting. Plast. Reconstr. Surg. 111: 2215, 2003.

28. van der Steen, L. P. E., Hage, J. J., Kon, M., and Monstrey, S. J. Validity of astructured method of selecting abstracts for a plastic surgical scientificmeeting. Plast. Reconstr. Surg. 113: 353, 2004.

29. Neuberger, J., and Councell, C. Impact factors: Uses and abuses. Eur. J.Gastroenterol. Hepatol. 14: 209, 2002.

30. Lee, K. P., Schotland, M., Bacchetti, P., and Bero, L. A. Association of journalquality indicators with methodological quality of clinical research articles.J.A.M.A. 287: 2805, 2002.

31. Saha, S., Saint, S., and Christakis, D. A. Impact factor: A valid measure ofjournal quality?J. Med. Libr. Assoc. 91: 42, 2003.

32. Garfield, E. The quality and influence of JAMA. J.A.M.A. 259: 1946, 1988.33. Frank, E. Authors' criteria for selecting journals. J.A.M.A. 272: 163, 1994.

51