is an article in a top journal a top article?

17
Financial Management • Winter 2004 • pages 133 - 149 Is an Article in a Top Journal a Top Article? Stanley D. Smith* This study ranks 15 leading finance journals by the average number of Social Sciences Citation Index cites per articles for articles published in 1996. It also defines a “top article,” compared to an “article in a top journal.” Using different criteria for top articles, I examine the Type I error (a “top” article is rejected by a particular decision rule, e.g., in top three journals) and the Type II error (a “non-top” article is accepted as a top article) for each journal and combinations of the journals. Due to the high error rates, the results suggest that identifying top articles requires looking beyond the Top 3 journals, as well as examining each article more carefully for its intrinsic quality. The author thanks Pat Fishe, Melissa Frye and Drew Winters for very useful comments. The article benefited greatly from comments by Lemma Senbet and Alex Triantis (the Editors) and two anonymous referees. * Stanley D. Smith is the SunTrust Chair of Banking and a Professor of Finance at the University of Central Florida in Orlando, FL. A colleague was recently told by his dean that to be promoted to professor, he needed to have more publications in the “top three” finance journals. Another colleague, an assistant professor at another university, was told that only articles in the top three journals would count towards his promotion and tenure decision. But the top three journals were not the same in these two cases. Assistant professors are often told that they need publications in the top three journals or “top N” journals. The purpose of this article is to investigate whether a “top N journals” approach provides a reasonable decision rule in terms of identifying top articles in the finance literature. I use Type I and II errors to evaluate the accuracy of these decision rules. I define a Type I error as one when a “top” article is rejected by the decision rule. I call this a Reject Top Article (RTA) error. I define a Type II error as one when a “non-top” article is accepted as a top article. I call Type II errors an Accept Non-Top Article (ANTA) error. In this study, I rank 15 leading finance journals by the average number of Social Sciences Citation Index (SSCI) cites per articles from 1996 to January 2004 for articles published in those 15 journals in 1996. Recent studies often use Journal Citation Reports (JCR) impact factors to rank journals. The JCR impact factor for a journal is the number of cites in year t for articles published in years t-1 and t-2 divided by the number of articles published in years t-1 and t-2. Given the possibly long life over which an article can be cited, as measured by citations the impact factor is a relatively short-term measure of the impact. Although I use one year of publications, 1996, the number of citations is over a much longer period than the JCR impact factors. I also define a “top article” (the average number of cites is above the median, mean, 90 th percentile, or 95 th percentile for a set of leading finance journals) as opposed to an “article in a top journal.” The results show that using the top three (JF, JFE, RFS) approach to identify top articles (based on the mean number of cites per article) leads to an RTA error 44% of the time and an ANTA error 33% of the time. I provide similar information for the 15 individual journals and for different levels of citations. I find that at the 90th and 95th percentiles, the RTA error decreases but the ANTA error increases dramatically. I examine the trade-off between these two errors. My

Upload: independent

Post on 25-Nov-2023

0 views

Category:

Documents


0 download

TRANSCRIPT

Financial Management • Winter 2004 • pages 133 - 149

Is an Article in a Top Journal aTop Article?

Stanley D. Smith*

This study ranks 15 leading finance journals by the average number of Social SciencesCitation Index cites per articles for articles published in 1996. It also defines a “top article,”compared to an “article in a top journal.” Using different criteria for top articles, I examinethe Type I error (a “top” article is rejected by a particular decision rule, e.g., in top threejournals) and the Type II error (a “non-top” article is accepted as a top article) for eachjournal and combinations of the journals. Due to the high error rates, the results suggest thatidentifying top articles requires looking beyond the Top 3 journals, as well as examining eacharticle more carefully for its intrinsic quality.

The author thanks Pat Fishe, Melissa Frye and Drew Winters for very useful comments. The article benefited greatlyfrom comments by Lemma Senbet and Alex Triantis (the Editors) and two anonymous referees.*Stanley D. Smith is the SunTrust Chair of Banking and a Professor of Finance at the University of Central Florida inOrlando, FL.

A colleague was recently told by his dean that to be promoted to professor, he needed to havemore publications in the “top three” finance journals. Another colleague, an assistant professorat another university, was told that only articles in the top three journals would count towardshis promotion and tenure decision. But the top three journals were not the same in these twocases. Assistant professors are often told that they need publications in the top three journalsor “top N” journals.

The purpose of this article is to investigate whether a “top N journals” approach provides areasonable decision rule in terms of identifying top articles in the finance literature. I use TypeI and II errors to evaluate the accuracy of these decision rules. I define a Type I error as onewhen a “top” article is rejected by the decision rule. I call this a Reject Top Article (RTA) error.I define a Type II error as one when a “non-top” article is accepted as a top article. I call Type IIerrors an Accept Non-Top Article (ANTA) error.

In this study, I rank 15 leading finance journals by the average number of Social SciencesCitation Index (SSCI) cites per articles from 1996 to January 2004 for articles published in those 15journals in 1996. Recent studies often use Journal Citation Reports (JCR) impact factors to rankjournals. The JCR impact factor for a journal is the number of cites in year t for articles publishedin years t-1 and t-2 divided by the number of articles published in years t-1 and t-2. Given thepossibly long life over which an article can be cited, as measured by citations the impact factor isa relatively short-term measure of the impact. Although I use one year of publications, 1996, thenumber of citations is over a much longer period than the JCR impact factors.

I also define a “top article” (the average number of cites is above the median, mean, 90th

percentile, or 95th percentile for a set of leading finance journals) as opposed to an “article in atop journal.” The results show that using the top three (JF, JFE, RFS) approach to identify toparticles (based on the mean number of cites per article) leads to an RTA error 44% of the time andan ANTA error 33% of the time. I provide similar information for the 15 individual journals andfor different levels of citations. I find that at the 90th and 95th percentiles, the RTA error decreasesbut the ANTA error increases dramatically. I examine the trade-off between these two errors. My

Financial Management • Winter 2004134

results demonstrate that if someone is interested in identifying top articles then the RTA andANTA errors associated with a top three journals approach suggest that one should look ata broader set of journals and should examine each article more closely for its intrinsic quality,rather than the general quality of its journal.

The article is organized as follows. Section I discusses the literature and method. SectionII presents the results. Section III summarizes and concludes.

I. Literature and Method

I examine the 5,979 SSCI cites (as collected on February 7-12, 2004) for 626 articles publishedin 1996 in 15 leading finance journals. The 15 journals include 11 of a core 16 set of journals,as defined and studied by Chan, Chen, and Steiner (2002) and four other journals included inother similar studies.

Chan et al. (2002) use Journal of Finance-equivalent sized pages in their 16 finance journalsfrom 1990 to 2001 to rank finance programs. The set of 11 journals from their set of core 16journals comprises Journal of Finance (JF), Journal of Financial Economics (JFE), Reviewof Financial Studies (RFS), Journal of Financial and Quantitative Analysis (JFQA), Journalof Business (JB), Journal of Financial Intermediation (JFI), Journal of International Moneyand Finance (JIMF), Financial Management (FM), Journal of Banking and Finance (JBF),Journal of Futures Markets (JFM), and Journal of Portfolio Management (JPM). I do notinclude Chan et al.’s other journals [Financial Analysts Journal (FAJ), Journal of FinancialServices Research (JFSR), Journal of Financial Research (JFR), Journal of Business Finance& Accounting (JBFA), and Financial Review (FR)] because these journals are not coveredor fully covered by the SSCI during the sample period.

Chan et al. (2002) also rank programs based on the top three of JF, JFE and RFS. They alsoexamine four other journals covered by the SSCI, Journal of Money Credit and Banking(JMCB), Journal of Risk and Insurance (JRI), Real Estate Economics (REE), and Journal ofReal Estate Finance and Economics (JREFE). These journals examine a broad range offinance areas that consists of money and banking, real estate finance, and insurance.

I note here that the definition of the top three may vary by time and measure. For example,Niemi (1987) uses total research productivity (pages) in the top three of JF, JFE, and JFQAfrom 1975 to 1986 to rank programs. Chung, Cox, and Mitchell (2001) also use citations in JF,JFE and JFQA to identify the most often cited authors from 1974 to 1998. Chung et al. (2001)note that they exclude RFS because of its shorter existence.

Alexander and Mabry (1994) use citations in the four leading finance journals (JF, JFE,JFQA and RFS) from January 1987 to March 1991 to rank journals. Their top three comprisesJFE, JF, and JB, with JFQA in fourth place.

Arnold, Butler, Crack, and Altintig (2003) rank finance journals according to number ofcitations in JF, JFE, RFS, JB, JFQA, and FM during 1990-1999. They also rank journals bynumber of important papers, number of recent important papers, and an impact factor. The topfive journals in all rankings are JF, JFE, RFS, JB and JFQA. However, the rankings varied withdifferent measures. FM was in ninth place after Econometrica, Journal of Political Economy,and American Economic Review. The literature generally seems to agree on a top two of JF andJFE, but third place seems to be a tie between several candidates, RFS, JFQA, and JB.

Although the literature on finance journal quality most often uses citations as the bestapproach, they may suffer from an inherent bias, such as self-citing. These studies alsocannot identify the perspective of individuals who might have different research interests or

Smith • Is an Article in a Top Journal a Top Article? 135

come from different geographic areas.There are two approaches other than citations that can be used to rate journals or

departments. One is to survey department chairs (e.g., Coe and Weinstock, 1983; and Borde,Cheney, and Madura, 1999), or to survey faculty (e.g., Oltheten, Theoharakis and Travlos, 2003;and Christoffersen, Englander, Arize, and Malindretos, 2001). The second approach is to rankdepartments on the basis of memberships on journal editorial boards (Chan and Fok, 2003).

I select 1996 because it gives a recent time frame of seven to eight years. This or an evenshorter time period often equates with many evaluation situations, such as promotion andtenure decisions or filling new positions within a department. More recent records can alsobe used for public relations purposes to illustrate the quality of an academic program toalumni and potential students and donors. Chung et al. (2001) show that the number ofcitations for articles cited in JF, JFE, and JFQA between 1974 and 1998 increases sharplyduring the first three years after publication, reaches a peak during the fourth year, and thendeclines gradually after that. My sample period should contain the peak period for citations.However, to the extent that citations occur for these articles in later years their total citationrecords will be understated. For example, in 2002 the JCR’s cited half-lives in years for thesample of journals are: JFE, JFQA, and JB (>10), JF (9.3), FM (9.2), JRI (9.1), JFM (8.5), JPM(7.9), RFS (7.8), JMCB (7.7), JIMF (7.3), JBF (6.5), JREFE (6.3), JFI (5.8), and REE (5.0). JCRdefines the cited half-life as the number of years after the publication year, which account for50% of citations received for an article.

Chung et al. (2001) identify the most cited authors from 1974 to 1998 using citations in JF,JFE, and JFQA. Chan (2001) examines citation data for 1998 and 1999 from JF, JFE, JFQA, andRFS to rank journals by citation proportions over several time lags.

In this article, I use a broader set of citations, the SSCI, which allows me to determine therelative influence of more specialized journals in areas such as banking, real estate, andinsurance. These areas are less likely to be reflected in the top journals of JF, JFE, JFQA andRFS. For example, of the 15 journals I consider in this article, JMCB, JRI and REE rank in the5th, 10th, and 13th places, respectively, but in Chan’s study JMCB is tied for 15th place withJournal of Empirical Finance and Journal of Financial Research, JRI is not ranked at all,and REE is in 27th place. These differences highlight the segmentation of these importantareas and the lack of their recognition in the top three or four journals.

The SSCI cites approach contains cites in closely related fields of economics, accountingand other areas of business and law. The JCR impact factors that are often used to rankjournals includes cites in these areas. Further, Borokhovich, Bricker, and Simkins (2000) findthat approximately one-third of all citations to articles in JF and JFE come from journalsoutside finance, especially economics journals with 15.3% for JF and 19.4% for JFE. Chan(2001) examines the journals cited in the 1998 and 1999 issues of JF, JFE, RFS, and JFQA. Interms of the percentage of citations in this set of journals, 67.73, 18.73, 8.55, 2.72 and 2.26%were from finance, economics, statistics, accounting, and miscellaneous (business, law andregulator) journals, respectively. These two studies illustrate the importance of non-financejournals to the finance literature and vice versa, and support the usage of a broader set ofcitations to fully measure an article’s impact on the literature.

Although the SSCI cites approach reflects a broader set of journals and fields than a topthree or four approach, it does not include all cites. For example, in my study, I do not includesome or all of the cites in other finance journals such as FAJ, JFSR, JFR, JBFA, FR, Journalof Real Estate Research (JRER), and Journal of Financial Markets (JFINMKT) becausethey are either not fully covered or covered at all by the SSCI during my sample period.

Another example of the differences in these areas is a 1995 survey of FMA members on

Financial Management • Winter 2004136

rankings of journals in subfields by Christoffersen et al. (2001). They find the top fourjournals in corporate finance are JF, JFE, JFQA, and FM; the top two journals in financialinstitutions and markets are JBF and JMCB; the top journal in finance and insurance is JRI;the top journal in international finance is JIMF; the top three journals in investments areJPM, JF, and FAJ; and the top three journals in real estate finance are JREFE, REE, and JRER.

Using a global faculty survey, Oltheten et al. (2003) find that the research interest of the respondents(corporate finance, investments and derivatives, financial institutions, and international finance,institutions and markets), geographic location (North America, Europe, Asia, and Australia/NewZealand), seniority, and affiliation with a journal affects the quality perceptions.

Krishnan and Bricker (2004) attempt to separate the citation performance of an article forthe year of publication and the next two years between the quality of the article and thevalue added by the journal. They use the author reputation and the school reputation asproxies for the quality of the article. They use journal age, editorial board quality, andreadership characteristics as proxies for the value added by the journal. In their analysis ofarticles published from 1990 to 1998 in JF, JFE, RFS, JFQA, and JB, they find that JF, JFE andRFS add significant value in terms of citations over and above inherent article quality. Forexample, in their Table V, they find that after controlling for the article quality for the overallsample period, JF, RFS, JFE, JB, and JFQA add 4.1655, 3.1805, 2.8590, 0.4142 and 0.3967 SSCIcites, respectively. Only the coefficients for JF, RFS, and JFE are statistically significant.

From this brief review, it is clear that the literature related to the quality of journals has takenmany approaches. The rankings for the survey studies, particularly for the highest rankedjournals, are generally consistent with the rankings that use the citations-based measures.

As have past authors in this literature, I recognize the limitations of any one approach todetermining the quality of a journal. Even with objective measures such as citations, thereare limitations beyond the different citation measures that I can use. I consider author self-citations in my analysis.

I also note that a citation may be a “correction” citation, but in this case the citation maybe more the reflection of a poor article, rather than a top article. To define a citation as acorrection requires a subjective judgment by the citing author. Many articles extend otherstudies in the literature, which can be construed as correcting previous works or building onthose earlier works. My experience suggests that citations are based on building on earlierresearch. In addition, an objective measure of a correction cite is difficult to define.

An article can be categorized by type of article, e.g., survey, empirical, theoretical, orsubject area. The articles I cite in this study are not categorized by type of article. To theextent that a reviewer of an author’s publication and citation record values thesecharacteristics in different ways, then the value assigned to the citations associated with anarticle will vary by reviewer.

Another issue is the “hot topic” area. Suppose I could objectively define a hot topic areaand that I could identify the time frame for the topic. The number of article citations wouldthen appear to be positively related to how early in the sample’s time frame the articleappeared, the contribution of the article, and the number of articles published in the hottopic area. These factors are difficult to objectively define and separate. And again, basedon these factors a reviewer may attach different values to the citations.

II. Results

Table I presents a comparison of 5,979 SSCI cites (as collected on February 7-12, 2004) for

Smith • Is an Article in a Top Journal a Top Article? 137T

ab

le I. D

istr

ibu

tio

n o

f S

ocia

l S

cie

nces C

itati

on

In

dex (

SS

CI)

Cit

es f

or 1

996 J

ou

rnal A

rtic

les (

Co

llecte

d o

n F

eb

ru

ary 7

-12, 2004)

The

15 jo

urna

ls se

lect

ed fo

r cita

tion

anal

yses

are

Jour

nal o

f Fin

ance

(JF)

, Jou

rnal

of F

inan

cial

Eco

nom

ics (

JFE)

, Rev

iew

of F

inan

cial

Stu

dies

(RFS

), Jo

urna

l of

Fina

ncia

l and

Qua

ntita

tive

Ana

lysis

(JF

QA

), Jo

urna

l of

Mon

ey C

redi

t and

Ban

king

(JM

CB),

Jour

nal o

f Bu

sines

s (J

B), J

ourn

al o

f Fi

nanc

ial I

nter

med

iatio

n (J

FI),

Jour

nal o

f In

tern

atio

nal M

oney

and

Fin

ance

(JI

MF)

, Jou

rnal

of

Bank

ing

and

Fina

nce

(JBF

), Jo

urna

l of

Risk

and

Ins

uran

ce (

JRI),

Jou

rnal

of

Futu

res

Mar

kets

(JFM

), Fi

nanc

ial M

anag

emen

t (FM

), Re

al E

state

Eco

nom

ics (

REE)

, Jou

rnal

of R

eal E

state

Fin

ance

and

Eco

nom

ics (

JREF

E), a

nd Jo

urna

l of P

ortfo

lio

Man

agem

ent (

JPM

). Th

ey a

re so

rted

from

hig

h to

low

by

the

mea

n SS

CI c

ites d

urin

g th

e sa

mpl

e pe

riod.

In th

e bo

ttom

two

row

s, su

mm

ary

cita

tion

statis

tics a

re

prov

ided

for 6

26 a

rticl

es in

the

15 jo

urna

ls (la

bele

d as

ALL

15)

and

the

626

artic

les w

ith se

lf-ci

tes e

xclu

ded

(labe

led

as A

LL 1

5 W

ithou

t Sel

f-Cite

s). S

elf-c

ites

are

defin

ed a

s any

cite

whe

re o

ne o

r mor

e of

the

cite

d au

thor

s is o

ne o

f the

citi

ng a

utho

rs.

Jo

urn

al

Min

. 10

%

20

%

30

%

40

%

50

%

60

%

70

%

80

%

90

%

Max.

Mean

Std

.

Erro

r

No

. o

f

Arti

cle

s

%

Au

tho

r

Self

-

Cit

es

Mean

Wit

ho

ut

Self

-

Cit

es

Me

dia

n

Wit

ho

ut

Self

-

Cit

es

JF

0 3

6 9

13

19

24

27

38

54

158

24.3

2.

95

69

6.9

22.6

18

JF

E 1

6 8

10

13

18

20

27

34

45

95

23.1

2.

98

47

6.1

21.7

18

RF

S 0

3 4

5 8

11

12

18

22

39

77

16.0

2.

93

37

9.8

14.4

10

JF

QA

0

2 3

5 7

7 10

14

26

32

53

13

.2

2.55

29

5.

2 12

.5

7 JM

CB

0 1

2 3

4 6

7 13

15

25

86

10

.4

1.88

55

8.

4 9.

5 5

JB

1 1

2 5

8 9

9 13

15

21

34

9.

8 1.

88

19

13.3

8.

5 7

JFI

0 4

4 4

5 5

7 8

15

18

22

8.2

1.62

15

8.

2 7.

5 5

JIM

F 0

1 1

1 3

3 4

6 7

12

66

5.9

1.40

52

9.

9 5.

3 3

JBF

0 1

1 1

2 2

3 4

6 12

30

4.

6 0.

58

89

11.1

4.

1 2

JRI

0 0

1 1

2 2

4 6

10

12

14

4.4

0.81

29

12

.4

3.9

2 JF

M

0 0

1 1

2 3

3 4

6 8

26

3.7

0.67

47

8.

7 3.

4 3

FM

0 0

0 1

1 2

3 5

8 9

12

3.5

0.67

32

6.

2 3.

3 2

REE

0 0

1 2

2 2

3 3

6 7

9 3.

0 0.

49

28

7.1

2.8

2 JR

EFE

0 1

1 1

1 1

2 3

4 5

19

2.6

0.58

34

22

.3

2.0

1 JP

M

0 0

0 0

0 1

1 1

2 3

11

1.3

0.33

44

4.

0 1.

2 1

All

15

0 0

1 2

3 4

6 9

14

24

158

9.55

0.

59

626

8.1

8.78

4

All

15

With

out

Self-

Cites

0 0

1 2

2 4

5 8

13

22

151

8.78

0.

56

626

Financial Management • Winter 2004138

626 articles published in 1996 in 15 leading finance journals. I sort the journals by the meannumber of cites per article. JF is at the top with 24.3 cites, followed closely by JFE with 23.1cites. RFS, JFQA, JMCB, and JB are next with 16, 13.2, 10.4, and 9.8 cites, respectively. I notethat if the journals are rated by the cites in the 50th percentile (median), JB is in fourth place,followed by JFQA and JMCB. These results are consistent with past studies, which suggeststhey are robust and generalizable to other years. However, I caution that rankings may varysome by measure and by year. This caution applies to all the studies on finance researchproductivity and rankings.

I use a test of differences of means with equal or unequal variances, given the situation, toperform a test of whether one journal’s mean is higher than another’s. I use this test for eachpair of this group of six journals. Using a 5% level of significance, I find that JF and JFE aresignificantly higher than the other four; RFS is not significantly higher than JFQA or JMCB butis significantly higher than JB; and JFQA, JMCB, and JB are not significantly higher than eachother. For all 15 journals, the median and mean average cites are four and 9.55, respectively.

Table I also provides information on the distribution of average cites. The table presentsstandard errors and more detailed information, from the minimum to the maximum in incrementsof every tenth percentile. Given the skewness of these distributions, more detailed informationon the distributions allows a better picture of each of the 15 journals. Although these resultsfor all 15 journals are not shown in the table, the percentage of articles with zero cites is 10%,with one or zero cites is 26%, with two or fewer cites is 37%, with three or fewer cites is 45%,and with four or fewer cites is 53%. For example, if this set of journals is truly representativeof the finance literature, then having more than four cites over the seven to eight years afterpublication places an article in the upper half of the distribution. If I use the mean of averagecites of 9.55 as a measure of central tendency, then ten or more cites places an article in thetop 29%. For the all 15 journals, if an article is going to place in the top 10%, then it must becited 25 or more times.

When researchers examine an article’s citation record, they often include information onself-cites. I define an article self-cite as a cite where one or more of the authors is an authorof the citing article. Author self-cites should not be confused with journal self-cites, whichare cites to articles in the same journal, e.g., Chan, Fok, and Pan (2000). Since the focus in mystudy is the article, I examine only author self-cites. The percentages of author self-cites forthe 15 journals vary from lows of 4% and 5.2% in JPM and JFQA, respectively, to highs of13.3% and 22.3% percent in JB and JREFE, respectively.

I also provide the means and medians without self-cites. The rankings by the means withoutself-cites are the same as the rankings of the means with self-cites. The medians for bothmeasures for all 15 journals, and for 11 of the 15 journals, are the same. Compared to themedians with self-cites, the medians without self-cites decrease by one for JF, RFS, and JMCB,and by two for JB. The rankings for the two median measures are similar. The bottom two rowsof Table I show the statistics for the two all-15 distributions. Using a paired two-sample meanst-test, the difference in means is significant at the 0.0001 level. However, except for the differencesin means of 0.77 cites, the distributions appear similar. If an article’s citations include self-cites, evaluating that record may partially or completely discount those cites.

Table II presents information for the 15 journals by the percentage of articles that meetcertain measures that could define a “top article.” In Table II, I use the measures of median,mean, 90th percentile, and 95th percentile for all 15 journals to define a top article. If I define atop article as one that has above the median of four cites per article, then the ANTA error isrerelatively low for JF (6%) and JFE (14%). JB, RFS, and JFQA are clustered around 26% to 28%,followed by JFI and JMCB at 33% and 47%, respectively. When I define a top article as being

Smith • Is an Article in a Top Journal a Top Article? 139

Journal

Journal % > All Median

(4 Cites)

Type II Error % (Accept

Non-Top 50%

Article)

Journal % > All Mean (9.55 Cites)

Type II Error % (Accept

Non-Above

Average Article)

Journal % > All 90%ile

(24 Cites)

Type II Error % (Accept

Non-Top 10%

Article)

Journal % > All 95%ile

(35 Cites)

Type II Error % (Accept

Non-Top 5% Article)

JF 86 14 70 30 35 65 23 77 JFE 94 6 74 26 38 62 17 83 RFS 73 27 54 46 16 84 11 89 JFQA 72 28 45 55 21 79 7 93 JMCB 53 47 36 64 11 89 5 95 JB 74 26 37 63 5 95 0 100 JFI 67 33 27 73 0 100 0 100 JIMF 38 62 15 85 4 96 2 98 JBF 30 70 17 83 1 99 0 100 JRI 34 66 21 79 0 100 0 100 JFM 28 72 6 94 2 98 0 100 FM 31 69 9 91 0 100 0 100 REE 21 79 0 100 0 100 0 100 JREFE 15 85 3 97 0 100 0 100 JPM 4 96 2 98 0 100 0 100 All 15 (N = 626 Articles)

47 53 29 71 10 90 5 95

Table II. Type II Error is Accept a Non-Top Article as a Top Article When theArticle Does Not Meet the Category Criterion (ANTA)

The 15 journals selected are Journal of Finance (JF), Journal of Financial Economics (JFE), Review ofFinancial Studies (RFS), Journal of Financial and Quantitative Analysis (JFQA), Journal of Money Creditand Banking (JMCB), Journal of Business (JB), Journal of Financial Intermediation (JFI), Journal ofInternational Money and Finance (JIMF), Journal of Banking and Finance (JBF), Journal of Risk andInsurance (JRI), Journal of Futures Markets (JFM), Financial Management (FM), Real Estate Economics(REE), Journal of Real Estate Finance and Economics (JREFE), and Journal of Portfolio Management(JPM). The last row includes summary statistics for 626 articles in the 15 journals (labeled as ALL 15).The four criteria for a Type II Error are the percentage of articles in the identified journal not having citesabove: 1) the median of 4 cites, 2) the mean of 9.55 cites, 3) the 90th percentile of 24 cites, and 4) the 95thpercentile of 35 cites. The criteria measures are based on the statistics for 626 articles in the 15 journals.

above the All 15 mean of 9.55 cites per article (also the top 29% of All 15 articles, asshown in the last row), the error of accepting a non-top article increases for the top sixjournals. For JF and JFE, the error rates are 30% and 26%, respectively, followed by RFSat 46% and JFQA at 55%. The rest of the journals range between 63% and 100%. WhenI define a top article as being above the 90th percentile of 24 cites per article, the error ofaccepting a non-top article increases dramatically even for the top two journals, JF andJFE with errors of 65% and 62%, respectively, followed by JFQA and RFS with error ratesof 79% and 84%, respectively. The error rates for the other journals range from 89% to100%. When I use above the 95th percentile of 35 cites per article, the error of acceptinga non-top article is very high, even for the top two journals, with errors of 77% and 83%,respectively. RFS, JFQA, JMCB, and JIMF are very high at 89%, 93%, 95%, and 98%,

Financial Management • Winter 2004140

respectively. The error rates for the other journals are 100%.The ANTA error can be viewed on an individual journal basis, as in Table II, or on a

cumulative top N journals basis as in Table III and Figure 1. Table III examines the ANTAerrors on a cumulative basis, i.e., as the number N in the “top N” journals increases. Figure1 shows the same information in graphic form.

When I use the median or mean as the measure of a top article, the results for the top fouror top seven journals are not much higher than for the top three journals. For example, theANTA error for the median measure is 15% for the top three journals and 17% and 25% forthe top four and top seven journals, respectively. The rate of increase is lower when I use the90th and 95th percentiles, because the errors start at higher levels.

Table IV examines the RTA errors on a cumulative basis, i.e., as the number N in the top Njournals increases. Figure 2 shows the same information in graphic form. Using above themedian to define a top article, the cumulative RTA error for the top three journals (JF, JFE,and RFS) is 56%. In other words, a decision rule of accepting only articles in these threejournals rejects 56% of the top articles in this category. A top five journals approach rejects39% of the top articles. A top ten journals (JF to JRI) approach reduces the RTA error to 12%.If I use above the mean to define a top article, the cumulative RTA error of rejecting a toparticle with a top three journals of JF, JFE, and RFS is 44%. A top five journals approachrejects 26% of the top articles. A top nine (JF to JBF) reduces the RTA error to 8%. Usingabove the 90th percentile as the definition of a top article, the cumulative RTA error of rejectinga top article with a top three journals of JF, JFE, and RFS is 26%. The error decreases to 8%with the addition of JFQA and JMCB. Using above the 95th percentile as the definition of atop article, the RTA error with the top three of JF, JFE, and RFS is 18%. The error decreasesto 12% when I add JFQA and to 3% when I add JMCB, JB, and JFI.

Figure 2 shows that as the definition of a top article increases from the median to the 95th

percentile, the initial slopes of the lines increase and the lines level off much faster.When I compare the effects of the Type I and II errors of a decision rule, I usually assign

a cost to each type of error. When I compare the ANTA and RTA errors, there does notappear to be a difference in the costs. If I wish to assign different costs to the two errors, Ishould weight the error or costs in some manner. I assume that the costs are equivalent, andso combine the ANTA and RTA errors and show them in Table V and Figure 3.

Across each row in Table V, I see that the combined error increases as the definitionincreases from median to 95th percentile. Looking down the four columns related to thedifferent definitions and Figure 3, I see that the combined errors reach a minimum, but thenseem to plateau. For example, for the above the median definition of a top article the combinederror decreases from 71% for a top three approach to a minimum of 49% for a top 12-13approach, and only increases to 53% for all 15 journals.

I also ask how well impact factors predict the future cites. With my sample, all 15 journalshave a JCR impact factor in each year from 1996 to 2002.

JCR defines an impact factor for a journal as the number of cites in year t for articles publishedin years t-1 and t-2 divided by the number of articles published in years t-1 and t-2. Thisdefinition is a relatively short-term measure of the impact as measured by citations, and doesnot necessarily predict citation performance. However, to the extent that people use impactfactors as measures of potential cites, they bestow a predictive power on the factors.

Table VI presents the JCR impact factors for each year, the average impact factors from1996 to 2002, and the mean actual cites for the sample period. Table VI also presents theestimated number of cites, based on a regression with the mean cites for each journal as thedependent variable and its average impact factor as the independent variable. The regression

Smith • Is an Article in a Top Journal a Top Article? 141

Journal

Top N

Journals

Cumulative

% Median

(4 Cites)

Cumulative

% Mean

(9.55 Cites)

Cumulative

% 90%ile

(24 Cites)

Cumulative

% 95%ile

(35 Cites)

JF 1 14 30 65 77 JFE 2 11 28 64 79 RFS 3 15 33 69 82 JFQA 4 17 36 70 84 JMCB 5 24 43 75 86 JB 6 24 44 76 87 JFI 7 25 46 77 88 JIMF 8 31 52 80 89 JBF 9 39 59 84 92 JRI 10 41 60 85 92 JFM 11 44 63 87 93 FM 12 45 65 88 93 REE 13 47 67 88 94 JREFE 14 49 69 89 94 JPM 15 53 71 90 95

Table III. Cumulative Type II Error (Accept a Non-Top Article as a TopArticle (ANTA) When the Article Does Not Meet the Category Criterion of

Top N Journals)

The journals [Journal of Finance (JF), Journal of Financial Economics (JFE), Review of Financial Studies(RFS), Journal of Financial and Quantitative Analysis (JFQA), Journal of Money Credit and Banking(JMCB), Journal of Business (JB), Journal of Financial Intermediation (JFI), Journal of InternationalMoney and Finance (JIMF), Journal of Banking and Finance (JBF), Journal of Risk and Insurance (JRI),Journal of Futures Markets (JFM), Financial Management (FM), Real Estate Economics (REE), Journalof Real Estate Finance and Economics (JREFE), and Journal of Portfolio Management (JPM)] are cumu-lated from the top down. The four criteria for a Type II Error are the percentage of articles in thecumulative set of journals not having cites above: 1) the median of 4 cites, 2) the mean of 9.55 cites, 3) the90th percentile of 24 cites, and 4) the 95th percentile of 35 cites which are based on the statistics for 626articles in the 15 journals. For example, for the Top 3 journals (JF, JFE, RFS), 33% of the articles in thosejournals did not have the cites above the mean of 9.55 cites.

has an adjusted R2 of 0.859 and is significant at the 0.001 level.There is little doubt that the mean number of journal cites is highly correlated with average

impact factors. However, the actual cites for some specific journals are significantly differentfrom the estimated cites. For example, JFQA’s actual mean cites are 185% of the estimatedcites and FM’s actual mean cites are 38% of the estimated cites. The standardized residualsfor JFQA and FM are outside the 95% confidence intervals for the estimates. Even with thehigh R2 for the regression model, researchers need to be careful about using it to predictcites for a specific journal.

I also note that the ranks differ between the average impact factor (IF) and the actual meancites. JFQA, JB, JRI, JFM, FM, and JREFE differ by at least two ranks. This example showshow similar measures can provide quite different ranks and provides another example of howdifficult it is to pick an “N set” of top journals.

I run a similar regression model, this time using the actual cites per article as the dependentvariable for the 626 articles in the 1996 issues of the 15 journals. The model has an adjusted

Financial Management • Winter 2004142

Figure 1. Cumulative ANTA (Accept a Non-Top Article as a Top Article) Error asNumber of Top N Journals Increases

There are four criteria for being a top article: Median (> 4 cites), Mean (> 9.55 cites), 90th %ile (> 24 cites),95th %ile (> 35 cites).

0 10 20 30 40 50 60 70 80 90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Top N Journals Added in the Following Order: JF, JFE, RFS, JFQA, JMCB, JB, JFI, JIMF, JBF, JRI, JFM, FM, REE,

JREFE, JPM.

Erro

r P

erc

en

t

90 %ile 95 %ile Mean Median

R2 of 0.261, which is significant at the 0.001 level. The estimated model is similar to theprevious regression model in that the estimated cites per article is equal to -0.188 + (9.7 XAverage Impact Factor). There is little doubt that the actual article cites are correlated withthe average impact factors. However, if I look at the results from another perspective, 73.9%of the variation is unexplained. These results indicate that with this sample there is a strongpositive overall relation between JCR impact factors and actual cites. However, again, theresearcher should be very careful about using an impact factor’s predictive ability for aspecific journal and even more careful about using an impact factor’s predictive ability for aspecific article.

I ask, “Can the total number of cites in lesser journals or cites in journals not included onthe list meet the impact test?” For example, one professor has two FM articles and each hasfive cites. Another professor has one RFS article with seven cites. Which one makes thelarger impact on the literature? I can make the case that the two FM articles have a largerimpact than does the one RFS article.

But I note that cites in some journals are not captured in the SSCI database. Shouldresearchers consider cites in journals not captured in the SSCI database? Other types ofarticles may not appear on the journal lists, such as articles in the publications of federalregulators such as regional Federal Reserve Banks or publications of major trade organizationssuch as the Appraisal Journal. These journals may not appear on a formal list of journals,but some of these articles may be cited in the SSCI database. Should the impact of thesearticles be considered?

Smith • Is an Article in a Top Journal a Top Article? 143

Journal Top N

Journals

Cumulative % > Median

(4 Cites)

Cumulative % > Mean

(9.55 Cites)

Cumulative % > 90%ile (24 Cites)

Cumulative % > 95%ile (35 Cites)

JF 1 80 74 63 53 JFE 2 65 55 35 29 RFS 3 56 44 26 18 JFQA 4 49 37 17 12 JMCB 5 39 26 8 3 JB 6 35 22 6 3 JFI 7 31 20 6 3 JIMF 8 25 16 3 0 JBF 9 15 8 2 0 JRI 10 12 4 2 0 JFM 11 8 3 0 0 FM 12 4 1 0 0 REE 13 2 1 0 0 JREFE 14 1 1 0 0 JPM 15 0 0 0 0

Table IV. Cumulative Type I Error: Reject a Top Article as a Non-TopArticle When the Article Meets the Category Criterion (RTA) But Is Not

Published in the Top N Journals

The journals [Journal of Finance (JF), Journal of Financial Economics (JFE), Review of Financial Studies(RFS), Journal of Financial and Quantitative Analysis (JFQA), Journal of Money Credit and Banking(JMCB), Journal of Business (JB), Journal of Financial Intermediation (JFI), Journal of InternationalMoney and Finance (JIMF), Journal of Banking and Finance (JBF), Journal of Risk and Insurance (JRI),Journal of Futures Markets (JFM), Financial Management (FM), Real Estate Economics (REE), Journalof Real Estate Finance and Economics (JREFE), and Journal of Portfolio Management (JPM)] are cumu-lated from the top down. The four criteria for a Type I Error are the percentage of articles not in thecumulative set of journals having cites above: 1) the median of 4 cites, 2) the mean of 9.55 cites, 3) the 90th

percentile of 24 cites, and 4) the 95th percentile of 35 cites which are based on the statistics for 626 articlesin the 15 journals. For example, for the Top 3 journals (JF, JFE, RFS), 44% of the articles that have citesabove the mean of 9.55 cites are not in those three journals.

To the extent that a school is more interested in the actual impact of a professor’s workthan the journals in which the articles appear, the school should ask for more detailedinformation. In addition to a regular publication list, an evaluator may want to see a list of thosejournals and articles that cite a person’s collection of works. Fishe (1998) examines suchinformation in his analysis of research standards for promotion to full professor at 90 USfinance departments ranked one to 96 in Borokhovich, Bricker, Brunarski, and Simkins (1995).Fishe (1998) shows that 26 of the 76 professors promoted to full professor at the financedepartments rated 21 to 96 did not publish in one of the top three (JF, JFE, RFS), but many ofthem did have numerous SSCI cites. The number of SSCI cites per year since Ph.D. degree forthe 76 professors ranged from 0.1 to 124.8 with a median of 7.0 and a 25th percentile of 3.9.

Fishe (1998) also provides the cumulative cites per year since Ph.D. degree for eachprofessor. I combine and reorganize his data to shed further light on the subject in my paper.For each grouping of individuals by number of top three articles, the number of professorsand the mean cites per year are: zero top three articles (n = 26, 6.59 cites); one top three article

Financial Management • Winter 2004144

Figure 2. Cumulative RTA Error (Reject a Top Article Not in Top N Journals) asNumber of Top N Journals Increases

The four criteria to be a top article are: Median (> 4 cites), Mean (> 9.55 cites), 90th %ile (> 24 cites),95th %ile (> 35 cites).

0

10

20

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Top N Journals Added in the Following Order: JF, JFE, RFS, JFQA, JMCB, JB, JFI, JIMF, JBF, JRI, JFM, FM,

REE, JREFE, JPM

Ty

pe

1 E

rro

r P

erce

nt

Median Mean 90 %ile 95 %ile

(n = 12, 7.88 cites); two to three top three articles (n = 15, 8.67 cites); four to five top threearticles (n = 12, 8.98 cites); and more than five top three articles (n = 11, 34.05 cites).1

One result of this analysis is that the mean number of cites per year for the differentgroups does not become statistically significantly different at the 0.05 level until a professorhas more than five top three articles.

When I run a regression of the 76 professors using the cites/year as the dependent variableand the number of top three articles as the independent variable, I find that the adjusted R2

is 0.357 and the model is significant at the 0.001 level. When I run the same regression bydropping the 11 observations with more than five top three articles, the adjusted R2 drops to-0.004 and is not significant at any reasonable level of confidence.

Fishe’s (1998) work illustrates that approximately one-third of 76 full professors are able tohave an impact on the literature, as measured by SSCI cites per year, without publishingeven one article in JF, JFE, and RFS. These professors make an impact by publishing in otherfinance, economics, and business journals. Based on this study, the citation list may be abetter reflection of an individual’s impact on the literature than that person’s publication list.

Other information that reflects a person’s impact on the literature may include the relativepercentage of self-cites or the number of co-authors for each article cited. If an article is

1Fishe (1998) also provides similar information on 51 full professors in finance programs ranked 1-20. Theaverage number of top three articles was 6.45 and the mean cites per year was 37.5 cites.

Smith • Is an Article in a Top Journal a Top Article? 145

Journal Top N

Journals

Cumulative Error % Median

(> 4 Cites)

Cumulative Error %

Mean (> 9.55 Cites)

Cumulative Error %

90%ile (> 24 Cites)

Cumulative Error %

95%ile (> 35 Cites)

JF 1 94 104 128 130 JFE 2 76 83 99 108 RFS 3 71 77 95 100 JFQA 4 66 73 87 96 JMCB 5 63 69 83 89 JB 6 59 66 82 90 JFI 7 56 66 83 91 JIMF 8 56 68 83 89 JBF 9 54 67 86 92 JRI 10 53 64 87 92 JFM 11 52 66 87 93 FM 12 49 66 88 93 REE 13 49 68 88 94 JREFE 14 50 70 89 94 JPM 15 53 71 90 95

Table V. Combined Reject Top Article (RTA) Error and Accept Non-Top Journal(ANTA) Error: Cumulative Basis by Top N Journals

The journals [Journal of Finance (JF), Journal of Financial Economics (JFE), Review of Financial Studies(RFS), Journal of Financial and Quantitative Analysis (JFQA), Journal of Money Credit and Banking(JMCB), Journal of Business (JB), Journal of Financial Intermediation (JFI), Journal of InternationalMoney and Finance (JIMF), Journal of Banking and Finance (JBF), Journal of Risk and Insurance (JRI),Journal of Futures Markets (JFM), Financial Management (FM), Real Estate Economics (REE), Journalof Real Estate Finance and Economics (JREFE), and Journal of Portfolio Management (JPM)] are cumu-lated from the top down. The four criteria for an RTA (Type I) Error are the percentage of total articles notin the cumulative set of journals but having cites above: 1) the median of 4 cites, 2) the mean of 9.55 cites,3) the 90th percentile of 24 cites, and 4) the 95th percentile of 35 cites which are based on the statistics for626 articles in the 15 journals. The four criteria for an ANTA (Type II) Error are the percentage of articlesin the cumulative set of journals not having cites above: 1) the median of 4 cites, 2) the mean of 9.55 cites,3) the 90th percentile of 24 cites, and 4) the 95th percentile of 35 cites. For example, for the Top 3 journals(JF, JFE, RFS), there is a combined error rate of 77% (44% RTA error plus 33% ANTA error) for thecriterion of > 9.55 cites.

newly published, then the journal ranking may be the best information on the potentialimpact of a work. However, if the article is three years old or older, then an evaluator shouldrequire a record of where that work has been cited. If the evaluator does not ask for thisinformation, then he or she is more likely to accept a non-top article in a top journal as a toparticle, or reject a top article in a non-top journal.

Even with a complete citation record for an article, evaluators can continue to disagree. Asdiscussed earlier in this article, an evaluator might consider many characteristics related to acitation. Where is the article cited? Is the citation a self-cite? Is the evaluator familiar with thecited or citing journals because of his or her research interests or geographic location? Is thecited work being corrected in the citing article or is the citing article extending the work of thecited article? Does the cited article provide a survey of previous literature, an empirical test ofan already established theory, or an important new theoretical insight? All three article types

Financial Management • Winter 2004146

Figure 3. Cumulative Combined RTA (Reject Top Article) Error and ANTA (AcceptNon-Top Article) Error as Number of Top N Journals Increases

The four criteria to be a top article are: Median (> 4 cites), Mean (> 9.55 cites), 90th %ile (> 24 cites), 95th%ile (> 35 cites).

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Top N Journals are added in the following order: JF, JFE, RFS, JFQA, JMCB, JB, JFI, JIMF, JBF, JRI, JFM, FM,

REE, JREFE, JPM

Co

mb

ine

d E

rro

r P

erce

nt

Median Mean 90 %ile 95 %ile

provide information, but each evaluator might view the relative contributions differently.Another evaluative factor that is pertinent to this article is the citation record for an article

relative to its journal. Krishnan and Bricker (2004) find that JF, JFE, and RFS add significantvalue in terms of citation over and above inherent article quality, but JFQA and JB do not.Krishnan and Bricker’s paper indicates that the number of citations may be a function of thearticle quality and the method of dissemination, i.e., the journal. Referring back to Table I, sixcitations for a JF article would put it in the bottom quartile and at 25% of the mean cites of24.3 for that journal. Six citations for a JBF article would put it in the upper quartile and at130% of the mean of 4.6 cites for that journal. Given Krishnan and Bricker’s results, anevaluator might argue that when he or she controls for the journal of publication, the sixcites in JBF show a higher quality than do the six cites in JF.

III. Summary and Conclusion

In this study, I rank 15 leading finance journals by the average number of Social SciencesCitation Index (SSCI) cites per articles from 1996 to January 2004 for articles published inthose 15 journals in 1996. Other recent studies often use Journal Citation Reports (JCR)impact factors to rank journals. Although my study uses one year of publications, 1996, thenumber of citations is over a much longer period than one year as with the JCR impact factors.

Smith • Is an Article in a Top Journal a Top Article? 147

Tab

le V

I. J

ou

rn

al C

itati

on

Rep

orts

(JC

R)

Imp

act

Facto

rs, A

ctu

al, a

nd

Esti

mate

d C

ites

The

JCR

impa

ct fa

ctor

(IF)

for a

jour

nal i

s the

num

ber o

f cite

s in

year

t fo

r arti

cles

pub

lishe

d in

yea

rs t-

1 an

d t-2

div

ided

by

the

num

ber o

f arti

cles

pub

lishe

d in

ye

ars t

-1 a

nd t-

2. T

he jo

urna

ls ar

e Jo

urna

l of F

inan

ce (J

F), J

ourn

al o

f Fin

anci

al E

cono

mic

s (JF

E), R

evie

w o

f Fin

anci

al S

tudi

es (R

FS),

Jour

nal o

f Fin

anci

al a

nd

Qua

ntita

tive

Ana

lysis

(JFQ

A),

Jour

nal o

f Mon

ey C

redi

t and

Ban

king

(JM

CB),

Jour

nal o

f Bus

ines

s (J

B), J

ourn

al o

f Fin

anci

al In

term

edia

tion

(JFI

), Jo

urna

l of

Inte

rnat

iona

l Mon

ey a

nd F

inan

ce (

JIM

F), J

ourn

al o

f Ba

nkin

g an

d Fi

nanc

e (J

BF),

Jour

nal o

f Ri

sk a

nd I

nsur

ance

(JR

I), J

ourn

al o

f Fu

ture

s M

arke

ts (J

FM),

Fina

ncia

l Man

agem

ent (

FM),

Real

Esta

te E

cono

mic

s (R

EE),

Jour

nal o

f Rea

l Esta

te F

inan

ce a

nd E

cono

mic

s (J

REFE

), an

d Jo

urna

l of P

ortfo

lio M

anag

emen

t (J

PM).

Estim

ated

Cite

s are

bas

ed o

n a

regr

essio

n m

odel

: Esti

mat

ed C

ites =

-0.2

68 +

9.7

72 (A

vera

ge IF

). M

odel

adj

uste

d R2 =

0.8

59, w

hich

is si

gnifi

cant

at t

he

.001

leve

l.

Jo

urn

al

200

2 I

F

200

1 I

F

200

0 I

F

199

9 I

F

199

8 I

F

199

7 I

F

199

6 I

F

199

6-2

002

Ave

rag

e I

F

Ave

rag

e

IF R

an

k,

Cit

es

Ran

k

Ac

tual

Mean

Cit

es

Esti

mate

d

Mean

Cit

es

JF

3.49

4 2.

958

2.75

3 2.

646

2.13

7 2.

173

2.12

3 2.

612

1, 1

24

.3

25.2

6 JF

E 3.

248

2.57

7 1.

904

1.70

5 1.

767

2.50

6 2.

609

2.33

1 2,

2

23.1

22

.51

RFS

1.85

1 1.

671

1.34

3 1.

452

1.01

4 1.

329

1.12

9 1.

398

3, 3

16

.0

13.4

0 JF

QA

1.

259

0.90

4 0.

596

0.54

0 0.

727

0.69

4 0.

591

0.75

9 8,

4

13.2

7.

15

JMCB

0.

682

0.76

8 0.

915

1.05

7 1.

115

0.84

3 0.

586

0.85

2 6,

5

10.4

8.

06

JB

1.72

7 1.

357

1.16

2 0.

889

1.16

4 1.

410

0.77

5 1.

212

4, 6

9.

8 11

.58

JFI

0.92

9 1.

536

0.51

9 0.

444

0.85

2 0.

774

0.55

7 0.

802

7, 7

8.

2 7.

56

JIM

F 0.

565

0.68

9 0.

394

0.56

0 0.

835

0.57

3 0.

494

0.58

7 9,

8

5.9

5.47

JB

F 0.

688

0.76

6 0.

533

0.66

4 0.

465

0.35

1 0.

469

0.56

2 10

, 9

4.6

5.23

JR

I 0.

370

0.19

6 0.

554

0.26

8 0.

421

0.51

7 0.

483

0.40

1 13

, 10

4.4

3.65

JF

M

0.25

0 0.

364

0.33

7 0.

312

0.38

0 0.

281

0.38

9 0.

330

14, 1

1 3.

7 2.

96

FM

1.20

5 0.

741

0.23

8 1.

500

0.88

3 1.

119

1.14

5 0.

976

5, 1

2 3.

5 9.

27

REE

0.28

8 0.

679

0.76

4 0.

386

0.28

1 0.

469

0.55

7 0.

489

12, 1

3 3.

0 4.

51

JREF

E 0.

437

0.62

9 0.

676

0.48

5 0.

588

0.45

6 0.

384

0.52

2 11

, 14

2.6

4.83

JP

M

0.33

3 0.

215

0.25

3 0.

411

0.21

3 0.

305

0.29

5 0.

289

15, 1

5 1.

3 2.

56

Financial Management • Winter 2004148

I also define and analyze a “top article” (average number of cites is above the median,mean, or 90th or 95th percentiles, for a set of leading finance journals) as opposed to a “topjournal article.” The results show that a top three (JF, JFE, RFS) approach to identifying toparticles (based on being above the mean number of 9.55 cites per article) leads to a Type Ierror (a top article is rejected by the decision rule) 44% of the time and a Type II error (a non-top article is accepted as a top article) 33% of the time. I provide similar information for the15 individual journals and for different levels of citations. At the 90th and 95th percentiles,the Type I error decreases but the Type II error increases dramatically. These resultsdemonstrate that if an evaluator is interested in identifying top articles, the Type I and IIerrors associated with a top three journals approach should lead him or her to look at abroader set of journals and to examine each article for its intrinsic value, rather than thegeneral quality of its journal.

I also examine the relation between the JCR impact factors for the 15 journals and theiractual number of cites. The first regression uses the mean cites per journal as the dependentvariable and the average JCR impact factor (1996-2002) as the independent variable. A secondregression uses the actual cites per article as the dependent variable and the average JCRimpact factor as the independent variable. The results indicate that there is a strong overallpositive relation between JCR impact factors and actual cites. However, I urge caution on animpact factor’s predictive ability for a specific journal or a specific journal article.

Based on this study, the citation list may be a better reflection of an individual’s impact onthe literature than that person’s publication list, and an evaluator may want a record ofwhere that work has been cited. If the evaluator does not solicit or develop this information,then he or she is more likely to accept a non-top article in a top journal as a top article, orreject a top article in a non-top journal.

References

Alexander, J.C. and R.H. Mabry, 1994, “Relative Significance of Journals, Authors, and Articles Cited inFinancial Research,” Journal of Finance 49, 697-712.

Arnold, T., A.W. Butler, T.F. Crack, and A. Altintig, 2003, “Impact: What Influences Finance Research?”Journal of Business 76, 343-362.

Borde, S.F., J.M. Cheney, and J. Madura, 1999, “A Note on Perceptions of Finance Journal Quality,” Reviewof Quantitative Finance and Accounting 12, 89-96.

Borokhovich, K.A., R.J. Bricker, K.R. Brunarski, and B.J. Simkins, 1995, “Finance Research Productivityand Influence,” Journal of Finance 50, 1691-1717.

Borokhovich, K.A., R.J. Bricker, and B.J. Simkins, 2000, “An Analysis of Finance Journal Impact Factors,”Journal of Finance 55, 1457-1469.

Chan, K.C., 2001, “A Citation-based Ranking of Journals in Financial Research: Some New Results,”Journal of Financial Education 27, 36-52.

Chan, K.C., C.R. Chen, and T.L. Steiner, 2002, “Production in the Finance Literature, Institutional Reputation,and Labor Mobility in Academia: A Global Perspective,” Financial Management, 31, 131-156.

Chan, K.C. and R.C.W. Fok, 2003, “Membership on Editorial Boards and Finance Department Rankings,”Journal of Financial Research 26, 405-420.

Smith • Is an Article in a Top Journal a Top Article? 149Chan, K.C., R.C.W. Fok, and M. Pan, 2000, “Citation-Based Finance Journal Rankings: An Update,”

Financial Practice and Education 10, 132-141.

Christoffersen, S., F. Englander, A.C. Arize, and J. Malindretos, 2001, “Sub-Field Specific Rankings ofFinance Journals,” Journal of Financial Education 27, 37-49.

Chung, K.H., R.A.K. Cox, and J.B. Mitchell, 2001, “Citation Patterns in the Finance Literature,” FinancialManagement 30, 99-118.

Coe, R.K. and I. Weinstock, 1983, “Evaluating the Finance Journals: The Department Chairperson’sPerspective,” Journal of Financial Research 6, 345-349.

Fishe, R.P.H., 1998, “What Are the Research Standards for Full Professor of Finance?” Journal of Finance53, 1053-1079.

Krishnan, C.N.V. and Robert Bricker, 2004, “Top Finance Journals: Do They Add Value?” Journal ofEconomics and Finance (Forthcoming).

Niemi, A.W. Jr., 1987, “Institutional Contributions to the Leading Finance Journals, 1975-1986: A Note,”Journal of Finance 42, 1389-1397.

Oltheten, E., V. Theoharakis, and N.G. Travlos, 2003, “Faculty Perceptions and Readership Patterns ofFinance Journals: A Global View,” Journal of Financial and Quantitative Analysis (Forthcoming).