In my last musing upon this topic only two years ago [1], I used the title "Bibliometrics: An IT perspective", so I wish to start by giving good reasons for revisiting the topic, and for now declaring "Bibliometrics out...". Firstly, a revisit is mandated by the ARC's publication of its "full ERA 2010 Ranked Journal List" [2, 3], a must read for researchers who are selecting a journal to receive their work. Secondly, the ARC's vision of what it can do with citation analyses seems to have moved on since their 2008 version of the ranked list [4], from an orientation towards an earlier and well-known bibliometric, Thomson Reuters' Journal Impact Factor [5], to a newer orientation favouring more recent "journalmetrics" marketed by Elsevier's Scopus [6]. Hence, "Bibliometrics out, journalmetrics in!"
To begin with, Table 1 presents data extracted from three large files, the ARC's 2008 [4] and 2010 [3] ranked lists, and Elsevier's Scopus 2009 file [7] listing their new "journalmetrics", SNIP and SJR. Whilst Table 1 seems large, it is only a relatively modest sample from current ranked list of some 20,712 journals, and because the ARC has provided very minimal documentation on its derivation and implications for rank and file researchers, it is necessary to try and work things out by examining groups comprising a reasonably significant number of journals.
| Journal and URL | Tier 2008 | Tier 2010 | OA # | FoR | SNIP 2009 | SJR 2009 |
| Asia-Pacific Journal of Teacher Education http://www.tandf.co.uk/journals/titles/1359866X.asp | A* | A* | No 3 | 1301 | 0.553 | 0.037 |
| Higher Education Research and Development http://www.tandf.co.uk/journals/titles/07294360.asp | A* | A | No 6 | 1303 | Not listed | Not listed |
| Australian Educational Researcher http://www.aare.edu.au/aer/contents.htm | A* | B | Yes 3 | 13 | 0.335 | 0.035 |
| Australian Journal of Education http://www.acer.edu.au/aje/ | A* | B | No 3 | 13 | 0.296 | 0.040 |
| Australian Journal of Adult Learning http://www.ala.asn.au/c143/Publications+About+AJAL.aspx | A | A | No 3 | 1301 | Not listed | Not listed |
| Australian Journal of Teacher Education http://ajte.education.ecu.edu.au/ | A | A | Yes 6 | 1303 | Not listed | 0.000 |
| Australasian J. of Educational Technology (AJET) http://www.ascilite.org.au/ajet/ [online only] | A | B | Yes 6 | 1303 | 0.563 | 0.041 |
| Australian Educational Computing http://www.acce.edu.au/item.asp?pid=1120 | A | C | Yes 2 | 1303 | 0.351 | 0.034 |
| Distance Education http://www.tandf.co.uk/journals/titles/01587919.asp | B | B | No 3 | 1301 | 0.892 | 0.048 |
| Journal of University Teaching and Learning Practice http://ro.uow.edu.au/jutlp/ | B | B | Yes 2 | 1301 | Not listed | Not listed |
| Australasian Journal of Engineering Education http://www.aaee.com.au/ | B | B | Yes 2 | 1301 | Not listed | Not listed |
| Australian Universities' Review http://www.aur.org.au/ | B | B | Yes 2 | 1301 | Not listed | Not listed |
| Issues in Educational Research http://www.iier.org.au/ | B | B | Yes 3 | 13 | 0.215 | 0.035 |
| International Education Journal http://www.iejcomparative.org/ | B | B | No ? | 13 | Not listed | Not listed |
| Evaluation Journal of Australasia http://www.aes.asn.au/publications/ | Not listed | B | Part 2 | 11 | Not listed | Not listed |
| Australasian Journal of Economics Education http://www.uq.edu.au/economics/AJEE/ | C | C | Yes 2 | 1302 | Not listed | Not listed |
| Journal of Institutional Research http://www.aair.org.au/jir/html/Journal.htm | C | C | Yes 2 | 1303 | Not listed | Not listed |
| Journal of Academic Language and Learning http://journal.aall.org.au/http://journal.aall.org.au/ | Not listed | C | Yes 2 | 1302 | Not listed | Not listed |
| Studies in Learning, Evaluation, Innovation and Development http://sleid.cqu.edu.au/ | Not listed | C | Yes 3 | 1303 | Not listed | Not listed |
| Journal of Learning Design http://www.jld.qut.edu.au/ | Not listed | C | Yes ? | 1302 | Not listed | Not listed |
Notes for Table 1:
| ||||||
Table 1 concentrates upon Australian based educational research journals that may be of interest to HERDSA News readers. Recently I prepared a similar table for AJET Editorial 26(1) [10], oriented towards educational technology journals, both Australian based and overseas based. If you inspect Table 1, and the table in the AJET Editorial if Table 1 is not a sufficient dose, I'm fairly certain that most readers would be a bit peeved. What does it mean? What's the pattern? What are "they" (ARC, ERA, the "Feds") up to? Why should I be induced to believe that my hard work is downgraded because I have chosen to publish in (for example) an Australian or regionally based, open access, professional society journal, rather than an American or European based "for profit" journal?
Well, those questions are very good questions. If you are looking for precise answers, I cannot give them, and perhaps you should rush back to your research work, to put in the extra time you may need in order to get into a Tier A* or Tier A journal compared with a Tier B or C journal. However, if perhaps you are interested in some reflective thoughts about bibliometrics and journalmetrics, and some guesses about potential avenues for research into research assessment exercises, I invite you to read on. After all, I claim to have done some reasonable guessing into the future with "Bibliometrics: An IT perspective" [1] two years ago!
To begin with, there are some "patterns" or suggestions of patterns emerging from Table 1. The change from the Australian oriented, practitioner oriented, broadly consultative, esteem based ranking processes underlying Tier 2008 [11] to the unspecified and vaguely defined processes underlying Tier 2010 [12] has led to a predictable outcome for Australian educational research journals. Whilst lower ranked journals tended to remain at the same ranking, higher ranked journals have tended to fall by one or even two Tiers. The latter category includes HERDSA's, AARE's, ACER's, ACCE's and ASCILITE's flagships. In Tier 2010, Australian and regional support for the flagship journals of these societies has been swamped by the big numbers of American and European journals. Another pattern suggested in Table 1, and other extracts from the ranked list, is that open access journals have been pushed towards the lower tiers in 2010 to a greater extent than the for profit journals. That is also a not unexpected outcome from what one presumes is a "more international" perspective introduced somehow into the ranking processes underlying Tier 2010. Open access journals are a new phenomenon, stimulated in relatively recent times by the Internet technologies [13], whilst the for profit journals are mostly older and better established (and therefore are better known and more prestigious), and were purchased by their current owners in pre-Internet times.
However, Table 1 (and other extracts from the 2010 ranked list) is perhaps most interesting for its lack of patterns. Maybe I'm getting too long in the tooth for this sort of thing, or perhaps the relevant data has not been published, but I cannot discern any precise relationships between Tier ranking and Elsevier's new "journalmetrics", SNIP and SJR (Table 1). All I can see is some quite admirable advances in bibliometrics [6], and astute generalship that has enabled Scopus to encircle the Impact Factor forces with a twin pronged outflanking by its SNIP and SJR divisions. Thus Thomson Scientific, citation supplier for the RQF in 2007 [14] was replaced by Elsevier Scopus for ERA 2010 [15] The tactical marketing offensive by Scopus has been hard hitting, for example consider these two quotations [16]:
Visionary progressAlso noteworthy, the brief quotations above contain the phrase "expertise in research performance measurement", which brings into mind the fuzzy dividing line between bibliometrics for analysing a journal and for analysing the research work of the authors of individual articles in journals. The former purpose is soundly grounded in the technologies underlying bibliometrics or journalmetrics, but the latter purpose may be pushing a good technology beyond its limits and into a quite inappropriate use. We may need to remind ourselves, again and again, about the limitations. In the words of a Scopus executive who was writing on new alternatives to the Thomson Reuters Impact Factor [5]:
Thanks to a special collaboration between Scopus and two research groups known for their expertise in research performance measurement, a suite of context driven metrics are available that:
- are publicly accessible, free of charge at www.journalmetrics.com
- Apply to 18,000 journals, proceedings and book series
- Are refreshed twice per year to ensure currency of metrics...More than just a number
The many applications of modern bibliometrics cannot be served by a single tool. Researchers have long demanded the next generation of context-based metrics reflected in SNIP and SJR.
Originally it [the Impact Factor] was intended as a collection management tool, but has since evolved into a metric used for evaluation of science at all levels as well as evaluation of authors. This can have far-reaching consequences for an author's grant applications, promotion and tenure since the metric is directly influenced by the performance of specific journals and is thus for a large part beyond the author's control. [17, de Mooij, 2007; quoted also in 1.]So, "... beyond the author's control"! This is a prime avenue for further research. What would authors like to do about it? What can authors do? Do researchers feel they will be better off with or without "tiering" of journals? In my humble opinion there seems to be much more publishing by the assessors about how to assess research, and the justification for their actions, and too little into the impact upon the assessees [18].
| Rank (a) | No. journals | % (b) | Overseas examples from higher education research (c) |
| A* | 1,030 | 4.97 | Studies in Higher Education. http://www.tandf.co.uk/journals/carfax/03075079.html Cambridge Journal of Education. http://www.tandf.co.uk/journals/carfax/0305764X.html |
| A | 3,054 | 14.75 | Journal of Higher Education. http://www.ohiostatepress.org/Journals/JHE/jhemain.htm Higher Education. http://www.springer.com/education/higher+education/journal/10734 Teaching in Higher Education. http://www.tandf.co.uk/journals/titles/13562517.asp |
| B | 5,667 | 27.36 | Internat. J. for Academic Development. http://www.tandf.co.uk/journals/routledge/1360144X.html Internat. J. of Teaching and Learning in Higher Education. http://www.isetl.org/ijtlhe/ J. of Further and Higher Education. http://www.tandf.co.uk/journals/titles/0309877X.asp |
| C | 10,682 | 51.57 | Higher Education Management and Policy. http://www.ingentaconnect.com/content/oecd/16823451 College and University. http://www.aacrao.org/publications/candu/index.cfm College Teaching. http://www.heldref.org/pubs/ct/about.html |
| Unranked | 279 | 1.35 | (a category for journals started in 2008) |
| Total | 20,712 | 100.00 | |
Notes for Table 2:
| |||
Table 2 suggests a second prime avenue for further research. The conundrum is the extent to which the Tiering of journals within specialist areas of educational research may deviate from the overall targets for the Tiers (5%, 15%, 30% and 50% [14]). Given what appears to be an absence of detailed information on the ranking processes underlying Tier 2010, there seems to me to be an opportunity for research into the impact of Tiering upon scholarly publishing processes. In compiling Table 2 and Table 1 in AJET Editorial 26(1) [10], I began to feel that researchers in "130306 Educational Technology and Computing" had been allocated a much smaller percentage of high ranked outlets (A* and A, nominally 20%) compared with researchers in "130103 Higher Education". To translate that impression into one potential research question, "What is the impact of Tiering upon lower ranked journals?" Presumably, lower ranked Australian based journals (Table 1) will have to increase their reliance upon overseas authors, as Australian authors switch their preferences to higher ranked journals, even if that means migrating from their preferred group of journals to a more distantly related group of journals.
Now, to what extent is that likely to happen? Only time and further research will tell, but just two weeks after the publication of the 2010 Tier rankings [3], I received the first Tiers related withdrawal of an AJET submission made by Australian authors. In response to my acknowledgment, and my expression of interest in "obtaining more details concerning what you and your co-authors perceived about the importance of journal ranking", one author replied [21]:
In answering your question I must say that two of us the authors are about to apply for promotion and to them the ranking is vital even though we all know that such ranking may not be fair to many good and well-established journals such as AJET. [21]
| Author: Roger Atkinson retired from Murdoch University's Teaching and Learning Centre in June 2001. His current activities include publishing AJET and honorary work on the TL Forum and ascilite Conference series, and other academic conference support and publishing activities. Website (including this article in html format): http://www.roger-atkinson.id.au/
Please cite as: Atkinson, R. J. (2010). Bibliometrics out, journalmetrics in! HERDSA News, 32(1). http://www.roger-atkinson.id.au/pubs/herdsa-news/32-1.html |