Meetings & Conferences Science Online

Real-time statistics in science (Science Online London 2009)

Victor Henning, Richard Grant, Virginia Barbour

Academic prestige, setting research trends, getting jobs and tenure, grant funding – they are largely based on publishing in high-Impact Factor journals and getting citations. Not only are these measures flawed and widely critized: “You could write the entire history of science in the last 50 years in terms of papers rejected by Science or Nature”, said Nobel laureate Paul Lauterbur. Citation measures are also subject to a considerable time-lag. If you write a paper today, it takes a year to get it published, and another year passes by until citations of it appear. What if there were alternative measures of scientific impact? What if these measures were available in real-time, letting you track the trends in your discipline as they develop? That’s what we’ll discuss in this session.

Richard Grant:

Employers like metrics to discover if they’re spending money in the right places. Researchers want to see that what they’re doing is relevant. This is why we want metrics. But what can metrics do, and what can’t they do? Impact factors: doesn’t actually tell you how good research is in a given journal. He is involved in the qualitative assessment of articles. More like a FriendFeed method of assessment. Corporate bit: http:/ The crucial thing they want to have is quality. What they do at f1000 is pretty slow, by necessity. There is also, though, a tying-in with the community.

Virginia Barbour:

She’d like to reclaim the word “impact” from “impact factor”. How do you assess quality: usage, media coverage, blog coverage, expert ratings, discussion thread activity, who is reading it, who is citing it, where the research was done, effect on public policy? No single one is one you should rely on. Traditional measures are often not the most important. Many feel that the way papers are being evaluated is actually detrimental to the research process. Most users of journal sites are not coming via the home page – they’re coming via Google and other methods: people just don’t start at the first page of a journal and read through.

NEJM is changing the way their front pages look and the Journal of Vision is changing the way the metrics are displayed. At PLoS, in Phase 1 they want to have data that isn’t owned by someone else – that we can actually use and verify. In Phase 2, they also want to have the number of downloads of the article. This data will be broken down by the type of views. They also want to make the metrics more sophisticated, with more sources for each data type, more sophisticated web usage data, provide tools for analysis, and more.

Victor Henning:

Used as an analogy for article metrics, and as an introduction to Mendeley. In this way, you can track article pervasiveness in reference manager libraries, track article reading time in PDF viewers, and track user tags and ratings. One key difference with Mendeley and is privacy: they believe that some scientists don’t want others to know what literature they find interesting.

They have synchronization with citeulike, and will shortly have synchronization with Zotero. The goal of all this is to aggregate statistics for their users. All of the information is available by academic discipline, geographic region, and more. Once we’re at the point where there are true article metrics, this can be the basis for individualized recommendations.


Question: It seems we’re replacing a single impact factor with a large number of new ones. How do you forsee people managing and understanding all of those metrics? RG: we’re not in the business of replacing the impact factor – just providing more information to the researcher.

VB: I can imagine that people will be able to go to grant funding agencies and tell them how much coverage in all sorts of media your paper received.

Question (Phil Lord): I worry about reading times as a measure of quality. In music the listeners and musicians are largely disjoint. In science this is definitely not true. Many of the metrics mentioned are very much open to fiddling and self-citation. What do you say about this? VH: We’re not advocating replacing the impact factor. However, it is always better to have more data, more metrics.

Question: I print out my articles. How will that affect things?

Missed most of the rest of the discussion because of a phone that wouldn’t stop ringing – see the Twitter hashtag #solo09 for all the gory details.

FriendFeed Discussion

Please note that this post is merely my notes on the presentation. They are not guaranteed to be correct, and unless explicitly stated are not my opinions. They do not reflect the opinions of my employers. Any errors you can happily assume to be mine and no-one else’s. I’m happy to correct any errors you may spot – just let me know!