Counting what counts: The debate over citation metrics
Counting what counts: The debate over citation metrics
A new wave of data-driven research tools is reigniting an age-old conversation around what metrics can be used... and abused... when statistically analysing scientific literature quality.
There is one unspoken, but unavoidable, mantra of academic research: It’s (almost) all about citations, citations, citations.
At each level, the number of citations for a paper, a single researcher, or an entire institution, determines their perceived influence. At the most basic level, the impact of a paper is measured by its number of citations. But this importance quickly bubbles up. Citations determine a researcher’s h-index, which in turn may be used to determine if they should have a raise or promotion, whether or not that’s really fair. The impact factor of a journal is determined by how often it’s cited. At the most macro-level, the number of citations across a university contribute to its world-wide ranking.
However, citations are far from a perfect measure. The most serious problems stem from citation hacking, wherein researchers may try to increase the influence of their work by getting unmerited citations. For example, peer-review gatekeepers and editors can request a newly submitted paper to cite their own work before letting it get published. According to a 2022 paper analysing ~20,000 highly published authors in PubMed, up to 16% of authors may have engaged in reference list manipulation. Moreover, citation hacking itself is hard to control because of the decentralised nature of independent journals, allowing “chronic offenders” to continue unnoticed.
Despite their flaws, citations are still our gold-standard to measure impact and influence, as well as connection and relevance. After all, finding papers through the citation network (ahem, Litmaps) is often faster and more comprehensive than traditional keyword search alone. Yet it begs the question: given the known issues with citations, does relying on them for search exacerbate these existing biases? Perhaps. Although the same biases would likely occur with traditional search. After all, the issue is not with our reliance on citations but the compromised integrity of the citations themselves.
Solutions for citation hacking often turn to control and transparency. For example, keeping track of which references in a paper were added as a result of the reviewer’s or editor’s requests. However, such solutions are arguably along the same lines as those contrived for plagiarism among students (especially in the new-wave of ChatGPT cheating) — fixing without preventing.
Although it’s necessary to address these problems quickly, short-term solutions gloss over the motivations and systems that cause the indiscretion in the first place. Instead, citation hacking may be viewed as a part of the broader integrity issues in academia, like paper mills and manipulated peer reviews. Systematic and institutional changes, although the hardest to enact, are likely to be the most impactful in diminishing such abuses from occurring.
Where do you stand on citation tracking and hacking? Does it affect how you do research?
Join the discussion and share your opinions by commenting below or tweeting at us @LitmapsApp.
Resources
How Junk Citations Have Discredited the Academy: Part 4, March 2023
Fast-growing open-access journals stripped of coveted impact factors, March 2023
Gender inequality and self-publication are common among academic editors, January 2023
Another extraordinary year for citation impact: how COVID research changed citations, June 2022
Citation Cartels: The Mafia of Scientific Publishing, 2022
The h-index is no longer an effective correlate of scientific reputation, 2021
Paper mills and on-demand publishing: Risks to the integrity of journal indexing and metrics, 2020
 | A guest post by
|
l like it
Thanks for the great stream of informative articles you're writing. Just discovered them and they're a treasure trove.
It is true that the Publish or Perish mentality, and the fact that citation-based metrics became the defacto way to evaluate academic proficiency. This is because the policymakers and funders were not capable of pooling much resources into ensuring its fairness. But I think that the AI revolution (which you hinted in your other great newsletter) will help revolutionize that a bit; AI-infused metrics are going to be better at measuring the performance of academics than the lossy, hackable, citation-based metrics. AI - hopefully, with some salt - will be able to look into each of these citations and give a true measure of value for an academic's work, relevant to the funder's purpose.
Now, I think that this will give rise to two issues: the first is, citation hacking will evolve using AI as well, so while it's great news that faux-citations, crappy self-citations and other hacks to raise h-index will be detected, usage of AI will make it seem as fair usage (and if policymakers opt for too good metrics, they'll end up with draconian selection criteria).
The second issue is that if the funder's goals are shallow, that will also make the research shallow. I was listening to Leonard Kleinrock in an interview (or a panel, don't really remember) and he was lamenting the fact how research was in the past. In the past, DARPA would come with a bunch of money, give it to them, and then they'd do whatever they like with it. ARPANET, and eventually the Internet, became the fruit of that work. Nowadays, it is not the case: how funds are spent is constrained and the funds are becoming smaller and smaller. I've heard a similar argument from an Emeritus professor who's in his eighties or nineties whom I met in transit while waiting for a flight.
Now, introduce AI-based metrics to this and creativity gets heavily constrained. Ultimately, policymakers and funders want economic benefits in the long run, but how they set goals and evaluate performance will limit the researchers' (mostly academic ones) creativity and eliminate serendipitous research. A big part of how great innovations come by, as you know "chance favors the prepared mind", exposure and coincidence. Good research isn't always planned.
Having AI-based metrics will give the illusion of planning and that might compel institutions to reward less creative faculty who can reach the policymakers/funders' goals. I mean, it's their money, but in the long term - and I'm hinting at what you've mentioned in the other article - academia will become as profit-seeking and industrious as industry. I appeal a lot to the Technology Readiness Level (TRL, please confuse it with our research lab so I can get more citations), which is a 9-level scale for how ready a technology is, and I believe that Academics should focus on TRLs 1-3, leaving 4-6 mostly to Industry R&D (we teach our students to be ready for that), and 7-9 for actual Industry experts. Of course a lot of academic research takes place at 4 and 5 and we get patents too. But conceptualization and designing, rather than implementing and operating, new technologies is what makes STEM Academia, particularly Engineering, unique. It is creativity that drives us. I think AI will help compress the TRL over later stages, but is it going to do so over TRLs 1-3? AI-directed Funding will focus more on 4-9 or even 3-9, making 1-2 something akin to how Mathematicians are funded. Doing important work, but not lucrative enough for people to "care" in the short term (they do groundbreaking work with very little funding in underground offices).
So, I don't know how to feel about this. Hopeful? Scared (because I think became PoP-bred over the way somehow). It is certainly exciting for myself someone whose answer to the industry/academia question is Academia just because of the freedom (something the emeritus professor told me he regretted when he went to to the Biotech Industry for a few years in his youth after getting his PhD) and the impact someone can have by educating others.