|
METRICS MASHUP
Simplifying Metrics in a Complex Landscape
by Elaine M. Lasda
Over breakfast the other day, I was contemplating the h-index (don’t judge). Jorge Eduardo Hirsch, who invented the h-index, ended his seminal explanation of the metric with this:
I suggest that this index may provide a useful yardstick with which to compare, in an unbiased way, different individuals competing for the same resource when an important evaluation criterion is scientific achievement.
—Proceedings of the National Academy of Sciences of the United States of America, v. 102, n. 46; doi.org/10.1073/pnas.0507655102 |
Hirsch wanted to combine two useful indicators of productivity: publication count and citation count. That h-index paper turned 20 years old in November 2025. Since then, there have been numerous criticisms of the h-index for a variety of reasons. Although he never doubled back on the ideas upon which the h-index is predicated, he disparaged both the misuse of the index and that the impact of his scholarly but unpopular work related to superconductors was eclipsed by the highly cited h-index proposal (“Superconductivity, What the H? The Emperor Has No Clothes,” International Journal of Modern Physics B, Vol. 38, No. 07, 2430001 (2024); doi.org/10.1142/S0217979224300019).
Hirsch was aiming for a simple metric. And maybe that’s the problem. Simple numbers may, in reality, be too easy to misconstrue. If the metric looks easy to understand at the surface, for example the h-index is an integer, often below 100, it is less likely that evaluators and others who would do best to dig into the nuance will do so. A simple number may appear simple to understand. In the case of the h-index and other metrics, this is not the case.
Take Omar Yaghi, the Nobel Laureate chemist and University at Albany alumnus. Google Scholar currently lists his h-index as 199; in Web of Science, it is 192; and in Scopus, it is 190. I went down a rabbit hole trying to reverse-engineer these scores to see what the difference would be if the Scopus score was mistakenly compared to the Google Scholar score. The presumption is that there are evaluators out there who read these scores as apples-to-apples despite being pulled from different back-end data.
I wanted to know the significance of an h-index of 190 in Scopus changing to one of 199. I thought the mathematics in a 2017 Scientometrics article by Lucio Bertoli-Barsotti and Tommaso Lando (“The h-index as an Almost-Exact Function of Some Basic Statistics”; doi.org/10.1007/s11192-017-2508-6) would provide insight. I consulted my colleague, mathematics professor Rongwei Yang, to help me understand the math. It turns out the paper describes a predictive model using four known variables: total citation count, citation count of highest-cited paper, total publication count, and number of papers with at least one citation. Yang explained the model uses a rather esoteric function called Lambert W.
We looked up Yang’s citation count in Google Scholar, which is 20, and in Web of Science, where it is 17. Using Google Scholar data, Yang was able to run the model in the paper, and the first thing he found was that since all of his published papers have been cited at least once, the model didn’t actually work. It would appear that the model only works if you have some papers that have zero citations. Yang then ran it again, adding to his publication count two preprints, because they currently have no citations.
FINDING TOTAL CITATION COUNT
This effort yielded a predicted h-index of 21. Yang told me that to work backward with the predictive model, the only one of the four variables you can determine is the total citation count. That requires the use of a different function, called the Redner function h_R. Therefore, the variables in this model do not provide enough information to figure out the magnitude of difference of an increase in the h-index, despite being able to predict the h-index fairly accurately.
Next, Yang leveraged AI tool DeepSeek to explain how to raise his h-index from 20 to 25. While DeepSeek provided some strategies for Yang to help increase his h-index; they focus on the distribution of citations per paper. Looking at the strategies offered by DeepSeek, there may be innumerable ways to get there, but the conclusion from DeepSeek is “broad and consistent increases in citations across your portfolio.” This is not new information.
To sum up: My head hurts trying to figure out how to explain to my library users how to draw any sort of comparison—except maybe a fuzzy understanding of the reputed correlation of scores from each source. Google Scholar scores “tend to be higher than” Scopus or Web of Science. But we can’t provide precise comparisons if they don’t know from what source the h-index scores come from, much less how to tell what an increase in the h-index of a given researcher actually signifies if you don’t know the citation distribution. If there are evaluators comparing an h-index of 190 in Scopus to another evaluator’s index of 199 in Google Scholar, will that look on the surface to be “better”? Pass the ibuprofen.
What needs to be in place in the research ecosystem that will mitigate perverse incentives, unintended consequences, and harmful outright misuse and gaming of evaluative indicators as the h-index?
In 2025, an interesting scoping review came out that frames the evolution of what they call “research impact science” into four broad “generations” in chronological order: quantitative bibliometrics, disciplinary and multidimensional frameworks, data-driven and predictive analytics, and finally altmetrics. The review is a great overview of the various types of metrics and ways of looking at research impact, as well as how new ways of computing have increased our capacity for ways to look at research effectiveness and impact (Arsalan, M., Mubin, O., and Al Mahmud, A. “Mapping the Generations of Research Impact Science: A Scoping Review of Metrics, Frameworks, and Predictive Approaches,” Journal of Library & Information Studies, v. 23, n. 1; doi.org/10.6182/jlis.202506_23(1).001). My take on the review is that the generations are not distinct but rather highly overlapping and cumulatively point to an ever-increasing complexity and variety of ways to measure different aspects of research output.
Considering the contents of this scoping review, it seems maybe the quest should not be for a better or easier-to-understand indicator. The scoping review made me realize we’ve got plenty of ways to show impact.
SIMILAR MEASUREMENTS
At the same time, some measurements are so similar, it is hard to see the point in having them. For example, the way Elsevier’s CiteScore is calculated seems almost exactly predicated on Clarivate’s Journal Impact Factor (JIF), with a few tweaks. Truly the more substantive difference in CiteScore and JIF is that there are distinctly different publication datasets from which the metrics are computed. It seems like a colossal waste of time and an exercise in ever-increasing confusion to have an indicator with different values for the same person/item depending on the data source (h-index), or all a panoply of different, yet sort of the same, metrics (CiteScore/JIF).
Differences in selection philosophies show that the nuances of what content is in and what is out, given the purveyor’s philosophy of coverage and how the content is evaluated, can sometimes make significant differences in the impact in indicators.
When Eugene Garfield invented the Science Citation Index (SCI), it was the one single authority for citation data. Flawed though the content was, the research landscape relied upon it. SCI, however imperfect, was canon. So, in one sense, there was a consistency and clarity by being the single source for this information. I might even argue that the decades-long position of being canon has been a significant factor in preserving perceived value and reputation of the Web of Science today. At the same time, for decades, we rallied against ISI for not making its publication list available to librarians and researchers. It was only when the competitors, in particular Scopus, arrived on the stage in the mid-1990s that ISI rethought that strategy.
Now, I’m wondering: Would an open, transparent, and universally accepted dataset be the solution to sorting out my boggled h-index machinations above? A unified, universally understood dataset that everyone uses could resolve many problems. Let users choose their preferred or most relevant metric or indicator; let the dataset be sliced and diced in any way the analyst or evaluator sees fit. The lynchpin is that the entire research ecosystem draws their analyses from the same data.
There are certainly efforts underway to leverage a single, all-inclusive, comprehensive, and authoritative data source on research output and citations. Dimensions for years has also included policy documents, clinical trials, and grants data; Clarivate and others are catching up on that front. Here is the point: If all players in the research game could agree on a data source, it would solve some of the thornier research impact problems.
I was intrigued to come across the Observatory of International Research (OOR; ooir.org/about.php). Andreas Pacher, a diplomacy researcher at the Vienna School of International Studies, created the project, which now seems to be about 4 years old. It originally focused on social science journal titles and became a comprehensive scraping of as many scholarly journals as Pacher could find. The back-end data is on GitHub. If Pacher can come up with a dataset this comprehensive as a self-described hobby, it is only a matter of time before another hobbyist figures out how to make an all-inclusive large language model (LLM) of nothing but scholarly output. Clarivate and Elsevier might not have any say in the matter.
A SINGLE AUTHORITATIVE DATASET?
Maybe if we can’t get all of the Elseviers, Clarivates, Springers, EBSCOs, and so forth to coordinate their content into one single platform, tools such as CrossRef and ORCID will move us forward by being effective at increasing the interoperability of all the proprietary systems. White-hat upstart OurResearch, a program of OpenNotes (see opennotes.org/ourresearch), is currently the promising front runner, along with OpenAlex (openalex.org), in creating something akin to a single authoritative dataset.
With the way black box LLMs are evolving, these companies should be looking to partner and join the leaders of the open community in greater depth. Already, we can’t really tell what research outputs might be swimming around in the murky data lakes of various LLMs. Even the creators of the LLMs often aren’t certain. As this article evolved, it seemed appropriate to end with a call to action for our for-profit corporate partners to jump in and support interoperability endeavors or work with OurResearch to further the value and authority of OpenAlex as a centralized authoritative resource. Fortuitously, a February 2026 blog from OpenAlex said it is moving toward a tiered institutional membership model (blog.openalex.org/a-new-way-to-support-openalex-become-a-member).
This may be cynical, but I’m not holding my breath for Elsevier and Clarivate to partner up in this capacity. Certainly, those of us in research-based institutions should consider reallocating some of our electronic resource budgets to an OpenAlex partnership to further its goals. I wonder what level of buy-in is needed to leverage a culture shift back to a single authoritative data source for citation analysis. Maybe we just have to wait for another hobbyist to come and build the thing we actually need. |
