|
METRICS MASHUP
The End of Metrics as We Know Them?
by Elaine M. Lasda
My library’s policy for researcher evaluation requests is to provide simple, basic citation metrics: total citation count, times self-cited, and h-index. The preferred source is Web of Science, although departments may request alternative sources.
A recent review of this policy prompted me to consider the evolution of bibliometrics and research evaluation. The landscape is now radically different than when I began my career. Examining the evolution of research impact assessment involves three components: datasets, search-and-retrieval platforms, and methods of analysis with resulting metrics.
DATASETS
The original ISI Citation Index datasets, precursors to today’s Web of Science (WoS), contained content limited by the costs and effort of collecting and organizing publication and citation data. ISI founder Eugene Garfield curated the data to include only what he considered influential or significant journal titles. At its inception, the Science Citation Index covered 613 journal titles and 1.4 million citations (historyofinformation.com/detail.php?id=817). Since then, coverage fluctuated as journals changed, merged, or lost and gained prestige as disciplinary trends evolved. Clarivate has expanded WoS data to 21,000 titles and 80 million publication records in the Core Collection alone (searchsmart.org/results/wos.corecollection?).
Scopus’ dataset is also curated, employing subject matter experts outside Elsevier. Launched in 2004, Scopus highlighted its larger number of journal titles compared to WoS. However, in 2010, I discovered missing metadata, including cited references, in a group of five journal titles ostensibly indexed in Scopus. It would seem that its marketing got ahead of its data cleaning. Today, Scopus covers approximately 25,000 journals, totaling nearly 85 million publication records (searchsmart.org/results/scopus?).
Google entered the field around the same time. Instead of curated content, Google Scholar uses an automated spider to crawl the web, analyzing HTML code and metadata to determine if online content is scholarly. This results in inconsistent citation search quality. Google Scholar continues to offer specifications for configuring the back end of online publications for discovery (scholar.google.com/intl/en/scholar/inclusion.html). SearchSmart estimates Scholar’s content at 389 million publication records (searchsmart.org/results/googlescholar?).
Comparisons among these three sources show overlapping items, yet each retrieves unique citing references. This is quite vexing, because if one were to perform an evaluative citation analysis using WoS, there is now a sense of incompleteness. When it was the only tool, searchers relied on that citation count as the authoritative indicator. With more tools, citation counts could become more accurate, but this requires searching across multiple platforms, deduplication, and synthesizing the results.
In 2018, Dimensions launched. Like Scholar, it relies on publisher agreements and open research repositories to populate its database. Dimensions uses linked data and machine learning (ML) to add policy documents, clinical trials, grant sources, and related content. Its approach and its commitment to OA principles made this database a bit of a game changer. Dimensions indexes roughly 100 million publications (searchsmart.org/results/dimensions.publicationsfree?).
Now we are seeing tools that leverage large language models (LLMs) that scrape rather than crawl. While crawling indexes metadata, scraping extracts data and content. About 10–15 years ago, researchers had begun scraping or downloading data (with vendor permission) to leverage ML for advanced bibliometric analyses. Now, LLMs harvest data with varying levels of curation and little transparency. Searchers cannot easily identify or exclude poor quality data. Clarivate and Elsevier have built proprietary LLMs from decades of collected publication data, and presumably their careful curation will result in higher quality than open source LLMs for bibliometric purposes. Dimensions uses a ChatGPT overlay on its dataset to achieve a similar result.
SEARCH AND RETRIEVAL
Decades ago, online retrieval leveraged command language and Boolean searching to balance precision and recall. Mediated searching was common, and a result list of 30–35 records was often considered ideal due to cost and data limitations.
Graphical user interfaces (GUIs) emerged in the mid-1990s, replacing command prompts with search boxes and menus for field limiters. Searching remained primarily index- and abstract-based. Citation searching through a GUI was similar to using a command prompt, with results ranked chronologically and options to sort by record elements. Back then, computer memory limitations prevented ISI databases from offering full text or citing references in most other databases.
Google’s single search box and PageRank relevancy algorithm transformed searching. Remember, PageRank is based on Garfield’s citation linking concept, using eigenvector analysis to determine relevance by counting authoritative page links.
With machine learning and Big Data analytics, natural lan-guage processing gained traction in the mid-2010s, enabling insights previously impractical to glean. Scite, for example, examines not only cited and citing references but also classifies them based on whether the citing publication supports, contrasts, or simply mentions the reference. Prior to Scite, few tools could collect citation context, though bibliometricians conducted such analyses on more of an ad hoc basis.
METRICS
I will admit to giving short shrift to some metrics in this section, but there has been significant evolution in measuring research impact since Garfield’s day. The Journal Impact Factor (JIF), for all its foibles, is a relatively simple to understand ratio of citation counts per number of articles. The cited half-life, immediacy index, and other metrics track the speed of idea dissemination through citations. While imperfect, ISI metrics are straightforward.
Hirsch’s h-index provides a simple numerical indicator reflecting publication count and citation impact. Scopus created and adopted metrics such as Source Normalized Impact per Paper (SNIP) and the Scimago Journal Rank (SJR), which are based on similar concepts as WoS using slightly different calculations.
Scopus and WoS now offer indicators powered by ML algorithms, allowing comparisons of impact across disciplines with different citation patterns. Clarivate offers the Category Normalized Citation Impact (CNCI), and Scopus provides the Field Weighted Citation Impact (FWCI). Dimensions uses the Relative Citation Ratio (RCR) and Field Citation Ratio (FCR), which contextualize publication impact within disciplines.
Altmetrics track research impact in social media, news, and other online sources. Elsevier’s PlumX and Digital Science’s Altmetric are the major players. The usefulness of these metrics remains debated.
A recent trend is the development of values-based impact indicators. The first tool I was aware off that leveraged values in this way was Impactstory. The author profiles used badges to indicate alignment with open science principles, though the tool has not seen much recent activity.
Another approach, known as the Becker Model, assesses societal impact through various lenses (becker.wustl.edu/impact-assessment/model). Initially, tracking all potential areas of impact in this model seemed daunting and potentially required a longitudinal approach. AI and new data capture methods might now be making it easier to connect data from diverse sources, creating a more holistic picture of societal impact.
Public Library of Science (PLOS) developed Open Science Indicators (OSI; figshare.com/articles/preprint/PLOS_Open_Science_Indicators_principles_and_definitions/21640889?file=38446724) to reflect the adoption of open science practices. The data behind OSI is open, and calculation methods are transparent (theplosblog.plos.org/2025/07/theres-more-to-research-than-citations-understanding-knowledge-sharing-practices-with-open-science-indicators).
This vast and ever-increasing range of metrics begs the question, what is impact? I may sound pie-in-the-sky here, but impact ought to be the effect of scientific discoveries and knowledge to better understand our society, world, and beyond—or dare I go as far as to say to change the world for the better?
WHERE ARE WE NOW?
The current bibliometrics landscape is best described as proliferation. There are more options for research evaluation and bibliometric analysis than ever. While the growing array of tools is fascinating and offers many ways to measure impact, it can also create confusion for those unfamiliar with bibliometrics.
Kudos (growkudos.com) is moving in an interesting direction, using an LLM from Cactus to create plain language stories about research. After reading about its new partnership with Pfizer (blog.growkudos.com/news/pfizer-signs-multi-year-agreement-with-kudos-to-optimize-discoverability-of-medical-publications), I requested a demo of the interface. Kudos offers visually appealing, plain language stories designed to shift focus from researcher-to-researcher impact, as measured by traditional citation metrics, to researcher-to-reader, broader accessibility, and impact for nonexperts, especially in medical and health sciences. While most viewers remain researchers, Kudos also serves practitioners, journalists, and the public. Its metrics stem from basic web tracking that is displayed in dashboard-style visualizations.
Another notable tool, Overton (overton.io), links scientific publications with public policy activity. Created in 2018 by Euan Adie, the founder of Altimetric, Overton in 2024 launched Overton Engage, which connects researchers with policymakers by providing information about public engagement opportunities for addressing social issues. These opportunities include consultations, advisory panels, funding, and training. Researchers can view these requests and align their research accordingly. Overton is quickly becoming an authoritative repository for grey literature, a resource that has eluded many previous efforts.
Clarivate and Elsevier are working to improve their recently launched gen AI research assistants. I have not been impressed so far. According to the Gartner Hype Cycle, I am waiting to move from the “Trough of Disillusionment” to the “Plateau of Productivity” as vendors deliver more meaningful AI applications. Meanwhile, Kudos seems to be on the right track, combining web and altmetrics, plain language narratives, and visual appeal to increase understanding and impact of scientific findings beyond the research community.
FINAL THOUGHTS
The expanding range of data, platforms, and analytics available today for evaluation is both helpful and challenging. The landscape now includes traditional citation-based measures as well as sophisticated algorithmic and alternative metrics that aim to capture the multifaceted influence of research. As methodologies and technologies continue to advance, our ability to assess and understand scholarly impact becomes increasingly nuanced and comprehensive.
A common refrain in research evaluation is that the only way to truly assess a publication is to read it. Gen AI could provide concise context and plain language descriptions of research. If implemented well, this could reduce the need for expert interpretation and facilitate the societal impact of scientific work beyond disciplinary echo chambers that often result through traditional citation-based metrics. |
