FEATURE
Research Evaluation on a Tight Budget: Tools and Use Cases
by Connie Stovall and Doug Dechow
| [T]he importance of research impact information and research intelligence ... offers opportunities for librarians to demonstrate their expertise and value. |
Librarians have long leveraged bibliometrics and citation databases for internal purposes such as collection development. Now, the tools of disciplines such as scientometrics, bibliometrics, infometrics, and altmetrics are increasingly used beyond the library. Campus stakeholders ask librarians to work more broadly to assess research impact and foster research intelligence. In a 2023 Brookings Institution commentary, senior fellow Joseph Parilla and research associate Glencora Haskins write, “America’s network of research universities is one of its greatest sources of talent, entrepreneurship, and [R&D]. …” Around the same time, the National Center for Science and Engineering Statistics’ “Higher Education Research and Development Survey” indicated that “total academic R&D reached $108.8 billion in FY 2023,” and “expenditures funded by federal sources accounted for $5.6 billion of the total increase.” With increasing tuition costs and rising inflation, institutions are often required to demonstrate campus research impact to various audiences to justify spending levels.
Additionally, assessment of research plays a crucial role in university rankings. Among the 17 measures in the 2026 U.S. News & World Report college rankings are citations per publication, field-weighted citation impact, and publication share in top journals. Universities benchmark to better understand their rankings and employ research intelligence to better position themselves with data-driven decision strategies. As a result, researchers Cassidy R. Sugimoto and Vincent Larivière state in their book Measuring Research: What Everyone Needs to Know, “The unparalleled fervor around research evaluation has given rise to a proliferation of tools and available data sources.”
This context—the importance of research impact information and research intelligence—offers opportunities for librarians to demonstrate their expertise and value. Relevant library services include citation analysis, bibliometrics, social network analysis, and benchmarking for a range of collaborative opportunities: promotion and tenure evidence, grant application support, institutional rankings analysis, and collaborator identification. For these new responsibilities, libraries have either created research impact librarian positions or incorporated that work into existing positions. However, these positions are not uniformly supported, and many institutions struggle with the high cost of proprietary tools. This article provides examples of use cases, free tools that support them, and an acknowledgment of the importance of responsible research assessment (RRA).
Research Assessment/Intelligence Tools
 |
OpenAlex functions primarily as a bibliographic data source and provides a limited set of tools for analysis. OpenAlex favors a model in which users download, clean, and ingest data into a dedicated bibliometric visualization tool. |
OpenAlex: An Open Source Bibliographic Database
At the core of quantitative research evaluation methods are databases containing the traces of academic research: extramural funding, patents, policy papers, and bibliographic metadata. The proprietary and pricey Web of Science and Scopus are viewed as the standard-bearers for citation databases. Fortunately for budget-minded institutions, OpenAlex (openalex.org) provides alternative free access to the world’s academic data—with a premium subscription option—and makes this bibliographic data available with a CC0 public domain dedication. For indexed bibliographic materials, OpenAlex is the largest scholarly knowledge graph. A 2025 article in Scientometrics by Jack Culbert et al. finds that OpenAlex includes more than 240 million indexed items versus approximately 70 million for Web of Science and 65 million for Scopus. OpenAlex functions primarily as a bibliographic data source and provides a limited set of tools for analysis. OpenAlex favors a model in which users download, clean, and ingest data into a dedicated bibliometric visualization tool.
OpenAlex has already changed the bibliographic database landscape. In 2023, France’s Centre National de la Recherche Scientifique dropped its Scopus subscription and committed to OpenAlex and other open science tools. Also in 2023, Leiden University’s Centre for Science and Technology Studies (CWTS), producer of the CWTS Leiden Ranking and the VOSviewer visualization tool (discussed in the next section), announced an open data version: the CWTS Leiden Ranking Open Edition, which draws its data from OpenAlex.
 |
VOSviewer helps librarians visually convey results from large datasets. |
VOSviewer: A Network-Oriented Visualization Tool
No matter the source of the bibliographic metadata, research impact librarians have embraced tools to assist with analysis and visualization. One of the most popular—and relatively easy to use—is VOSviewer (vosviewer.com), a free network analysis tool that incorporates natural language processing techniques. VOSviewer requires the Java Runtime Environment and operates on Mac, Windows, and other systems. After running a search and exporting data from a database such as OpenAlex, users can import data into VOSviewer. Options include creating co-authorship networks, co-citation networks, and bibliographic coupling networks. Network maps display at the author, institution, or journal-title level, in addition to options for term and keyword co-occurrence network maps.
VOSviewer helps librarians visually convey results from large datasets. Specifically, it aids in seeing collaborations at the researcher or university level, who’s citing a researcher’s or university’s work, and terms and keywords associated with a researcher’s or university’s body of publications. At a glance, librarians can see research strengths and evidence of impact in scholarship. High-resolution visualizations can be downloaded or shared via an interactive VOSviewer webpage.
 |
Bibliometrix is an open source library developed in the R programming language for comprehensive science mapping. |
Bibliometrix/Biblioshiny: Open Science Mapping
Bibliometrix, along with its app Biblioshiny (bibliometrix.org), provides many of the same visualizations as VOSviewer, plus additional ones. Bibliometrix is an open source library developed in the R programming language for comprehensive science mapping. For those who prefer a no-code option, Biblioshiny is a Shiny-based (another open source R library) web app. Like VOSviewer, Bibliometrix and Biblioshiny users can import data via downloads or APIs. Bibliometrix goes beyond VOSviewer’s network science-centric approach to supply users with more choices for advanced statistical methods and parameter selections. Visualizations and charts can be downloaded or compiled in a static report. However, unlike VOSviewer, Bibliometrix does not have an interactive web version. Bibliometrix/Biblioshiny has a wide range of free tutorials and training options through websites, YouTube, and GitHub repositories. Additionally, fee-based training is available.
Use Cases in Research Assessment/Intelligence
Faculty
Librarians and the faculty members they support can benefit from a strong understanding of metrics-based research impact in the preparation of tenure and promotion (T&P) files. T&P research evaluation should begin by establishing a single identity for the faculty member’s scholarly record. Ideally, faculty members are registered with ORCID. After faculty name variants and affiliations are addressed, OpenAlex can serve as the backbone for retrieving publications (including OA publications), citations, topic concepts, funders, collaborators, h-index, and additional metrics. For those wishing to avoid using APIs, cleaning data, and analyzing in a dedicated tool, OpenAlex offers simple, straightforward analysis through its user interface (UI).
A UI-only workflow can be built directly from OpenAlex’s Works results panel by applying the author filter to assemble the person’s publication set, then reading the OA data donut to report an OA-focused headline (overall OA share and the count of OA items). The Year (time profile) histogram summarizes productivity over time—e.g., peak years, recent momentum, steadiness—and, if needed, specific bars focus on the last 5 years. From the Top listed works, three to five examples can be identified with the highest Cited By counts and visible full-text badges (e.g., PDF) to showcase both influence and accessibility. The Topic widget is used to name the researcher’s main concept areas and the number of works in each area. If the faculty member has a scholarly record across institutions, the Institution widget can be used to highlight key affiliations and collaboration footprints visible in the record. Alternately, librarians may use the researcher’s profile page to gather publication sets and metrics.
Departments/Schools/Colleges
Annual reporting is required for many campus administrators. Research impact librarians can collaborate with various university offices (e.g., institutional effectiveness, office of research) to provide data and analysis. In most cases, administrators seek numbers of publications, citations, and journal-level metrics to demonstrate a unit’s impact and productivity. Other select reporting metrics can include the h-index, h5-index, citations per publications, and number of publications per the United Nations’ Sustainable Development Goals (SDGs). Additional reporting might focus on the list of journals in which researchers have published, along with corresponding impact factors or quartile rankings. Freely available journal rankings can be found at websites for SCImago Journal & Country Rank (SJCR) or CWTS Journal Indicators. Alongside shedding light on a unit’s productivity and impact, these metrics can help to benchmark a department relative to those at other universities. Benchmarking helps to illuminate a unit’s strengths, weaknesses, and opportunities and may shape planning—all core features of research intelligence.
Librarians can assemble datasets of a unit’s scholarly works using OpenAlex and import the dataset into a science-mapping tool, such as VOSviewer. For creating a thematic visualization, VOSviewer counts and visually represents terms and phrases used in titles, keywords, and abstracts. This kind of visualization helps end users quickly see the range of research topics with which a unit engages and identify prominent terms. The resulting term map also helps end users see gaps in research expertise, which can inform hiring efforts. Additionally, entity-level co-authorship maps assist administrators in at least two ways. One, a co-authorship map can reveal how a direct report is collaborating, which can be particularly useful if interdisciplinary collaboration is emphasized. Two, these maps offer an administrator bragging points (e.g., connections with high-profile companies such as Google or high-ranking universities such as Oxford).
University
At the university level, research impact metrics use is similar to that of units. One key difference is reporting to external agencies. Although large institutions may struggle with data size limitations depending on report type, it’s plausible to use OpenAlex and VOSviewer for external reporting. Usually, universities and colleges will need at least total number of publications, total citations, and metrics beyond the bibliographic. Additionally, more universities now benchmark and report on the number of OA publications, which have taken on greater significance as a result of federal OA mandates.
Grant Teams
At least three scenarios exist in which impact metrics can be useful in extramural funding, such as grants. The pre-award scenario involves providing metrics for a grant proposal. While agencies and programs differ, grant program managers reviewing proposals commonly request research metrics. The post-award scenario involves providing program managers with metrics for performance monitoring and can include setting up search alerts with a funding number to retrieve new publications and provide citation reports. The final scenario includes using metrics to identify potential collaborators.
Using Metrics Responsibly
Universities’ research communities are largely composed of faculty, and those same faculty members often experience the pressure of research evaluation most acutely. RRA represents a significant shift in how academic contributions are recognized, moves away from a primary reliance on quantitative metrics to a more holistic evaluation process, and values qualitative analysis and reflection. Early influences on RRA include the San Francisco Declaration of Research Assessment (DORA), published in 2012, and the Leiden Manifesto for Research Metrics, authored by experts in bibliometrics, social science research, and research administrators in 2015.
Conclusion
The importance of research impact and research intelligence in university, unit, and individual faculty member decision making means that librarians’ expertise in bibliometrics and research databases is incredibly valuable. Tools such as OpenAlex, VOSviewer, and Bibliometrix/Biblioshiny are free and especially useful for individual faculty tenure and promotion, internal and external reporting by units and institutions, and external grant processes. Furthermore, RRA balances quantitative with qualitative analysis. When librarians employ these and other tools to assess research impact and foster research intelligence, a variety of campus stakeholders can be supported.
|
Resources
Culbert, J.H., Hobert, A., Jahn, N., Haupka, N., Schmidt, M., Donner, P., and Mayr, P. (2025). “Reference Coverage Analysis of OpenAlex Compared to Web of Science and Scopus.” Scientometrics, 130(4), 2475–2492.
doi.org/10.1007/s11192-025-05293-3.
Edwards, M. and Roy, S. (2017). “Academic Research in the 21st Century: Maintaining Scientific Integrity in a Climate of Perverse Incentives and Hypercompetition.” Environmental Engineering Science, 34(1), 51–61.
doi.org/10.1089/ees.2016.0223.
National Center for Science and Engineering Statistics. (2024). “Higher Education R&D Expenditures Increased 11.2%, Exceeded $108 Billion in FY 2023.” National Science Foundation.
ncses.nsf.gov/pubs/nsf25313.
Manghi, P. (2024). “Challenges in Building Scholarly Knowledge Graphs for Research Assessment in Open Science.” Quantitative Science Studies, 5(4), 991–1021.
doi.org/10.1162/qss_a_00322.
Parilla, J. and Haskins, G. (Feb. 9, 2023). “How Research Universities Are Evolving to Strengthen Regional Economies.” Brookings Institution.
brookings.edu/articles/how-research-universities-are-evolving-to-strengthen-regional-economies.
Sugimoto, C.R. and Larivière, V. (2018). Measuring Research: What Everyone Needs to Know. Oxford University Press. |
Connie Stovall (L) is the director for research impact and intelligence at Virginia Tech, where she has worked since 2006. In this role, Stovall leads a team that provides institutional and industry insights via bibliometric analyses to Virginia Tech’s Office of Research and Innovation and Innovation and Partnership units, in addition to providing research and grant support to others on an ad hoc basis. She received an M.L.I.S. and M.A. from the University of Alabama and a B.B.A. from the University of North Alabama.