|
THE SYSTEMS LIBRARIAN
Building Data About Libraries and Their Technology Systems
by Marshall Breeding
| The data underlying the library directory has been collected through a variety of methods over a period of 3 decades. The data was developed to support my research and for the benefit of the broader library community. Libraries make good use of the data to inform their processes to evaluate, select, or retain their core automation systems. |
This edition of The Systems Librarian tells the story of developing a body of data over the course of 3 decades that supports the practical needs of libraries and that documents trends about their core technology systems. What started out as a list of the first libraries to launch their own websites has grown into a key resource for the field of library automation.
Libraries depend on many sources of data as they make strategic decisions. Operational data from their business systems informs resource allocation and strategic initiatives. As libraries consider making changes in their technology environment, they benefit from data about their peer institutions and relevant local, regional, and global trends. A directory on Library Technology Guides called libraries.org (librarytechnology.org/libraries) provides many layers of data to help libraries make informed decisions regarding their core technology systems.
I have been developing this library database since 1996. In that year, I edited the World Wide Web Yellow Pages, published by Mecklermedia and IDG (International Data Group) Books Worldwide, and compiled the listings for all of the libraries offering websites at that time. Although creating a printed directory of the web was an unsustainable project, it provided a starting point for a resource I have continually expanded and enhanced ever since. That initial list provided the seed for the library directory component of Library Technology Guides, first launched on the web as lib-web-cats in 1999. At that time, other resources were available, including LibWeb, created by Thomas Dowling, which listed library websites, and Peter Scott’s LibCats, which listed web-based library catalogs. Both of those resources disappeared long ago.
Multiple Layers of Data
That initial effort behind the lib-web-cats directory has since expanded, both in the number of libraries included and in the categories of data it collects. Now known as libraries.org, it offers several layers of data about libraries. On one level, it serves as a basic directory providing the physical address, geographic coordinates, phone number, and website for each library, including branches or facilities. A second layer holds demographic and operational data such as the service population, collection size, and volume of circulation transactions. The scale of this dataset falls within what can be assembled through a combination of manual compilation, automated processes, and a bit of crowdsourcing. Currently, libraries.org has more than 200,000 entries.
Notably, libraries.org is synchronized with other public datasets, such as those produced by IMLS for public libraries and the Integrated Postsecondary Education Data System (IPEDS) for academic libraries in the U.S. Coordinating with these datasets ensures that all U.S. public and academic libraries are represented and also provides additional layers of information regarding the statistical measures collected by these organizations. A few national datasets from other countries have been similarly used. Unfortunately, not all countries collect or make available systematic data about their libraries.
Focus on Technology Systems
The layers of demographic information provide context for the data regarding the technology systems used. The directory originally focused on ILSs, but it has expanded to include discovery services, digital collections tools, and resource-sharing tools and networks. Each record tracks each migration cycle of ILS products, including each previous system and selection date. Other product categories show only the current product and its implementation date.
Keeping the ILS data current has been a continuous effort since the database was first created, mostly through manual processes. In most cases, the system used can be determined from clues in the online catalog URL. I have also developed tools that message the designated automation contact for a library to verify systems used and implementation dates. When vendors issue announcements about new sales or migrations, the listings for the libraries mentioned are updated. Thirty years of manual attention to maintaining this data has produced a more complete and accurate resource than could have been created through automated processes.
Since library entries from the site are highly visible on the web, errors or omissions are often reported by library workers and community members. These help with the accuracy of the resource, although the overall level of crowdsourcing is minimal compared to the other channels of data collection.
Data Analysis Tools
Library Technology Guides provides many reports, visualizations, and resources to explore trends in library systems. The ILS data includes the dates and products that are currently used and each previous migration. These structures enable many different forms of analysis, including market share reports, which can be segmented by geographic regions, library types, size categories, and other dimensions. Other built-in tools show migration trends, addressing questions such as what products are gaining or losing ground in the library community. Although the data on discovery services does not go back as far as that for ILSs, queries and reports can demonstrate trends related to how these products interrelate. Additionally, the data enables many different analyses that address questions libraries might have as they consider changes in their automation environment.
Proprietary or Open Dataset?
The data underlying the library directory has been collected through a variety of methods over a period of 3 decades. The data was developed to support my research and for the benefit of the broader library community. Libraries make good use of the data to inform their processes to evaluate, select, or retain their core automation systems.
The creation of a unique dataset such as libraries.org poses an interesting dilemma regarding intellectual property. Should it be treated as an open dataset or a proprietary one? I have opted for a hybrid approach. Anyone can access the records through the website, including all of the tools and visualizations that I have made to explore trends and research questions. However, the terms of service prohibit commercial use of the data without some type of agreement. Library Technology Guides offers a commercial membership program inviting organizations that use the site for marketing, competitive intelligence, or related activities to make voluntary contributions to support the site, along with an agreement that they don’t republish the data publicly. The basis of this program is that those that benefit from the dataset commercially should contribute to its sustainability.
Resisting AI Harvesting
The advent of generative AI and its insatiable hunger for data presents new challenges. These services operate aggressive bots that scrape sites and repurpose the data regardless of restrictions and often without attribution. This aggressive scraping poses multiple issues. Incessant swarms of bots hammer the site mercilessly and can cause episodes of degraded performance. This unauthorized scraping also can be considered a parasitic activity that exploits a carefully curated dataset without making any contribution in return.
I have developed an application firewall that offers some protection against bots and other data harvesting agents. The firewall delivers control over which requests are allowed and which are turned away. This approach has provided substantial relief. Not surprisingly, some of the bot swarms use AI to bypass detection methods. The next phase of development of the application firewall will employ more sophisticated ways to detect these patterns.
Ongoing Work
The development and maintenance of Library Technology Guides and the libraries.org directory continue. While its coverage is comprehensive in some countries and strong in others, many global regions and countries remain underrepresented. Filling these gaps will require creating tools for automated data gathering and projects to ingest datasets that might be available.
I also appreciate any assistance from the library community. At a minimum, libraries can review the listings for their own organizations. Other helpful contributions would include identifying other datasets of libraries to expand coverage or overlay existing or new categories of data. I’m always interested in feedback on how this resource could be enhanced to be more useful and to address practical issues libraries face in their technology strategies. |
