CC BY
22
ОБМЕН ОПЫТОМ
Обзорная статья
УДК 027.7:004.738.5(73) ББК 78.65+78.34(7сое)
DOI https://doi.org/10.20913/1815-3186-2020-2-83-95
Discovery layer decisions, configurations and strategies
Raymond Schwartz
William Paterson University, New Jersey, Wayne, USA, e-mail: schwartzr2@wpunj.edu
Raymond Schwartz, Masters of Library Science, Associate Professor, Head of Library Information Systems, William Paterson University of New Jersey, USA
ORCID: 0000-0001-9538-1544 e-mail: schwartzr2@wpunj.edu
Abstract. The problem that patrons frequently encounter with searching via a discovery layer linked to a knowledgebase is the overwhelming number of hits. Few if any research or discussion compares how libraries resolve this difficulty. To begin a discussion, the author disseminated an open invitation for academic libraries to participate on three panels from October 2018 to June 2019 to discuss their implementation processes and decisions. Five academic libraries accepted the invitation. This paper reviews the approaches of the five American academic library systems with the intent to encourage further examination of the issues of design, implementation, and maintenance of discovery layers. The conclusions reached point to the evolving nature of discovery layer interfaces and the struggle for libraries to adapt their needs to this still changing product type. Within this group, the larger institutions with more financial and staff resources have designed a ''bento box'' interface with open source software, while the other institutions with lesser means have chosen to create different instances of their subscribed proprietary discovery layers to meet their specific institutional demands and the specific needs of their patrons. A statistically relevant examination with rigorous comparison of usability testing would have to be done to see how these and other factors play out in the wider field.
Keywords: discovery layers, library discovery service, web-scale discovery, resource discovery service, academic libraries
Citation: Schwartz R. Discovery layer decisions, configurations and strategies. Bibliosphere. 2020. № 2. P. 83-95. DOI: https://doi. org/10.20913/1815-3186-2020-2-83-95.
Received 19.11.2019 Revised 10.03.2020
Accepted 15.04.2020 © Raymond Schwartz, 2020
Решения по представлению поисковых уровней, конфигурации и стратегии поиска
Рэймонд Шварц
Университет Вильяма Патерсона, Нью-Джерси, Уэйн, США, e-mail: schwartzr2@wpunj.edu
Рэймонд Шварц, магистр библиотековедения, доцент, заведующий отделом библиотечных информационных систем, Университет Вильяма Патерсона, Нью-Джерси, США; ORCID: 0000-0001-9538-1544 e-mail: schwartzr2@wpunj.edu
Статья поступила в редакцию 19.11.2019 Получена после доработки 10.03.2020 Принята для публикации 15.04.2020
Аннотация. При поиске через поисковые уровни, связанные с базой знаний, читатели часто сталкиваются с проблемой, о решении которой говорится в немногих исследованиях. Чтобы начать дискуссию, автор распространил анкету среди университетских библиотек с просьбой принять участие в трех панельных дискуссиях с октября 2018 г. по июнь 2019 г. для обсуждения того, как они решают эту проблему. Приглашение приняли пять библиотек университетов. В статье рассматриваются подходы этих пяти американских университетских библиотечных систем, чтобы поощрить дальнейшее изучение ими вопросов проектирования, внедрения и ведения интерфейсов для поисковых уровней. Полученные выводы указывают на эволюционирующий характер интерфейсов поисковых уровней и борьбу библиотек за адаптацию своих потребностей к этому все еще меняющемуся типу продукта. В рамках этой группы более крупные университеты с большими финансовыми и кадровыми ресурсами разработали интерфейс «bento box» с программным обеспечением с открытым исходным кодом, в то время как другие университеты с меньшими средствами решили создать различные варианты для своих прописанных собственных уровней поиска для удовлетворения своих конкретных институциональных потребностей и конкретных потребностей своих пользователей. Необходимо провести статистически значимое исследование с тщательным сравнением юзабилити-тестирования, чтобы увидеть, как эти и другие факторы проявляются в более широкой области.
Ключевые слова: уровни поиска, поиск информации, библиотечная услуга, интернет-поиск, поиск ресурсов, библиотеки университетов Для цитирования: Шварц Р. Решения по представлению поисковых уровней, конфигурации и стратегии поиска // Библиосфера. 2020. № 2. С. 83-95.
DOI: https://doi.org/10.20913/1815-3186-2020-2-83-95.
Introduction
Compared to the online library catalog (OPAC), the discovery layer1 is relatively new in the world of library services. The first discovery layer services were introduced in the early 2000s (Antelman et al., 2006). Some examples of proprietary products are Primo, Summon, WorldCat Discovery and Ebsco Discovery Service (EDS), and open source software options are VuFind and Blacklight. Oya Y. Rieger (Rieger, 2019) states in her issue brief that "while there are disciplinary differences, most researchers prefer using search engines, academic networks, familiar databases, and peer networks to find and filter information, and the library catalog is no longer the primary point for discovering and accessing content" (Blankstein, Wolff-Eisenberg, 2019). Changes in the information environment characterized by information scarcity to one of information abundance has encouraged the devel-
1 The discovery layer has been also referred to as discovery tool,
discovery service, discovery system, web-scale discovery, resource discovery service (RDS) to name a few.
opment of these products. To meet this challenge, discovery layers are central indexes (also known as a knowledgebase) that are web-based, highly customizable and independent of the library management system, be it an integrated library system (ILS) or a library services platform (LSP). In addition to the library's catalog records, discovery layer indexes can include the metadata from various commercial database products (both full text and A&I services), open access collections, institutional repositories, and many more. This combined search of all available databases offers the advantage of leveraging all resources. Discovery layers provide an interface with search and retrieval capabilities, with relevancy ranking and various facets to narrow the results, and interoperability with a link resolver (Breeding, 2015). Many discovery layer products can also limit access to materials by the user's location, the content type, and/or license arrangement.
Given that the technology is still evolving and experience is limited, decisions abound when implementing a discovery layer. Does the library eliminate the OPAC or keep it alongside with the discovery
layer? Or does the library use one discovery layer to access all their materials? Do they select multiple discovery layer products, or create different instances of the same product? And if a library does have more than one discovery layer or several instances of the same product, which parts of the collection are targeted? Examples such as separating search options for the catalog, or various types of collections, be it music, the sciences, research datasets, or institutional collections. And how have they reached their decisions? What research was done to inform the process? Did their selection of a knowledgebase affect their decision? Then how were the discovery layers implemented? And most importantly, how did the libraries assessed their impact? Which usability testing techniques did they use? Did they also use surveys and focus groups? When did they access the impact and how did they implement changes?
At William Paterson University, we were unsatisfied with the functionality of our online library catalog Webvoyage - the OPAC module of Voyager. So in 2011, we implemented VuFind as our OPAC. Then in 2014, we implemented Summon as our discovery layer, and kept VuFind as the interface to our catalog. In the summer of 2015, we switched from Summon to EDS, still keeping VuFind as our OPAC. Early last year, we signed a contract to subscribe to Ex Libris' Alma and Primo. This summer, we replaced our Voyager system and VuFind with Alma and Primo. Prior to and during our Primo implementation, I organized panels at three different conferences from October 2018 to June 2019 for various library to discuss their implementation processes and decisions2. Five academic libraries accepted my invitation to participate. This paper reviews the discovery layer planning, implementation, and current assessment and maintenance practices at these five institutions. They are the New School of New York City, Ball State University of Muncie, Indiana, Skid-more College of Saratoga Springs, New York, the University of Michigan of Ann Arbor, and Temple University of Philadelphia.
The Schools and their Configurations
The institutions range in size from small to large, urban to college town, and research to teaching institution. The New School has about 13,000 students of which 10,544 are enrolled in a degree program
2 Discovery layer decisions and configurations - a panel discussion. Ex libris Northeast user group (ENUG) annual conference 2018, Albany, New York, USA. URL: http://e-nug.org/ (accessed: 27.03.2020) ; Discovery layer decisions and configurations: a panel discussion. Ex Libris Users of North America (ELUNA) annual meeting 2019, Atlanta, Georgia, USA. URL: http://documents.el-una.org/1940/ (accessed: 27.03.2020) ; LITA Heads of Library Technology Interest Group presentations. American Library Association annual conference 2019, Washington, USA. URL: http://www.ala.org/lita/lita-highlights-2019-ala-annual-conference (accessed 27.03.2020).
(7,436undergraduates, 3,108 graduates), 409 fulltime faculty and 1,837 part-time, and degree programs from certificate to doctorate. They are part of a consortium with New York University, Cooper Union, the Brooklyn Historical Society, the New York Historical Society, and the New York School of Interior Design. Ball State has about 22,000 students of which 16,160 are undergraduate and 5,724 are graduate. They have over 1,046 faculty and have degree programs from associate to doctorate. Skidmore College has about 2,600 studentsall are undergraduates. They have over 320 faculty and offer bachelors degrees. The University of Michigan has over 46,000 students of which 30,318 are undergraduate and 16,398 are graduate. They have over 8,000 faculty of which 6,706 are full-time and over degree programs from bachelors to doctorate. Temple University has near 40,000 students of which 29,484 are undergraduate and 10,256 are graduate. They have almost4,000 faculty, of which 2,296 are full-time. They offer degrees from certificate to doctorate.
New School and Ball State University are Ex Libris customers. The New School uses Primo as their discovery layer, while Ball State University uses Summon. Both libraries emphasize the 'search all' feature over a limited catalog search. Skidmore College uses Ebsco Discovery Service and emphasizes the library catalog over the metasearch functions. The two large universities use open source solutions linking to the Summon Knowledgebase. The University of Michigan developed a homegrown solution with a variety of open source software, and Temple University relies heavily on the open source software Blacklight. They both developed an interface deploying a 'bento box' design - referred to as such because of its resemblance to the shape of Japanese lunch boxes (Singley, 2016).
The New School
The New School defined their functional requirements as 1) they should have a single discovery product and a single search box, 2) keep the search within the discovery layer as much as possible, 3) the layer should display as much of the collection as possible, 4) the Library should be able to control and tune the search experience, 5) linking to interlibrary loan, holds and article requests should be directly from the discovery layer, and 6) it should support ^ microservices interactively. Their decision in select- O ing which product was made within the constraints ]□ of their consortium. q
The New School's single search box decision3 i is based on the changes in what they collect. Eighty percent of their physical collection is stored off-site. q
3 The New School libraries and archives. URL: https://library. newschool.edu/ (accessed 27.03.2020). 85
Table Data from the Integrated Postsecondary Education Data System, National Center for Educational Statistics, United States Department of Education. Retrieved November 2019*
Таблица Данные, полученные из интегрированной системы данных о послевузовском образовании, Национальный центр статистики образования, Министерство образования США на ноябрь 2019*
College Колледж Total Students Всего студентов Total full time students Всего студентов-очников Total part time students Всего студентов-заочников Total undergraduates Всего студентов последнего курса Total graduates Всего выпускников Total full time faculty Всего очных факультетов Total part time faculty Всего заочных факультетов City Size Размер города Discovery Layer Поисковые уровни Knowledgebase База знаний
Ball State 21,884 15,907 5,977 16,160 5,724 1,046 253 Citysmall Summon Summon
New School 10,544 9,228 1,316 7,436 3,108 409 1,837 City-large Primo Primo Central
Skid-more 2,612 2,612 - 2,612 - 298 71 Suburb- small EDS EDS
Temple 39,740 27,621 2,932 29,484 1,069 2,296 1,554 City-large Custom Summon
U of Michigan 46,716 44,191 2,525 30,318 16,398 6,706 1,544 City-midsize Custom Summon
* https://nces.ed.gov/collegenavigator, and from interviews with the libraries
* https://nces.ed.gov/collegenavigator и полученные из интервью библиотек университетов
In 2016, they implemented demand driven acquisitions (DDA). As much of their 'collection' is either electronic or physically stored in a variety of locations, their goal was to move from strictly owned to access as needed, display availability and wait times, integrate as many other services into the discovery layer as possible, enable to explore other parts of the collections and build relationships between those items. Their assessments show a marked increase in requesting
of material and that labeling in the system is critical for patron use of the system.
As displayed in the figure below, the first tab is for searching the entire collection, when the other two tabs are subsets, visual resources and digital collections from the archives respectively.
Once a search has been performed, aside from the facets to refine the results, along the top of the screen the results page contains five options, 1) new search, 2) search for journal titles, 3) search specific databases
ш о z
Ш GC Ш CL X Ш
LL
О
Ш
о
z <
X
о
X ш
Fig. 1. Opening search page for the New School Рис. 1. Начальная страница the New School
Fig. 2. Search results from the New School via Primo Рис. 2. Результаты поиска в the New School с использованием Primo
for more specialized content via Libguides, 4) search for newspaper content, and 5) browsing via specific fields such as subject or author through Primo. Their default search is a blended search, of articles, con-sortial holdings, and their own local holdings. While their local title count is less than 300,000, their affiliation with NYU exposes millions of titles potentially requestable through their borrowing network, PAL-Cl's EZBorrow resource sharing network (fig 1, 2).
Ball State University
Ball State's requirements were 1) they should have a single discovery product and a single search box, and 2) ease of maintenance. Ball State does not perform assessments of their patrons' satisfaction with search results, though they have an internal Alma Summon stakeholders group consisting of 15 members from all functional areas of the library.
With figure 3 below, we can see that they have taken the one search box approach4. Links to more specific tools are located lower on the page and under the links in the "Do Research" and "Use Services" boxes figure 4.
Their results page, as seen in figures 5 and 6, has some modifications depending on the search terms. For example, in figure 6, we see database recommendations, research guides, and the reference librarian's contact information included in the results, whereas in figure 5, they are not included. Also the Journal Search and Citation Linker links are available in the upper right corner of the page, if the patron logs into their library account.
Skidmore College
Skidmore's requirements were 1) a Google-like search experience, and 2) minimal customization. Originally the order of the tabs of the opening page were SearchMore, Subject Guides, Journals, Library Catalog, and Site Search. However, after working with the system, the librarians reported the results of the discovery layer were too overwhelming and that books and media materials were lost in the results. In addition, the lack of clarity in how the
4 OneSearch. URL: https://bsu.summon.serialssolutions.com (accessed 27.03.2020).
Fig. 3. Opening search page from Ball State Рис. 3. Начальная страница Ball State
Prospective Students Current Students Faculty / Staff Alumni Parents
Directory Contact Us Apply
BALL STATE Search bsu.edu Q Menu
University Libraries f @ » A
Do Research
RESERVE A ROOM
VIEW HOURS
Home > AcdäemiCi > University Libraries > Do Research
University Libraries
DO RESEARCH
Architecture Library
Archives and Special Collections
Drawi ngs & Documents Archive
Educational Technology & Resources Collection
GIS Research & Map Collection
Health Library
Music Collection
Reference Collection & Government Publications
OneSearch
find Articles, Books, Music, DVDs, and More
Advanced Search
IqGD
О
ASK A LIBRARIAN
II и и
= = =
RESEARCH GUIDES
О
ПП
"TT
WORKSHOPS
SERVICES VISIT AND STUDY
University Libraries
MORE HELP
I Course Reserves I Search WorldCat I Journal Search ■ Online Reference I Libraries' FAQ I Research Consultations I Software Locator
LOCAL RESOURCES
■ Ball State Theses and Dissertations
■ Cardinal Scholar
■ Digital Media Repository
Fig. 4. "Do Research" page from Ball State Рис. 4. Страница «Выполняем исследование» в Ball State
ш О Z ш ОС ш
CL X ш
LL
о
ш
о
Z
с
X
о
X ш
Fig. 5. Search results from Ball State via Summon Рис. 5. Результаты поиска в Ball State с помощью Summon
Home Course Reserves
Log in for access to all Ball State OneSearch content.
OneSearch
Q New Search Advanced ▼
Ш 0
Fig. 6. Search results with additional options from Ball State via Summon Рис. 6. Результаты поиска с дополнительными опциями в Ball State с помощью Summon
search relevancy algorithms worked was a factor. Hence many of the librarians do not use the discovery layer in their instructions. In general the students are directed to use Subject Guides to choose a particular database to perform their searches. Despite this practice, the Library's usability studies and tracking via Google Analytics show that both systems are used at an equal rate. In the end, Skidmore emphasizes the physical resources over the electronic - thus placing the
link to the catalog as the first tab and the discovery layer as the third tab5.
In figure 7, the first tab "Library Catalog" links to Webvoyage - the OPAC module of Voyager. While the Subject Guides tab links to Libguides, the SearchMore tab links to EDS, the Journals tab links to EDS Publication Finder, and Site Search links to a google search of Skid-more College Library's webpages (figure 8, 9).
My Accounts v
Hours today: 7:30 am -1:00 am More Hours
Fig. 7. Skidmore Library's Opening Page Рис. 7. Начальная страница в библиотеке Skidmore
5 Skidmore College Library. URL: http://lib.skidmore.edu/li-
brary/ (accessed 27.03.2020).
SKIDMORE
COLLEGE Catalog
Library Home Course Reserves Journals WorldCat Login New Search My Account My List My Searches
Fig. 8. Search results from Skidmore College via WebVoyage Рис. 8. Результаты поиска в Skidmore College с помощью WebVoyage
Fig. 9. Search results from Skidmore College via EDS Рис. 9. Результаты поиска в Skidmore College с помощью EDS
University of Michigan
University of Michigan's requirements were 1) to improve the user experience for all users -with emphasis on 'all kinds of users', 2) reduce instruction complexities, 3) modernize infrastructure - hopefully 'future-proofing' the architecture and 4) manage the interface - not the data. With a department of developers, they opted for an open source software solution 6
using a combination of Solr, Blacklight, Ruby on Rails, React.js, Citation Style Language, and many others 7. This direction allowed them to design a 'bento box' display of the 'everything' search results. Though like the other libraries they do have tabbed options for searching, the 'bento box' display breaks down the results in discrete sections. They felt that some users slanted toward novice researchers and would have like to have a breadth of resources available.
6 GitHub. University of Michigan Library. URL: https://github. com/mlibrary (accessed 27.03.2020).
7 University of Michigan Library Technical overview. URL: https:// search.lib.umich.edu/technical-overview/ (accessed 31.10.2019).
While others have needs that focus more narrowly. The assessments they perform are rigorous and often, ensuring that areas worked on have led to improvements and also identify areas that need future work.
Below one can see that the discovery layer 'everything' search option is first, followed by the
catalog, an articles only search, a database listing search, an online journal title search, and a search of the Library's website - all through the home grown interface8 (fig. 10).
The search 'everything' breaks down the results within five boxes (fig. 11, 12).
Ml LIBRARY Search My Account MyFa\
Q. Search | Advanced
Everything Catalog Articles Databases Online Journals Library Website
Search Everything to see a broad sampling of results from across 'Library Search' and to explore specific areas and records in greater detail.
You will see results from the Catalog, Articles, Databases. Online Journals, and Library Website pages.
Enter a search term in the search box to start your own Everything search.
Fig. 10. Opening search page for the University of Michigan Рис. 10. Начальная страница в the University of Michigan
Fig. 11. Bento Box search results display for University of Michigan via Blacklight, etc. Рис. 11. Результат поиска с помощью Bento Box в Мичиганском университете с помощью Blacklight и т. д.
8 University of Michigan Library search. URL: https://search. lib.umich.edu/everything (accessed 27.03.2020).
Ш О Z ш ОС ш
CL X ш
LL
о
ш
и
Z <
X
о
X ш
Ml LIBRARY Search My Account My Favorites Login
smith Advanced
Everything Catalog Articles Databases Online Journals Library website
Logjn tOMCV ы complete search results and to s ave and request ¡terns.
Articles: results from scholarty journal articles, newspaper articles, book chapters, conference proceedings, and more.
Filter your search
Q Articles from scholarty journals only
D Exclude newspaper articles [] Available Online D U-M library materials only
1 to 10 of 42,704,222 Articles results □ Smith
Sort by Relevance
Save to Favorites
Format Date
Pu Wished
S Reference 15/6/2018 Writers Directory
Publication Date
2010 - present 2000-2009 past-1949 1990-1999 1980-1989
15,800344 10,429,480 6,621,403 5,071,974
1,952,622
к Availability • 1 item
Action
Go to item
Description О Full text available
improving access
Full text bnk not working? Report a problem.
□ Smith
' Save to Favorites
Shawril a Publication Date: ilrr. Format
Newspaper Article 29,4
Format Date
Published
S Reference 2010
Acronyms, nraalisms ¿Abbreviations Dictionary VOL 4
Fig. 12. Article search results from University of Michigan via Blacklight, etc. Рис. 12. Результаты поиска статей в Мичиганском университете с помощью Blacklight и т. д.
Temple University
Temples requirements were similar to the University of Michigan's. Originally, they had two separate systems, Millennium for the catalog and Summon for article searching. Like the University of Michigan, they also have a department of developers. They have deployed Blacklight and designed a 'bento box' display of their search results. Their assessment studies and click tracking led to the one discovery system and the single search box.
Similar to the University of Michigan's interface, their bento box results display in five boxes - books and media, articles, databases, journals, and library website searches. Unlike the University of Michigan, only three of the five separate searches are available on the Library's opening page - books and media, articles, and databases as shown in figure 13. The bottom of the first column has a link to "Search our collections", that opens up an initial search page9 which includes the five direct searches (fig. 14, 16).
Fig. 13. Opening Page for Temple University Libraries Рис. 13. Начальная страница библиотек университета Temple
9 Temple University Libraries search. URL: https://library-search.temple.edu/ (accessed 27.03.2020).
Search Everything to discover results in Books & Media. Articles. Databases Journals, and Website What is Library Search?
Library search makes it easier to discover all that Temple Libraries has to offer. Use Library Search to find books, scholarly articles, databases, newspapers, archival material, images, streaming media, and much more.
Need help using Library Search?
Try our Contact the libraries service.
Want more information on Library Search?
Refer to our Frequently Asked Questions
Fig. 14. Library Search Page for Temple via Blacklight Рис. 14. Поисковая страница библиотек университета Temple с помощью Blacklight
Fig. 15. Bento Box search results display for Temple University via Blacklight Рис. 15. Результаты поиска через интерфейс Bento Box в библиотеках университета Temple с помощью Blacklight
Fig. 16. Article search results from Temple University via Blacklight Рис. 16. Результаты поиска статей в университете Temple с помощью Blacklight
О н
_Q IZ
О
х ш
ш
О
Conclusions
Academic libraries have shifted from an environment of information scarcity to information abundance to the point where the catalog is no longer the primary link to the library's resources. And with that abundance, the common concern is the overwhelming number of results displayed in a discovery layer search. A concern for which each library has developed different solutions to accommodate the various types of searching. The decision to emphasize the catalog or a 'search all' box varies from institution to institution. Given local conditions, organizational culture, and past history with handling technical transitions with patron responses, many academic libraries have grappled with the transition in a myriad of ways.
All choices of the five libraries are mostly based on functional requirements and on the resources available to the institution - not only budget funds but also the ability to manage technical debt. The New School, Skidmore and Ball State made the decision to go with an industry product rather than a custom built one (Primo, EDS, and Summon respectively). The industry products 'out of the box' solutions do allow customization and integrations to external systems, of which both the New School and Ball State have taken advantage. The New School, for example, heavily customized their user interface because they had some development resources and chose to focus on integrations with external systems such as Atlas-System's Illiad, Ares and OCLC's D2D
softwares (Rose, Jones, 2019). Skidmore has kept the functionality of the catalog, the discovery layer and other systems separate. This arrangement may have to do with Skidmore being solely an undergraduate liberal arts college and the library's organizational culture. The University of Michigan and Temple have large departments of developers and have chosen a custom built solution using open source software. The option to offer a bento box display of search results is only available through a customization of open source software - no commercial product offers it. In the future, it remains to be seen whether
there will be enough of a perceived need for such
an interface to motivate the commercial industry to offer it. Not all institutions have fully implemented a rigorous program of assessing their tools as the University of Michigan and Temple University. Where Ball State does not systematically assess their users' experiences, the New School does have targeted assessments that inform the design of their systems, and Skidmore's usability testing has not to a great extent informed the design of their systems.
Though this particular group of libraries is not statistically representative of the range of discovery layer implementations, it does lay out the pressures, constraints, and affordances that are part of each of the libraries' environments. Further examination with rigorous comparison of usability testing would have to be done to see how factors such as size of the institution, discipline areas, and undergraduate versus graduate versus faculty research are playing out in the wider field.
References
Antelman K, Lynema E and Pace AK (2006) Toward a 21st century library catalog. Information Technology and Libraries 25(3): 128-139. Blankstein M and Wolff-Eisenberg C (2019) Ithaka S+R US faculty survey 2018. Ithaka S+R. URL: https:// doi.org/10.18665/sr.311199 (accessed 12.04.2019). DOI: https://doi.org/10.18665/sr.311199. Breeding M. (2015) The Future of library resource discovery. Information Standards Quarterly 27(1): 24-30. DOI: https://doi.org/10.3789/isqv27no1.2015.04.
Bibliography
Aharony N and Prebor G (2015) Librarians' and information professionals' perspectives towards discovery tools - an exploratory study. Journal of Academic Librarianship 41(4): 429-440. DOI: https://doi.org/10.1016/j.acalib.2015.05.003.
Bonner S and Williams G (2016) A small academic library and the power of EBSCO discovery service. Serials Review 42(3): 187-191. DOI: https://doi.org/10.1080/ 00987913.2016.1205428.
Rieger OY (2019) What's a Collection Anyway? Ithaka S+R. URL: https://sr.ithaka.org/publications/whats-a-collection-anyway/ (accessed 12.04.2019). DOI: https:// doi.org/10.18665/sr.311525.
Rose K and Jones A (2019) Unfulfilled: a story of users and library request services. Journal of Inter-Library Loan, Document Delivery and Electronic Reserves 27(4/5): 77-107. DOI: https://doi.org/10.1080/1072303X.2019.1579152.
Singley E (2016) To bento or not to bento - displaying search results. Usable libraries. URL: https://emilysingley. net/usablelibraries/to-bento-or-not-to-bento-displaying-search-results/ (accessed 26.10.2019).
Boyer G, Bacon V and Hill K (2014) Creating a cohesive discovery service. Serials Review 40(3): 200-202. DOI: https://doi.org/10.1080/00987913.2014.949549.
DeMars JM (2017) Discovering our users: a multicampus usability study of primo. IFLA WLIC-2017, Wroclaw. URL: http://library.ifla. org/1810/1/S10-2017-demars-en.pdf (accessed 12.04.2019).
Dempsey M and Valenti AM (2016) Student use of keywords and limiters in web-scale discovery
searching. Journal of Academic Librarianship 42(3): 200206. DOI: https://doi.org/10.1016/j.acalib.2016.03.002.
Dogan G and Dogan SC (2013) Evaluation of web discovery services: reflections from Turkey. Procedia - Social and Behavioral Sciences 73: 444-450. DOI: https://doi.org/ 10.1016/j.sbspro.2013.02.074.
Foster AK (2018) Determining librarian research preferences: a comparison survey of web-scale discovery systems and subject databases. Journal of Academic Librarianship 44(3): 330-336. DOI: https:// doi.org/10.1016/j.acalib.2018.04.001.
Hanneke R and O'Brien KK (2016) Comparison of three web-scale discovery services for health sciences research. Journal of Medical Library Association 104(2): 109-117. DOI: https://doi.org/10.5195/JMLA.2016.52.
Johnston L and Brown C (2017) It takes two to tango... Public services and information technology staff collaborate to launch a new discovery system. IFLA WLIC-2017, Wroclaw. URL: http://library.ifla. org/1793/ (accessed 12.04.2019).
Kumar V (2018) Selecting an appropriate web-scale discovery service: a study of the big 4's. DESIDOC Journal of Library & Information Technology 38(6): 396402. DOI: https://doi.org/10.14429/djlit.38.6J2860.
Lee B and Chung E (2016) An analysis of web-scale discovery services from the perspective of user's relevance judgment. Journal of Academic Librarianship 42(5): 529-534. DOI: https://doi.org/10.1016/j. acalib.2016.06.016.
Marvin S (2017) Voyager to ALMA - space to soul process for discovery systems adaptation to changing search environments. IFLA WLIC-2017, Wroclaw. URL: http://library.ifla.org/1733/ (accessed 12.04.2019).
Minkin RM and Tobias C (2017) Bridging the (unit) divide: applying user experience to a discovery layer. IFLA WLIC-2017, Wroclaw. URL: http://library.ifla. org/1817/ (accessed 12.04.2019).
Narayanan N and Byers DF (2017) Improving web scale discovery services. Social Science Research Network pap. ID 3139357. URL: https://papers.ssrn.com/ abstract=3139357 (accessed 12.04.2019).
O'Connell R (2019) Beyond the bento: a new discovery experience at smith college. Information Today. URL: http://www.infotoday.com/cilmag/mar19/ OConnell-Beyond-the-Bento-A-New-Discovery-Experience-at-Smith-College.shtml (accessed 26.10.2019).
Rochkind J (2012) Article search improvement strategy. Bibliographic Wilderness. URL: https:// bibwild.wordpress.com/2012/10/02/article-search-improvement-strategy/ (accessed 26.10.2019).
Rodríguez-Bravo B and Frías J-A (2017) Resource discovery services versus OPACs in information searching. IFLA WLIC-21017, Wroclaw. URL http:// library.ifla.org/1816/ (accessed 26.10.2019).
Shapiro S (2018) Academic librarians, information overload, and the Tao of discovery. Journal of Academic Librarianship 44(5): 671-673. DOI: https://doi.org/ 10.1016/j.acalib.2018.07.004.
Spezi V, Creaser C and Conyers A (2015) The impact of RDS on usage of electronic content in UK academic libraries: selected results from a UKSG-funded project. Serials Review 41(2): 85-99. DOI: https://doi.org/10.10 80/00987913.2015.1035991.
Varnum K (2017) Unifying the library discovery experience. University of Michigan Library blogs. URL: https:// www.lib.umich.edu/blogs/library-tech-talk/unifying-library-discovery-experience (accessed 26.10.2019).
Wang X, Cui Y and Xu S (2018) Evaluating the impact of web-scale discovery services on scholarly content seeking. Journal of Academic Librarianship 44(5): 545552. DOI: https://doi.org/10.1016Zj.acalib.2018.05.010.
Welborn A (2014). Bento searching is here! Duke University Libraries blogs. URL: https://blogs.library. duke.edu/blog/2014/08/21/bento-searching/ (accessed 26.10.2019).
Wells D (2016) Library discovery systems and their users: a case study from Curtin University Library. Australian Academic & Research Libraries 47(2): 92-105. DOI: https://doi.org/10.1080/00048623.2016.1187249.
Zabicka P (2017) Knihovny.cz - building a discovery portal for Czech libraries. IFLA WLIC-2017, Wroclaw. URL: http://library.ifla.org/1814/ (accessed 26.10.2019).
O H _Q IZ O x
LU
S U3
O
