User Evaluation of Searchling and T-saurus: Multilingual Thesaurus-Enhanced Visual Interfaces for Digital Libraries / Évaluation par les usagers de Searchling et de T-saurus : Les interfaces visuelles à thesaurus multilingue pour les bibliothèques numériques

Abstract

The objective of this paper is to report on a comparative user evaluation of two multilingual thesaurus-enhanced visual user interfaces—namely, T-saurus and Searchling, designed and developed for digital libraries. The study used 25 academic users carrying out three search tasks on both user interfaces. It applied usability and affordance strength questionnaires, interviews, think-alouds, and direct observation to investigate users’ evaluation of the key components of both user interfaces—namely, multilingual features, thesaurus and search functions, and visualization and visual appeal. Results of the study show that users were able to successfully carry out the search tasks using thesaurus-enhanced search interfaces. However, they preferred Searchling, a faceted search user interface, for its flexible language option, thesaurus browsing, and visualization. Users preferred the interface to show the thesaural relationships along with the selected term without interacting with the interface. The empirical data gathered and the design ideas implemented within the two user interfaces will be useful for designers of search interfaces that make use of thesaurus and multilingual features. The design and methodological framework of the developed user interfaces and the study could be used to create more information-rich multilingual academic search user interfaces for teaching and learning purposes. The design and representation of thesaural terms and structures in these two interfaces are designed using novel visualization techniques.

Résumé

L’objectif de cet article est de rendre compte de l’évaluation comparative effectuée par des utilisateurs de deux interfaces utilisateur visuelles multilingues, comportant un thésaurus, nommément T-saurus et Searchling, conçus et développés pour les bibliothèques numériques. L’étude a mis à contribution 25 universitaires effectuant trois tâches de recherche sur chacune des deux interfaces utilisateur. Des questionnaires sur l’utilisabilité et sur la force d’affordance, des interviews, des séances de protocole verbal (thinkaloud) ont été utilisés, ainsi que l’observation directe pour rendre compte de l’évaluation par les utilisateurs des composants clés des deux interfaces utilisateur, nommément les caractéristiques multilingues, les fonctions de thésaurus & recherche, l’attrait du mode de visualisation. Les résultats de l’étude indiquent que les utilisateurs ont été en mesure de mener à bien les tâches de recherche en utilisant les interfaces de recherche enrichies par thésaurus. Toutefois, ils ont préféré Searchling, une interface usager à facettes, pour la souplesse de son option de langue, et pour la navigation et la visualisation de son thésaurus. Les utilisateurs ont préféré que l’interface montre les relations de thésaurus en même temps que le terme sélectionné sans qu’ils aient à interagir avec l’interface. Les données empiriques recueillies et les idées de conception mises en œuvre dans les deux interfaces utilisateurs seront utiles pour les concepteurs d’interfaces de recherche qui utiliseront des thésaurus et des fonctionnalités multilingues. La conception et le cadre méthodologique des interfaces utilisateur développés et cette étude pourraient être utilisés pour créer d’autres d’interfaces utilisateur de recherche universitaire multilingues riches en information aux fins de l’enseignement et de l’apprentissage. La conception et la représentation des termes de thésaurus et des structures dans ces deux interfaces sont conçues en utilisant des techniques de visualisation nouvelles.

Introduction

Highly interactive and dynamic user interfaces for exploratory browsing and searching of digital information collections have been the focus of several recent studies. White et al. (2006) note that in exploratory search, users generally combine querying and browsing strategies to foster learning and investigation. Marchionini (2006) argues that to engage people more fully in the search process and put them in continuous control, researchers are devising highly interactive user interfaces. These techniques are especially good for exploration where high-level overviews of a collection and rapid previews of objects help people to understand data structures and infer relationships among concepts. Semantically rich user interfaces have the potential to assist users in formulating queries, forming context for a particular search, and exploring and gaining a comprehensive view of collections. Wilson et al. (2010) use the notion of semantic zooming as a strategy that supports the user at information-retrieval and information-seeking levels of search in that the technique allows users to investigate deeper into a topic as they “zoom” into it. Providing useful semantic assistance, particularly through visualization, within user interfaces of digital libraries requires research into the type of visualization and the associated features to support users in the exploration, searching, and browsing of the collection.

Recently, several information-retrieval user interfaces, online public access catalogues, and web search engines have implemented visual, dynamic taxonomies and faceted structures to engage and maintain users and to support them in carrying out successful searches. Many commercial database and digital library vendors provide thesaurus-based search options as a separate procedure or step, or as an advanced search functionality. We argue that thesauri can be integrated into search interfaces to provide a seamless and highly interactive environment for users to explore the collection in the process of formulating and reformulating queries. There is, however, little research on how users may interact with thesaurus-enhanced search interfaces and what types of visualizations users would prefer to interact with when it comes to thesauri, taxonomies, and categorized views.

In this paper we report empirical evaluation of two visual, exploratory user interfaces that take advantage of dynamic views supported by the UNESCO (United Nations Education, Scientific, and Cultural Organization) multilingual thesaurus supporting English, French and Spanish languages. The key features of the two interfaces that we have developed are (1) combining searching and browsing; (2) supporting dynamic exploration of the conceptual structure of a thesaurus; (3) providing dynamic term-relation features to give high-level overviews of the terms and the collection; (4) supporting multilingual search and retrieval within the UNESCO digital collections; and (5) using a novel technique to implicitly show thesaural relationships using colour, size, and distance. A comparative user evaluation of the two user interfaces was carried out to examine the multilingual and visual interface features and functionalities that support users in exploring semantic information, formulating queries, and interacting with digital information. The results from the study contribute to our understanding of the factors affecting users’ interaction with visual user interfaces that provide thesaural semantic support for query formulation and information exploration.

Related work

To provide a context for the presentation of our study, this section presents a review of the literature related to the three key facets that form the underlying theoretical framework for this study. These key facets are the development of visual user interfaces for digital libraries, the use of thesauri and taxonomies in search user interfaces to support exploratory search, and the evaluation of thesaurus-enhanced search user interfaces.

Visual interfaces to digital libraries have recently received widespread attention. Börner and Chen (2002) identify three usage scenarios for visual interfaces to digital libraries: (1) support the identification of the composition of a retrieval result, understand the interrelation of retrieved documents to one another, and refine a search; (2) gain an overview of the coverage of a digital library to facilitate browsing; and (3) visualize user-interaction data in relation to available documents in order to evaluate and improve digital library usage. Zaphiris et al. (2004) explore the application of information visualization in digital libraries and identify three key tasks in digital libraries—namely, searching, browsing, and navigation to which information visualization can make a contribution.

The use of taxonomies in creating exploratory user interfaces has received particular attention. Sacco (2000) argues that dynamic taxonomies perform equally well in the locator phase (easing both query formulation and the inspection of large result lists) and in the navigation phase, where they provide a systematic, concise view of all the existing relationships among information atoms, including relationships not easily perceived by the information base builder. Using the notion of explicit and mined metadata, Papadakos et al. (2009) developed a user interface that takes advantage of dynamic and faceted taxonomies to support exploratory search and interaction behaviour. Other studies have made use of taxonomies and facets to design and develop exploratory user interfaces (Kules et al. 2008; Yee et al. 2003).

Several digital libraries and online initiatives have incorporated knowledge organization systems such as thesauri and classification systems into their user interfaces to provide support for query formulation, collection browsing, and other exploratory searching and browsing tasks (Shiri and Molberg 2005). A team led by A. S. Pollitt (Pollitt, Ellis, and Smith 1994a, 1994b; Pollitt, Smith, and Braekevelt 1996a; Pollitt, Treglown, and Braekevelt 1996) designed the HI-BROWSE (High Resolution Interface for Database Specific Browsing and Searching) as a thesaurus-enhanced search interface. The system was implemented as a front end to several bibliographic databases including MEDLINE, In-spec, EPOQUE (European Parliament Online Query System) and Embase. While MenUSE provides sequential access to menus for query formulation, the HIBROWSE system presents a multi-window search interface with the thesaurus at the centre. The interface presents views into the database through navigable hierarchies of subject descriptors from thesauri and provides simultaneous access to different bibliographic fields. One of the notable features of the HI-BROWSE interface is its ability to simultaneously show the thesaurus terms associated with two or more search facets in a single view. The interface allows navigating up and down the hierarchy for further query refinement. While both MenUSE and HIBROWSE have extensively used thesaurus knowledge, the HI-BROWSE system provides more dynamic and multidimensional access to the content of the databases.

Several user interfaces have used graphical as well as two- or three-dimensional category hierarchies using the MeSH thesaurus—for instance, TraverseNet (McMath, Tamaru, and Rada 1989), MeSHBrowse (Korn and Shneiderman 1995), Cat-a-cone (Hearst and Karadi 1997), Visual MeSH (Lin 1999), and the Integrated Thesaurus-Results Browser (Sutcliffe, Ennis, and Hu 2000). Other researchers have designed visual user interfaces incorporating subject headings (Leide et al. 2003) and the Government of Canada Core Subject Thesaurus (Stafford et al. 2008).

There have been several thesaurus-enhanced visual user interfaces that have been subject to evaluation. An evaluation of Déjà vu (Gordon and Domeshek 1998), a thesaurus-enhanced search interface developed for digital libraries which takes advantage of the Library of Congress Thesaurus for Graphical Materials, showed that the process of browsing through the thesaurus terms in Déjà vu improves users’ understanding of the relationship between the archive materials and the cataloguing resources. Sutcliffe, Ennis, and Hu (2000) evaluated users’ interaction with a thesaurus and results browser, and they found that better searchers used the visualizations more effectively and spent longer on the task, whereas poorer performances were attributable to poor motivation, difficulty in assessing article relevance, and poor use of system visualizations. Blocks et al. (2002) have reported the development of a prototype search interface enhanced with the Art & Architecture Thesaurus. The interface contains a thesaurus browser for users to access the hierarchies of the thesaurus, a search facility that maps an initial search term to the vocabulary terms, and a section showing the retrieved results. A formative evaluation of the interface showed that the interface was successful in allowing a person with little knowledge of the interface to make use of its functionality. However, the prototype interface did not provide non-expert searchers with sufficient guidance on query structure or when to use the thesaurus within the search process. Blocks, Cunliffe, and Tud-hope (2006) developed a reference model for user-thesaurus interaction that features the specific stages of the search process. They noted that the use of controlled vocabulary tools could assist in some aspects of the search—for example, refining query terms or analyzing results. In Stafford et al., (2008) we reported a user-centred evaluation of Searchling, a bilingual visual user interface, and found that integrating search and browsing features was particularly useful and that the semantically enhanced visual interface was most useful at the beginning of a research project on an unfamiliar topic, because users could start by browsing through general categories for relevant terms and the thesaurus could help them narrow or broaden their search. The study reported here draws upon Stafford et al. (2008) and evaluates two user interfaces designed and developed for the UNESCO multilingual thesaurus—namely, a new version of Searchling and a new interface called T-saurus. T-saurus was developed to experiment with the visualization of thesaural terms and relationships as well as of the result display. This was a different approach from the design we had adopted for Searchling, which had a faceted structure and no results visualization. The reason for developing T-saurus lies in our evaluation (Stafford et al. 2008) that suggested the simplification of the understanding of thesaural relationships.

There is, however, little research on ways in which users interact with and make use of multilingual thesaurus-enhanced visual user interfaces to support their interactive searching and dynamic browsing and exploration. The research reviewed in this section has focused on thesaurus-based search user interfaces; some interfaces are textual others are visual. Drawing upon this research, our study will introduce two visual user interfaces that we have developed to support multilingual searching, browsing and navigation based on the structure of the UNESCO multilingual thesaurus. Further, we will report on a user-centred and comparative evaluation of these two interfaces.

Research questions

The objectives of this research include (1) to extend our understanding of how users interact with semantically rich multilingual visual user interfaces enhanced with thesauri; (2) to examine how users evaluate and compare the two visual user interfaces developed for this research; and (3) to evaluate the usefulness and usability of user interface features and functionalities. In this study the following specific research questions were addressed:

1. 1. How do users evaluate the multilingual features of Searchling and T-saurus interfaces for query formulation?

2. 2. How do users evaluate the thesaurus and search functions of the two interfaces?

3. 3. How do users evaluate the user friendliness, visual features, and visual appeal of the two interfaces?

Methods

Evaluation of visual user interfaces is a challenging task due to the complexity of visualization; diversity of users, data sets, and tasks; and the measurement approaches and techniques adopted (Ellis and Dix 2006). In this study we used controlled experiments comparing two visual thesaurus-enhanced user interfaces (Plaisant 2004).

User Interfaces

We developed two different visual user interfaces using the UNESCO multilingual thesaurus. They are called Searchling and T-saurus, and their functional prototypes are available at http://markbieber.ca/tsaurus/FinalTsaurus2.html and http://markbieber.ca/tsaurus_old/index.php.

There are several significant differences between these two search user interfaces. The Searchling user interface provides a design similar to a faceted search interface. It uses the high-level facets of the UNESCO thesaurus along with a list of terms for each facet. An earlier version of Searchling was developed, evaluated, and reported in Stafford et al. (2008). In the current research project, we made several changes to Searchling based on our user evaluation, including thesaurus browsing features, search term selection mechanisms, and language options. The T-saurus search interface, which was developed from scratch, takes a different design approach, focusing on the visualization of thesaurus terms, thesaural relationships, and results display. It provides a more visualized and interactive interface where users are invited to interact with the interface to choose thesaural term relationships—such as specifying more general, more specific, or related terms—and to view and refine a visualized view of retrieved results. This T-saurus interface was created to not only integrate searching, browsing, and exploration of thesaurus terms and relationships, but also to experiment and explore result visualization mechanisms and the ways in which users make sense of and understand the interactive visualization of the search and retrieval process.

The theoretical framework for the design of these interfaces draws upon two key principles. The first is the idea of rich-prospect interfaces, in which individual representations of every item in a collection are combined with emergent tools (Ruecker and Chow 2003; Shiri, Revie, and Chowdhury 2002). The second principle is based on the design ideas for thesaurus-based search interfaces, including the following guidelines:

• • providing hierarchical and alphabetical lists to support different strategies;

• • allowing flexible ways of choosing terms;

• • catering for the selection of alternative Boolean operators;

• • providing a search term pool option for making terms available for later use and exploration;

• • integrating thesaurus and retrieved documents displays; and

• • making thesaurus options available in all stages of the search process.

Figure 1 shows Searchling, a user interface that provides the user with the following three spaces within a single screen: the thesaurus space, the query space, and the document space. The thesaurus space is on the left. It includes a browsable side panel of high-level categories next to a list of thesaurus terms. Each term has a number beside it, which indicates how many documents in the collection contain the term. When a term is queried or clicked, it moves to the top of the list and all related terms from the thesaurus appear below it. The table to the right of the thesaurus list indicates related terms that are broader, narrower, preferred, or non-preferred compared to the selected term; the user can also sort by these categories. Finally, there is a language switch at the top of the thesaurus list. The query space is located in the right panel of the screen. Users can search for a single term in the thesaurus by entering it in the query box, choosing a language, and clicking the button labelled “Find in Thesaurus.” If the term is entered in English but the user selects French as the query language, Searchling will search for the corresponding French term; however, the English term will also always be visible as a micro-text satellite below the query term. When users decide to add a term to their query, they do so by checking the box next to it in the thesaurus list, and it is added to the selected-terms list on the lower right-side panel. The document space forms the third section of the screen, running across the bottom. The documents are represented by standard bibliographic information; they can be sorted in various ways, and each item serves as a link to the actual document. The UNESCO digital portal, which consists of 400,000 indexed documents, has been used in this project. Detailed description and a user evaluation of a previous version of this interface are reported in Stafford et al. (2008).

Figure 1.

Searchling Interface

Figures 2 and 3 show the T-saurus search and the retrieved document spaces. The user interface makes use of visual objects, size, colour, location, and zoom-in and zoom-out features to distinguish between various types of thesaurus terms and their relationships. Figure 2 shows a core of visual elements consisting of a set of “buckets” organized in the centre of the screen. The size of a bucket represents the number of matches for a particular term, while proximity and opacity represent the scope and accuracy of the term in relation to pre-established hierarchies for the query: main term, related terms, and more specific, more general, and synonymous terms.

Figure 2.

T-saurus interface

Figure 3.

T-saurus retrieved documents display

The query space is located across the top and on the right side of the screen while the thesaurus space is located on the left and in the centre. Users can search for a single term in the thesaurus by entering it in the query box at the top of the page and clicking the “Find” button. If the term exists in the thesaurus it will appear in the centre of the screen with a number in parentheses beside it, which indicates the number of documents in the collection that include the selected term. Users can also browse all the terms in the thesaurus using the panel on the left, which can be sorted either alphabetically or hierarchically by category. Again, each term has a number beside it in parentheses indicating how many documents in the collection contain the term. When a term in the list is clicked, it will appear in the centre of the screen. When a term is selected by either method, it is represented by a square in the central thesaurus space. By using the checkboxes in the bottom of the right-hand panel, users can choose to view the thesaurus terms that are related, narrower (more specific), broader (more general), and preferred or non-preferred (synonyms) compared to the selected term. These associated terms are also represented in the thesaurus space by squares or diamonds, and their relationship to the selected term is represented by their relative proximity and opacity.

Users can also use the checkboxes in the right-hand panel to show the terms in more than one language at once and to view scope notes for selected terms. When users decide to add a term to their query, they do so by clicking on its square in the centre of the screen, at which time it is added to the summary of terms, or term pool, at the top of the right-hand panel. Users can add as many terms as they like, delete them at any time, choose to keep them in only one language rather than multiple languages, and combine them using the Boolean operators below the list. When they have finished formulating their query they click “Retrieve Documents” to view the results (Figure 3). The red dots in the middle around the green box represent the results retrieved for the chosen term. The green box in the middle shows the thesaurus term and its French equivalent as well as the number of documents indexed using that term.

Participants

Twenty-five participants from the University of Alberta were recruited for this study by purposive, maximum variation and snowball sampling. Although the participant pool included students and faculty members across departments, multilingual volunteers—particularly those from the Department of Modern Languages and Cultural Studies—were specifically targeted throughout the recruitment process. The resulting participant pool was diverse, comprising professors, graduate, and undergraduate students from a variety of disciplines, including applied linguistics, Latin American studies, French language studies, romance languages and literatures, library and information studies, humanities computing, English and film studies, education, chemical engineering, history, political science, and music. The group contained three professors, two doctoral students, seven masters students, and thirteen undergraduates. Twenty-three of these participants were women; two were men. Of these participants, thirteen were bilingual (seven spoke French fluently; two spoke Spanish; four participants respectively spoke Mandarin, German, Latin, and Russian). One participant spoke German (first language), English, French, and Latin. Six of these participants currently conduct research in more than one language. All participants cited the University of Alberta’s library catalogue and online databases as commonly used search tools. While many of the participants have had only a basic working knowledge of online search tools and thesauri, others—particularly those graduate students af-filiated with the School of Library and Information Studies and the Humanities Computing Program—have had extensive experience with databases, digital libraries, and in some cases thesauri.

Data gathering tools and procedure

This study used a wide range of data gathering tools, including pre-search, post-search and usability questionnaires, interviews (see Appendixes 1 and B), the think-aloud technique, and direct observation. Data from the interviews were collected through digital recording and in written form. For the first five to ten minutes of the interview, participants responded orally to a series of questions related to their academic background, the nature of their research, and their preferred online search tools. Participants’ responses were recorded in written format by the interviewer. Next, the participants were given a brief overview of the usability study before being asked to complete three identical tasks on the Searchling interface and T-saurus interface, respectively. The interface used first was alternated between the users, and users would move between interfaces as they completed first task 1 (on either interface), then task 2 (on either), and so forth. During this part of the session, which generally lasted for 25–45 minutes, participants were asked to verbally analyse Searchling and T-saurus using a think-aloud protocol as they interacted with the interfaces and completed the required tasks. The audio think-aloud data was captured, analysed, and triangulated with the data that were gathered through usability and post-search questionnaires. As users completed the tasks, the interviewer pointed out any aspects of Searchling or T-saurus the user had not already explored or seemed to have overlooked. This was important as it gave the interviewer an opportunity to ask the participants about particular features of the interfaces, while also ensuring that participants became familiar with each part of both interfaces. Users were asked to identify whether they perceived themselves as being visual learners or linear thinkers. This question aimed to capture their perceived learning and cognitive style as opposed to actual cognitive style, which could be measured using various cognitive style questionnaires available. Our visual learner / linear thinker question was in line with the questionnaire reported in Mayer and Massa (2003) in which they asked users to indicate, for instance, ‘I prefer to learn visually / I prefer to learn verbally.’ This question was asked to see if each group would show preference for a certain type of user interface. Finally, the interviewees were asked to fill out a comparative post-test questionnaire, which took between 10 and 15 minutes to complete. This questionnaire asked users to respectively rate the two interfaces’ usability and affordance strengths on a five-point Likert scale. Affordance strength (Ruecker, Radzikowska, and Sinclair 2011) provides a framework for comparing different tools in terms of their value offered by the affordances of a specific tool. Participants rated the effectiveness of the interfaces’ different functions (e.g., search and thesaurus functions), user friendliness, visual appeal, and research applicability (particularly for multilingual search and query formulation). Participants were also asked to provide comments and to indicate the interface they preferred.

Tasks

Valiati, Freitas, and Pimenta (2008) note that to be comprehensive, experiments with users interacting with visual user interfaces must rely on a set of tasks that covers the situations a real user will face when using the visualization tool. Also, in a meta-analysis of studies carried out on visual interfaces for information retrieval, Julien, Leide, and Bouthillier (2008) concluded that most user and usability studies have used browsing and searching tasks to evaluate visual user interfaces. In this study we used three tasks to evaluate the usability of the thesaurus, the search function, and the multilingual and exploratory experiences of users. We understand that designed search tasks have their own limitations compared to users’ genuine information search tasks. However, assigned search tasks allow for comparability and control. That is why we have decided to use a combination of tasks, including two assigned search tasks and one task focusing on users’ own research topics. The two assigned search tasks were designed to allow for the evaluation of various features and functionalities of the two interfaces. These features include user interaction with the structure of the thesaurus terms and their relationships, query formulation through the use of Boolean operators and the search box, selection of search terms for expanding or modifying a given search, and the resulting navigation, manipulation, and display. The search tasks used in this study are as follows:

1. 1. Prompt users to interact with the respective thesaurus functions of the two prototypes while searching for documents containing a specific keyword and one of its related terms (task 1). This first task was simultaneously directed (i.e., users searched for a specified keyword) and unfixed (i.e., users were prompted to extend their search by selecting a related term of their choice) to encourage a more sustained and exploratory interaction with the thesaurus space of each interface. While this task emphasized the thesauri features, browsing, search, and document retrieval were also required, thus ensuring users were forced to interact with all three spaces of each interface (i.e., thesaurus, query formulation, and document spaces). The instructions for this task were as follows: “For this task, you would like to use each interface to find out what kind of information the collection holds on democracy and one of its related terms.”

2. 2. Prompt users to interact with the prototype’s multilingual and filtering tools (task 2). Users were required to find and select two different terms from the thesaurus list to retrieve documents containing both terms using the AND Boolean operator, and to narrow results by language. This task was designed to draw attention to the flexibility of the thesaurus features, the Boolean operator functions, and the language filtering options available to seamlessly support multilingual browsing, searching, and query formulation. The instructions for this task were as follows: “You are interested in finding any documents in the collection that contain two separate English terms: Brazil and literacy. Once you have selected the terms, you want to view Spanish results only.”

3. 3. Encourage users to consider the implications the interfaces have for their own research through experimentation (task 3). This task was deliberately unfixed and encouraged participants to explore and be inquisitive. The instructions for this task were as follows: “Please feel free to carry out further searches on your own accord or to experiment further with either interface based on your own research needs.”

Task 1 aims to examine the ways in which users evaluate thesaurus and search functions of the two interfaces (research question 2). Task 2 addresses the multilingual features of the two user interfaces and the ways in which they may support users in query formulation (research question 1). And task 3 focuses on the user friendliness, visual features, and visual appeal of the two interfaces as well as on the thesaurus and search functions of the two interfaces (research questions 2 and 3).

Results

Tasks

All users chose a combination of browsing and searching strategies to carry out the three search tasks. Around half of the users chose to carry out first a search for each of their three tasks. The other half decided to use browsing strategies to find the term and its associated terms. However, browsing accounted for a significant part of their interaction, particularly for task 1, where users were asked to find the term “democracy” and one of its related terms. This task required that they interact with the thesaurus to browse and find a related term from among a list of terms that were hierarchically or semantically related to the term “democracy.” In Searchling they typically decided to use the high-level facets and the terms under each facet. Within T-saurus, users browsed the alphabetical list on the left-hand side of the screen to find the term. Several users liked Searchling for its results display as it showed the retrieved documents within the same interface without losing the context of the thesaurus or the search. In more than 10 searches, users found that the red-dot visualization representing the retrieved documents in T-saurus was vague and at times difficult to interact with. In this task, users used several interface features in both user interfaces, including the alphabetical list of thesaurus terms, the query box, and Boolean operators.

In task 2, users were asked to look for information on “Brazil” and “democracy” and also to retrieve documents in Spanish. Due to the multi-term nature of this task, almost half of the users conducted Boolean searches first and browsing next. The Boolean search features of both Searchling and T-saurus were found very useful by participants. Several commented that they would prefer an advanced search option built into the query formulation stage of the search process where they could use a combination of Boolean operators. The auto-completion feature within the T-saurus search box was found particularly useful and interesting. It seems that users of popular search engines, such as Google and Bing, have become more familiar with term suggestions and auto-completion features and therefore expect search systems to provide this option. Also, because Searchling did not have this feature, several users indicated that it would be helpful if it were available in Searchling. Most users expressed positive comments about the search term pool feature available in Searchling. The feature allows users to add or delete thesaurus terms without losing the search context. Also, when users browse and choose a term in the thesaurus, the selected term(s) gets automatically added to the search term pool area, making it particularly easy for the searcher to create a more sophisticated query statement. In task 2 users had to retrieve documents in Spanish. All users found the language option within Searchling flexible, intuitive, and easy to use. In this task, users made use of Boolean operators, the auto-completion function of the T-saurus interface, the search term pool in Searchling, and the language options within both user interfaces.

The results from the third task were mixed. This was, in part, due to the wide variety of search terms that users employed to carry out searches based on their own specific research interests and needs. Some users experienced frustration as they were not able to find terms that matched their query terms. Others found specific features of each interface appealing or useful. Most comments made by users for task 3 focused on various interface features, such as (in T-saurus) the breadcrumb feature and visual grouping of thesaurus terms, and (in Searchling) the sorting of results, search term pool, and linear organization. In all three search tasks, users preferred the results display features of Searchling and commented that the red dots in T-saurus representing retrieved results were not useful unless users could view the results or metadata representations of the documents in context. In this task, users made use of a wide range of interface features and functionalities including the search box, alphabetical and hierarchical browsing functions, result display options, and the search term pool in Searchling.

Multilingual features in Searchling and T-saurus

The first research question aimed to address and evaluate how users will make use of and interact with the multilingual functionalities of the interface in order to formulate queries. The findings presented in this section are based on the data gathered through the usability questionnaire (Appendix 1) and think-aloud data. As was discussed before, both interfaces allow users to choose thesaurus terms for searching in three different languages (English, French, and Spanish). Participants liked the multilingual features in both user interfaces for their easy access and contextual display of thesaurus terms in different languages. The majority of study participants found the Searchling interface user-friendly, intuitive, and particularly flexible across the languages. In terms of Searchling’s capacity for supporting multilingual thesaurus-enhanced searching, browsing, and query formulation, the majority of participants quickly adapted to the language options available. As can be seen from table 1, around 88% of users agreed or strongly agreed that the Searchling interface would help them locate relevant results in Spanish, French, and/or English, whereas 72% of users agreed or strongly agreed that the T-saurus interface provided useful language options. The main difference between the language features in Searchling and T-saurus is that the search results in all three languages are clickable in Searchling and upon clicking on each thesaurus term, that term becomes prominent in bold and the equivalent terms in the other two languages will be shown. In T-saurus, clicking on a thesaurus term will show the term in English with the other two equivalents. One user noted the generative attributes of the interface, stating that she would use Searchling “to find relevant information in English and French for a particular topic that I may have otherwise not thought of.” Another user commented that showing related English terms when carrying out a French search would be very useful in Searchling. Table 1 shows how users assessed the usability of the multi-lingual feature, visual appeal, and ease of use for both Searchling and T-saurus interfaces. More than 60% of the users thought that using a thesaurus-enhanced search interface would help them formulate research questions. They commented that thesaurus search results provided a knowledge map of a particular subject area and that through an examination of various broader, narrower, and related terms, users were able to formulate specific as well as broad research questions.

Table 1.

Usability of language feature and visual appeal

As table 1 shows, in general, users rated Searchling higher in terms of its multilingual support, ease of use, and the thesaurus function. The T-saurus interface was ranked higher only for its visual appeal and aesthetic aspects. The reason for this lies in the fact that T-saurus provides a more interactive thesaurus navigation interface where users are able to choose to display more general, more specific, and related terms to appear on the interface in a dynamic manner. Several participants also noted the potential for T-saurus to enrich their research process. One user stated, “I’ve never seen a Thesaurus-style database before so I found it very interesting and it was inspiring to see what other topics it suggested,” while another bilingual user noted that the interface would be useful “when searching for a topic where I want to return results in more than one language or where I cannot remember a term in a particular language.” Although it is evident that both interfaces have the potential to support and enhance interactive tasks such as multilingual query formulation and expansion, Searchling—described as “more polished” by several participants—had a slight edge over T-saurus. Fourteen participants preferred Searchling, ten preferred T-saurus, and one remained undecided. This observation is particularly interesting as T-saurus has more visual and interactive features but was not preferred over Searchling.

Table 2.

How easy or difficult was it to understand the search function?

Thesaurus and search functions

The second research question in this study focused on thesaurus-based search and browsing functions to examine how users will use and evaluate the thesaurus and search functions within Searchling and T-saurus search interfaces (based on the questionnaire in Appendix 2 and think-aloud data). We were interested to know what kind of thesaurus presentation and visualization would be easy to understand and easy to use by academic users. The Searchling user interface provides a design similar to a faceted search interface. It uses the high-level facets of the UNESCO thesaurus along with a list of terms for each facet. The T-saurus interface provides a more visualized and interactive interface where users have to interact with the interface to choose thesaural term relationships, such as more general, more specific, or related terms. Both Searchling and T-saurus allow users to browse thesaurus terms both hierarchically and alphabetically. The default option for Searchling is the faceted view of the thesaurus, whereas in T-saurus the alphabetical list is default.

To assess the affordance strength of the two user interfaces, we asked the users to give the thesaurus and search functions of Searchling and T-saurus a score ranging from “very difficult” to “very easy,” or “not at all” to “very much,” depending on the question. Tables 2 and 3 shows the ease of use for both search and thesaurus functions of the two interfaces.

As tables 2 and 3 shows, the search and thesaurus functions in Searchling were rated higher than T-saurus, indicating that Searchling’s search and thesaurus functions are significantly easier to use than those functions in T-saurus. Also, 16% of users found that T-saurus was more difficult to use than Searchling.

As part of the assessment of affordance strength, we asked users two additional questions. The first question was whether the thesaurus-based grouping of results provided by these interfaces was helpful in developing searches. Around 24% of users responded “very much” while 52% said “somewhat.” The second question asked them whether they would be motivated to use Searchling or T-saurus as an interface to their more frequently used databases. Table 4 shows that around 72% would be “very much” or “somewhat” motivated to use Sear-chlin while only 52% would be as motivated to use T-saurus.

A final question asked users which interface they generally preferred, Searchling or T-saurus. Around 56% preferred Searchling, while 40% thought T-saurus was a better user interface. This final question confirms the findings related to all the interface features discussed before—namely, multilingual, thesaurus, and search functions.

Table 3.

How easy or difficult was it to understand thesaurus functions?

Table 4.

If the interface was offered in a database you use, how motivated would you be to use it rather than other tools you currently use?

Participants responded positively to Searchling’s convenient features; almost half expressed appreciation in their think-aloud remarks when they realized that their queried terms were automatically added to the search term pool. The other function in the search term pool for fast and efficient removal of previously selected terms was found particularly useful. These two features were among the most useful and most frequently used functions of Searchling according to participants.

Users made several comments about the search and thesaurus functions in T-saurus. The study participants commented positively on the flexibility of the language options, the ability to see an overview of results, and the simultaneous support for browsing, searching, querying, and retrieval activities all in one integrative and dynamic space. Interacting with T-saurus’s thesaurus function, one user noted the potential for this tool to simultaneously broaden and narrow the scope of her research and take it in new directions: “I like how it enables me to pull in related terms and keywords I might not have necessarily thought of myself.”

Participants made several suggestions to improve the search and thesaurus functions of T-saurus, including a more streamlined search process with fewer steps (e.g., by having search terms automatically added to the term pool instead of having to click on a term to add it to the term pool), filters to organize the document list, the ability to perform a Boolean search in the query formulation box, and the ability to view one’s search history beyond the bread crumb.

It is interesting to note that these were all features available in Searchling. These comments indicate that users were able to effectively compare the two interfaces and identify what features would enhance the weaker interface. Almost half of the users attempted to perform a Boolean search at the beginning of task 2, and several commented that that they would prefer an advanced search option built into the query formulation stage of the search process in T-saurus. One user noted, “I found not being able to combine Boolean search terms for more complex searches to be a weakness of the interface.” Also, although several users commented on the usefulness and convenience of the integrated thesaurus and retrieved documents displays, many users suggested that additional filtering options for the search results—such as number of citations, publisher information, and date—would be helpful.

Visualization and visual appeal

The third research question of this study addresses the user friendliness, visual features, and visual appeal of Searchling and T-saurus interfaces. The answers to this question were found through the questionnaires in Appendixes 1 and 2 and think-aloud data. As was noted in table 1, around 64% of the participants favoured T-saurus for its visual appeal over Searchling. We noted that in all other usability categories Searchling was rated higher than T-saurus. Several participants explicitly commented on the design, usefulness, and visual appeal of Searchling’s thesaurus function. Interestingly, users who claimed to prefer linear organization schemas (e.g., lists and chronological structures) almost unanimously preferred the more traditional layout of Searchling, while participants who called themselves “visual learners,” or those who claimed to prefer concept web visuals, favoured T-saurus more often than not. It is also noteworthy that while the former group mainly comprised participants with backgrounds in languages, linguistics, and the library sciences, the latter group was made up of participants with backgrounds in English literature, education, and humanities computing.

Almost half of the participants responded positively to the interactivity and visual appeal of the T-saurus interface, many citing its “attractiveness.” It was also made clear by almost half of the participants that the dynamic visuals were very stimulating and engaging, particularly the visualization of thesaurus terms in T-saurus’s thesaurus space. One participant observed that of the two interfaces, “T-saurus is far more pleasurable to use,” while another user noted that T-saurus would be “[even] more fun with touch screen too!” As well, participants generally found the thesaurus terms straightforward, though more than half of the participants initially found it difficult to locate the legend titled “SHOW” in T-saurus to interact with the thesaurus, thus prompting nine participants to suggest in their in their think-aloud comments that the legend be made more visually prominent. Several users also noted that greater variation within the colour-coding scheme of the thesaurus visualization would be useful and more meaningful.

Furthermore, although there was positive response to the aesthetics of T-saurus, 60% of the users found the red box visualization of document results counter-intuitive, lacking in functionality and use, confusing, or all of the above. Though a few participants voiced appreciation for the ability to get a quick overview of every item in the collection related to their search keywords, many participants described the red box visualization using terms such as “random,” “superfluous,” “distracting,” and “meaningless.” A third of the participants explicitly stated the assumption that in the visualization of items held in the UNESCO collection, proximity to the main term indicated degree of relevance. In the survey comments, several participants noted that the usability of T-saurus would increase if the principle of proximity as an indicator of relevance was applied in the document results visualization. One user noted that she “would really like to use T-saurus if the scatter plot-like visual organized the dots (red dots) closest to the key term in order of relevance. The more relevant the paper, the closer the dot would be to the center.” One user noted that she would prefer to see all her keywords visualized in the thesaurus space as a Venn diagram instead of seeing one keyword at a time: “I would want all search terms to remain visually represented rather than just the most recent.” Several participants also noted that as more documents are involved, the document-results visualization becomes less useful for reducing the user’s cognitive load; many noted that it is hard to keep track of one’s cognitive “path” as one browses the dense results.

Table 5 provides a quick overview of some of the frequently mentioned useful and interesting features of the two user interfaces.

The most frequently cited user-comprehension problems in T-saurus were the red-dot visualization of retrieved results and finding the “SHOW” feature whereby users select more general, more specific, and related terms from the thesaurus, whereas in Searchling, users noted the hard-to-read grey thesaurus terms and the vertical display of terms. Many users found the integration of thesaurus browsing, query formulation, and results display very useful and interesting.

Overall, although it is evident that both interfaces have the potential to support and enhance interactive tasks such as multilingual query formulation, modi-fication, and expansion, the Searchling interface—described as “more polished” by several participants—had a slight edge over T-saurus. Fourteen participants preferred Searchling; ten preferred T-saurus; and one remained undecided. Several participants suggested that the strengths of each interface be integrated and implemented in a new prototype.

Conclusion

This comparative usability study has yielded promising implications for the multilingual thesaurus-enhanced user interfaces to support users in their information seeking process. The visualization in both interfaces was found to be comprehensible to users. Both Searchling and T-saurus were found useful for assisting users in multilingual and exploratory search tasks. In particular, users liked the idea of providing an information-seeking interface that integrates thesaurus, query, and document spaces. A common observation about both interfaces was that users found the thesaurus functions useful for broadening and narrowing down the scope of their research activities. In general, the Searchling interface was more favoured and was found to be easier to use in terms of multi-lingual features, thesaurus and search functions, and users’ motivation to use such an interface for research purposes. Though T-saurus was preferred by fewer users than Searchling, the most promising finding for T-saurus was that it has the potential not only to support browsing, searching, and query formulation but also to transform these processes. It was found that linear thinkers preferred Searchling, whereas visual learners preferred T-saurus. Searchling is a linear, sequential, and visual interface that uses a faceted structure as its default interface, and which allows users to automatically display more general, more specific, and related terms upon selecting a term. T-saurus, on the other hand, provides users with a more interactive and dynamic visualization interface, where users need to interact with and choose individual thesaurus term relationships to be shown. The results from this study indicate that for exploring and using thesaurus terms in a search user interface, most users prefer to have related, more specific, and more general terms shown along with the selected term without additional effort. In other words, upon searching for a term, users should be provided with all the related terms automatically for detailed view and selection. The Searchling interface is similar to faceted browsing interfaces where users are able to use high-level facets to explore the content of the collection. The finding that users found this interface more intuitive and easy to use suggests that user interfaces that make explicit use of thesauri will benefitmore from a faceted interface as implemented in the Searchling interface design compared to the interface of T-saurus, where the thesaurus terms and their relationships are all presented as a visualized environment. This study has several theoretical as well as design implications. One important theoretical implication is that design and evaluation of visual search user interfaces should take into account the different learning and cognitive styles of users. More importantly, a user interface design that aims to provide users with semantic support should strive to balance the cognitive loads that both visualization and thesaurus terms may impose on the user. This implies that sophisticated visualization techniques along with a too-information-rich user interface may put off users. Striking such a balance requires a careful examination of cognitive abilities and user preferences. Some of the practical and design implications of this study are as follows:

Table 5.

Top five favourable features of Searchling and T-saurus

• • Visual view of a thesaurus term along with associated terms should be displayed without any effort on the part of the user.

• • Visualization of result presentation should focus on the notion of relevance—that is, retrieved items displayed closer to the term selected by the user should be more relevant than other more remotely located retrieved items.

• • Browsing and searching options should be provided to support users’ varying search and interaction strategies.

• • Query formulation and search result spaces should be integrated to provide a seamless interaction environment.

• • Multilingual options should be immediately and prominently accessible to users as they carry out their searches.

• • Thesaurus-enhanced search user interfaces should provide faceted views and display of thesaurus terms on the interface to facilitate an understanding of the context of terms and their relationships.

• • User interfaces to multilingual digital libraries should accommodate basic as well as advanced searching and browsing functions to support multilingual access and retrieval.

• • Digital library interfaces should be able to provide seamless access to the various information-seeking strategies that users may adopt. For instance, browsing supports information exploration, and a rich interface that allows users to navigate the conceptual and semantic space of their query has the potential to support users’ exploratory search activities.

• • Digital library user interfaces should be designed in such way as to allow users to easily view and form a coherent perspective of their query, the retrieved results, and language options.

It would be interesting to observe how users with different cognitive, perceptual, and learning styles may have different preferences when they interact with visual user interfaces. Further research may use a verbalizer/visualizer cognitive inventory (Leutner and Plass 1998) to formally study how each learning style will affect users’ interaction with visual user interfaces enhanced with such semantic tools as thesauri. Many users noted positive implications of the thesaurus functions in both Searchling and T-saurus, from undergraduates new to a topic to multilingual experts well versed in the terminology of their field of research. Many participants were also excited by the potential of these experimental rich-prospect visual interfaces being integrated with digital libraries and information spaces beyond the UNESCO thesaurus.