Usability Evaluation of Learning Analytics: A Student-Centred Perspective / Évaluation de l'utilisabilité de l'analytique de l'apprentissage: Student-Centred Perspective : une perspective centrée sur l'étudiant

Abstract

The purpose of this article is to report on the findings of a usability study of a learning analytics tool developed for the eClass learning management system at the University of Alberta. The study was conducted with 39 graduate students using and interacting with a learning analytics tool. The study design consisted of watching an online tutorial on how to use and interact with a learning analytics tool, completing a number of tasks, and, finally, completing usability survey questionnaires. The majority of students found the tool easy to use, easy to learn, and visually appealing. It was also noticeable that 62% of the participants felt that they gained new insight into their interaction with digital learning objects and 72% felt that this would provide them with insight as to how they could improve their interaction with learning objects. Concerns were raised by students in relation to the potential for misinterpretation of student data by instructors.

Résumé

L'objectif de cet article est de présenter les résultats d' une étude d'utilisabil-ité d'un outil d'analytique de l'apprentissage développé pour le système de gestion de l'apprentissage en ligne à l'Université de l'Alberta. L'étude a été menée auprès de 39 étudiants diplômés utilisant et interagissant avec un outil d'analytique de l'apprentissage. La conception de l'étude consistait à regarder un didacticiel en ligne qui portait sur la façon d'utiliser et d'interagir avec un outil d'analytique de l'apprentis-sage, en effectuant un certain nombre de tâches et en remplissant pour finir des questionnaires de sondage d'utilisabilité. La majorité des étudiants ont trouvé l'outil facile à utiliser, facile à apprendre et visuellement attrayant. Il a également été remarqué que 62% des participants ont pensé avoir acquis de nouvelles perspectives sur leur interaction avec les objets d'apprentissage numérique et 72% ont estimé que cela leur donnait une perspective nouvelle sur la façon dont ils pourraient améliorer leur interaction avec les objets d'apprentissage. Certaines préoccupations ont été soulevées par les étudiants quant à la possibilité d'une mauvaise interprétation des données des élèves par les instructeurs.

Supplemental Material: click to download the Appendix.

Introduction

The widespread development of online teaching and learning and the introduction of numerous online courses and programs have presented new opportunities and demand for institutions of higher learning to develop new ways of monitoring and evaluating the online learning environment. Terms such as "educational data mining" and "academic analytics" as well as the more commonly adopted term "learning analytics" have been used in the literature to refer to the methods, tools, and techniques for gathering very large volumes of online data about learners and their activities and contexts. Learning analytics has more specifically been defined by the first International Conference on Learning Analytics and Knowledge (2011) as "the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs." The interest in, and demand for, the development of software tools to perform learning analytics has increased over the past several years (Ferguson 2012). Such tools can offer new insight into online teaching and learning. Siemens et al. (2011) and Siemens and Long (2011), for instance, have proposed several advantages to such learning analytics tools, including:

• • early detection of at-risk students and generation of alerts for learners and educators

• • personalization and adaptation of learning processes and content

• • extension and enhancement of learner achievement, motivation, and confidence by providing learners with timely information about their performance and that of their peer

• • higher quality learning design and improved curriculum development

• • more rapid achievement of learning goals by giving learners access to tools that help them to evaluate their progress.

While course and learning management systems, such as Moodle, hold very large datasets related to student interactions and activities, they typically lack, or have limited, built-in learning analytics capabilities (Dawson 2010). The depth of their extraction, aggregation, reporting, and visualization functionalities have often been basic or non-existent. With such limitations, instructors are not able to use available data points in a multidimensional way to make detailed and comparative inferences about student activities and interactions within one particular course activity or across the entire course content. For instance, it is not possible for an instructor to (1) comparatively and visually identify the most frequently used resources within a course; (2) detect the kinds of resources used by high-performing students in class; or (3) identify the nature of course materials not used by low- and average-performing students.

The purpose of this article is to report on the findings of a usability study conducted in October 2016 in relation to a learning analytics beta software application, eClass learning analytics tool, that was developed for the University of Alberta's eClass learning management system in 2016. Because this tool is incorporated into the university's learning management system, it can be used for any course that is available within the system, regardless of it being offered face-to-face, blended, or completely online. The study involved students from a graduate course, LIS 502: Organization of Information, in the School of Library and Information Studies at the University of Alberta in Edmonton, Canada. Consistent with the literature, the learning analytics tool was developed with the intention of supporting instructors in their ability to monitor students' online learning activities, interaction, and performance and to facilitate the provision of personalized and enhanced advice to students. It was also intended that the tool be used to provide students with a new means by which to regularly monitor and manage their learning activities and interactions and to enable them to compare their performance with their peers in an ongoing and real-time manner.

The study contributes to three areas of research and scholarship—namely, usability evaluation, learning analytics, and learning management systems. In particular, it provides an example of a user-centred study that makes use of a real-world operational learning management system and its associated learning analytics tools in a real classroom setting. The methodological framework and the use of affordance strength approach for interface usability can be used by any usability study that focuses on evaluating user interfaces in learning analytics systems. This study also sheds light on some of the emerging issues surrounding the use of student learning data for student assessment and the potential for misuse and misinterpretation of data that is processed and visualized by the learning analytics tools.

Prior research and context

Learning analytics tools

There have been a number of learning analytics tools developed for various learning management systems, such as Moodle, Desire2Learn, Canvas, and Blackboard. There are two general categories of learning analytics tools. The first category is designed exclusively to be used by instructors and course designers with features and functionalities for analysing and visualizing data related to student activities. The second category provides additional features and functionalities for students as well as instructors with access to learners' interaction and activity data. A significant number of these learning analytics tools are open source applications, some of which are still being developed and others that have not been kept up to date. Examples of recent open source learning analytics tools include the Social Networks Adapting Pedagogical Practice (SNAPP) (Dawson, Bakharia, and Heathcote 2010); the LOCO-Analyst (Jovanovic et al. 2008), the Graphical Interactive Student Monitoring Tool for Moodle (GISMO), and the Academic Analytic Tool (AAT) (Graf et al. 2011). Most of these applications are designed to allow only instructors to access and analyse student behaviour data in learning systems. One of the reasons for the limited number of learning analytics applications available in operational learning management systems is that the programmatic data generated in learning management systems is noisy and large and requires substantial resources to process, make sense of, and make effective use of in order to support learning. Furthermore, designing learning analytics tools that provide both students and instructors access to learning data in an easy-to-understand and meaningful manner is a major design and implementation challenge.

Rationale for the development of eClass learning analytics tool

Course and learning management systems such as Moodle hold very large datasets related to student interactions and activities. However, student-tracking capabilities in these systems are usually limited, and, as a result, the depth of extraction and aggregation, reporting, and visualization functionality of these built-in analytics have often been basic or non-existent (Dawson 2010). While Moodle has reporting functions for students and instructors, these functionalities are not easy to use. For instance, Moodle data can be downloaded as an Excel file, but it still requires analysis in order to be useful for students and instructors. The Moodle learning management system does not currently have learning analytics tools to provide support for analysing, visualizing, and making sense of very large student and instructor activities datasets. For instance, it is not possible for an instructor to (1) comparatively and visually identify the most frequently used resources within a course; (2) detect the kinds of resources used by high-performing students in class; or (3) identify the nature of course materials not used by low- and average-performing students. The overarching goal of the analytical tool developed at the University of Alberta was to facilitate access to, and making sense of, learning data for students and instructors.

At the University of Alberta in Edmonton, Canada, we have developed a learning analytics application for eClass, a learning management system that is based on Moodle. The Moodle learning management system is currently used by many Canadian and international universities around the world. The choice of Moodle also lies in the rationale that it is widely used by the instructors and students of the University of Alberta and serves as a realistic environment for development, implementation, and user evaluation. This new application provides learning analytics functionalities for both students and instructors, and, given that it was created for an open source learning management system, it can be adopted by other universities and colleges that use Moodle as their learning management system. The following figures will depict a number of screenshots of our newly developed learning analytics application. The screenshots provide visual representation of a learner's data analysed using our application.

Visual dashboard for engagement data

Our application provides a number of features to analyse and visualize content engagement, forum engagement, forum usage over time, and events by the user over time. Several figures demonstrating various features are included in the appendix to this article, available online at http://muse.jhu.edu/journal/497. Figure A1 shows an example of a graph that depicts students' engagement data across a number of activities, such as visiting various course web pages, interacting with forums, files, or blocks. Through this graph, students are able to visually identiiy how students within a class have viewed and interacted with different parts and components of a course. It also allows individual students to choose a particular time range to view their own engagement and activities for the time period.

Figure A2 provides a longitudinal view of engagement data for students. This functionality allows for a holistic view of all activities over a certain period of time. They can also choose a particular activity, such as contributing to a forum or blog and view how they have been using or interacting with the forum over time. This feature allows them to keep track of their own use of various learning objects over the period of a semester.

The granularity function shown in figure A3 allows students to narrow down the time line to days and hours. This function will be useful for identifying how active students are before or after a particular quiz.

Visual dashboard for weekly discussion and selected students

In order to allow students to gain a collective perspective of performance within a class, engagement data for all students are shown in figure A4. This graph is useful for instructors and students to quickly and visually see how students contribute to a discussion forum across several weeks.

Figure A5 allows a student to gain a comparative perspective of how other students, on average, interacted with course content and also where the particular student stands this regard. This will allow a student to see the level of engagement on the part of her fellow students.

Learning analytics usability

A number of user studies of learning analytics tools have been reported in the literature. For instance, in a qualitative user study of the LOCO-Analyst learning analytics tool by instructors, the researchers found that multiple ways of visualizing data increase the perceived value of the feedback types and that the participants wanted features that would support supplementing the textual (tabular) feedback with visual representations (Ali et al. 2012). Kelly et al. (2017), in a study of the relationships between student behaviours in a learning management system (LMS) and the learning outcomes of students, found evidence to support the idea that instructors do not strongly influence student use of a LMS and that higher-performing students use a LMS differently from lower-performing students. Dimopoulos et al. (2013) conducted a usability study of learning analytics-enriched rubric and found that teachers and practitioners found the tool easy to accept and use and placed particular emphasis on the enhancement of the system in terms of visualization and social network analysis. Lukarov, Chatti, and Schroeder (2015) present three case studies of learning analytics evaluation and argue that there is no standard way of evaluating learning analytics tools that would measure the impact and usefulness of learning analytics tools for educators and students. They argue that "these LA [learning analytics] tools should not only be usable and interesting to use but also useful in the context of the goals: awareness, self-reflection, and above all, improvement of learning." However, these studies have specifically examined the usability of a particular tool or prototype application. In this study, the findings shed light on a broader range of issues such as privacy, misinterpretation of data by instructors, and students concerns about using these tools as an assessment mechanism.

In this study, we conducted a student-focused usability evaluation of the eClass learning analytics tool to gain a better understanding of how students evaluate the tool in terms of ease of use, visual appeal, supporting them to understand their progress in a course, their interaction with learning objects, and the affordances of the interface and its usability.

Research design and method

Participants

The usability study reported here was completed as a student assignment in a graduate course on the organization of information at the School of Library and Information Studies at the University of Alberta in Edmonton, Canada. The researcher received ethics approval from the university to involve student participants and to conduct this study. In total, 39 graduate students, who had enrolled in the above-mentioned course in October 2016, participated in the study. All of the participants received full credit (10%) if they completed the assignment, regardless of the nature of the feedback they provided via the survey. Based on an initial question, it was confirmed that none of the participants had prior experience in using or interacting with learning analytics tools.

Procedure and data-gathering tools

The study consisted of three parts. Participants were first asked to watch an 11-minute online tutorial about the eClass learning analytics tool and its functionalities. The tutorial was created and shared with students on YouTube. They were then invited to use the eClass learning analytics tool to complete a series of tasks in their LIS 502 eClass website. Finally, participants completed a usability survey that served to gather data about their user experiences and impressions. The usability survey was designed using Google Forms.

The video tutorial consisted of two parts (i.e., two videos), figures from which are included in the appendix to this article, available online at http://muse.jhu.edu/journal/497. The first part provided instruction on how to access and use the "content engagement graph" (figure A6), while the second part provided instructions on how to access and use the "time-based engagement graph" (figure A7). A content engagement graph visualizes strictly the number of times a user viewed (i.e., accessed) or interacted (i.e., added or removed information) with various activities in a given site. The time-based engagement graph visualizes the number of times a user viewed or interacted with various activities within a given site over time. After viewing the two videos, participants were instructed to complete two sets of tasks corresponding to the video tutorials. They were asked to work through a set of activities related to each of the content engagement graphs and the time-based engagement graphs. Lastly, participants were asked to fill out a usability survey containing Likert-style and open questions, the data for which is reported and interpreted in this article.

Tasks

The tasks in this usability study were designed to test participants' initial reactions and general impressions of the usability of the eClass learning analytics tool and to gather data about its affordance strength. As Ruecker (2006) notes, the concept of affordance helps to establish a semantic space related to the fluid mediation of understanding that occurs between people and their environment as opposed to the unmarked term "functional." Day (2011) introduces the idea of cultural, social, and physical affordances and notes that the mediation with which human-computer interaction deals begins with the "subject" needs, suggesting the importance of a user's view of the functionalities of a user interface. The affordance strength approach has been used in a number of previous usability studies (Shiri et al. 2011; Ruecker, Radzikowska, and Sinclair 2011).

As part of the usability process, the goal was to solicit feedback on the participants' overall sense of the tool, how easy or difficult it was to use, what functional or aesthetic elements worked well or could be refined, and how they could see themselves using the tool. The tasks in this usability study were designed to:

• • provide participants with the opportunity to engage with the major functionalities of the tool, specifically the two types of available graphs

• • familiarize participants with how to add and modify content engagement graphs so that they could access visualizations about their engagement during specified time periods for specified activities

• • familiarize participants with how to add and modify time-based engagement graphs so that they could access visualizations about their engagement over a period of time for specified activities

• • familiarize participants with how to compare their data to class averages when using both types of graphs

• • encourage students to consider the potential uses (for students and/or instructors/professors) for the data visualized in both types of graphs.

Results

Given the nature of data-gathering methods, we collected quantitative and qualitative data from the participants.

Quantitative results

Qualitative data were mainly gathered using the Likert scale.

General usability questions

The participants were first asked to assess the eClass learning analytics tool with respect to its general usability. They were asked to provide a score between "strongly disagree" and "disagree" in relation to five questions, which would correspond to the values of one through five on the five-point Likert scale. A single open question—asking students to enter any comments about their thought processes and impressions—followed this set of questions. The general usability questions were modelled after the following form: representations of each point on the Likert scale for questions 1–5 of our post-test questionnaire about the usability of the eClass learning analytics tool:

1. 1. strongly disagree

2. 2. disagree

3. 3. neutral

4. 4. agree

5. 5. strongly agree.

The following are the quantitative results for the usability questions:

1. 1.

   I think this tool would help me understand my progress in the online course:

   • • 46% of participants chose "agree" or "strongly agree"

   • • 31% of participants chose "neutral"

   • • 23% of participants chose "disagree" or "strongly disagree."

2. 2.

   I think this tool would be easy to use:

   • • 87% of participants chose "agree" or "strongly agree"

   • • 10% of participants chose "neutral"

   • • 3% of participants chose "disagree" or "strongly disagree."

3. 3.

   This tool is visually appealing:

   • • 82% of participants chose "agree" or "strongly agree"

   • • 13% of participants chose "neutral"

   • • 5% of participants chose "disagree" or "strongly disagree."

4. 4.

   I feel confident that this tool will provide me with new insight into my interaction with digital learning objects:

   • • 62% of participants chose "agree" or "strongly agree"

   • • 26% of participants chose "neutral"

   • • 13% of participants chose "disagree" or "strongly disagree."

5. 5.

   The appearance of visualization features about my learning data helped me understand how I have been interacting with digital information sources and how to improve my interaction with, and use of, learning objects:

   • • 72% of participants chose "agree" or "strongly agree"

   • • 18% of participants chose "neutral"

   • • 10% of participants chose "disagree" or "strongly disagree."

As can be seen from the above responses, the majority of students found the tool easy to use and visually appealing. Less than 46% of students felt that the tool helped them to understand their progress in the online course. It was also noticeable that 62% of the participants felt that they gained new insight into their interaction with digital learning objects, and 72% felt that this would provide them with insight as to how they could improve their interaction with learning objects. This is particularly important since gaining insight into one's interaction with digital learning objects and their status compared with the rest of the class provides a quick overview of how the rest of the class interacts with digital learning objects.

Affordance strength questions

Following the usability questions, participants were asked to answer six quantitative affordance strength questions. These questions focused on gathering data on ease of use and ease of understanding the tool, willingness and motivation to use the tool, helpfulness of the tool for study planning, and the capability of the tool to support learning. The affordance strength questions were divided into two parts. Part 1 of the affordance strength questions was comprised of a set of two questions, and Part 2 was comprised of a set four questions. A single open question, asking students to enter any comments they had, followed each set of questions.

Affordance strength questions: Part 1

In Part 1, students were asked to provide a score between "very difficult" and "very easy" for two, five-point Likert-style questions that were modelled after the following form: representations of each point on the Likert scale for Part 1, questions 1–2 of our post-test questionnaire about the usability of the eClass learning analytics tool:

1. 1. very difficult

2. 2. difficult

3. 3. neutral

4. 4. easy

5. 5. very easy.

The following are the qualitative results for the Part 1 affordance strength questions:

1. 1. How easy or difficult was it to understand the learning analytics tool?

   • • 92% of participants chose "easy" or "very easy."

   • • 8% of participants chose "difficult" or "very difficult."

2. 2. How easy or difficult was it to understand the visualization features?

   • • 82% of participants chose "easy" or "very easy."

   • • 5% of participants chose "neutral."

   • • 13% of participants chose "difficult" or "very difficult."

The vast majority of students found the tool and its visualization features to be very easy or easy to understand. Prior research has found that understanding visualization was influenced by learning and cognitive styles, which was not examined in this study (Shiri, Ruecker, and Murphy 2013).

Affordance strength questions: Part 2

For Part 2, participants were asked to give a score between "not at all" and "very much" for four additional five-point Likert-style questions that were modelled after the following form: representations of each point on the Likert scale for Part 2, questions 1–4 of our post-test questionnaire about the usability of the eClass learning analytics tool.

1. 1. not at all

2. 2. maybe

3. 3. neutral

4. 4. somewhat

5. 5. very much.

The following are the qualitative results for the Part 2 affordance strength questions:

1. 1. Would you want to use this interface?

   • • 36% of participants chose "somewhat" or "very much."

   • • 13% of participants chose "neutral."

   • • 51% of participants chose "maybe" or "not at all."

2. 2. Could this tool be a helpful tool in your study planning?

   • • 51% of participants chose "somewhat" or "very much."

   • • 13% of participants chose "neutral."

   • • 36% of participants chose "maybe" or "not at all."

3. 3. If this tool was offered in the eClass learning management system, how motivated would you be to use it to support your teaching/learning?

   • • 21% of participants chose "somewhat" or "very much."

   • • 21% of participants chose "neutral."

   • • 59% of participants chose "maybe" or "not at all."

4. 4. Would this tool be capable of supporting your teaching/learning?

   • • 41% of participants chose "somewhat" or "very much."

   • • 21% of participants chose "neutral."

   • • 38% of participants chose "maybe" or "not at all."

The above responses provide mixed feedback on the helpfulness of the tool. 51% of the participants thought the tool was useful for study planning, and 41% of them felt that the tool would be capable of supporting learning. However, only 36% were of the view that they would use the interface, and only 21% stated that they would be motivated to use the tool. One explanation for this discrepancy stems from the fact that the idea of this tool as a mechanism to support learning is new to students and is compounded with the metacognitive aspect of the tool that allows students to learn about their learning process.

As mentioned above, a single open question followed each of Part 1 and Part 2, asking students to enter any comments (i.e., qualitative feedback) they had in relation to the quantitative questions in each part. The results of the qualitative feedback are discussed in the next section.

Qualitative results

Students were asked to provide written feedback on a total of four open questions as part of the usability study. After completing Part 1 of the affordance strength questions, participants were asked:

• • Affordance Strength Question 1: Please enter any comments regarding the grid (i.e., Part 1 quantitative affordance strength questions) above.

After completing Part 2 of the affordance strength questions, participants were asked:

• • Affordance Strength Question 2: Please enter any comments regarding the grid (i.e., Part 1 quantitative affordance strength questions) above.

Also, at the very end of the survey, participants were asked for their written responses to the two general "stand-alone" open questions:

• • General Question 1: Do you have any comments about improving the look and feel of the interface?

• • General Question 2: Can you describe how you would use this tool?

Affordance Strength Question 1: Please enter any comments regarding the grid (i.e., Part 1 quantitative affordance strength questions) above

Part 1 of the affordance strength quantitative questions focused on how easy or difficult it was to use the tool overall and how easy it was to understand the visualization component of the tool. This corresponding open question asked participants to provide any comments they wished to share. The most prominent themes identified within the comments made by the participants were (1) praise for general usability; (2) opportunities for refinement; and (3) data interpretation. Detailed analysis of these themes is provided below.

Praise for general usability

Generally, participants responded quite favourably with respect to the usability of the eClass learning analytics tool. It was complimented by several participants as being easy to use, intuitive, straightforward, and understandable." In addition, one participant specifically offered that "the use of [iconography enhanced the ease of using the Tool," suggesting that the visual design of the eClass learning analytics tool was instrumental in promoting usability. The role of the tutorial video in promoting ease of use was also acknowledged by several participants. They commented that the tutorial video enabled them to quickly and easily understand and use the eClass learning analytics tool.

Opportunities for refinement

While a later open question in the study's survey specifically requested feedback in relation to potential changes that could be made to the eClass learning analytics tool, participants seemed interested and invested in improving the tool from the very beginning and immediately identified some excellent opportunities for refinement. Suggestions centred mainly on requests for clarification of terminology and units in the graphs, layout changes, and minor changes to functionality and default views.

With respect to clarification of terminology and units, some students indicated that they were uncertain as to the meaning of the terms "all core" and "all blocks" and about the corresponding bar graph data that appeared in relation to them when a default content engagement graph was accessed (see figure 2). "All core" refers to the number of views of certain key site functions, as defined by Moodle. While this explanation was provided in the study's instructions, it continued to be confusing for users and, as such, has since been renamed as "system" and removed from the default view. In addition, the "all blocks" category has been removed altogether, as it had initially been included for testing purposes only and did not display useful data.

A few participants were also unclear as to what certain scales and units that sometimes appeared on the y-axis of graphs represented. For instance, the y-axis scale of the content engagement graph displayed partial events (i.e., views or interactions) in decimal form for some participants, causing confusion. In addition, the time-based engagement at times used the abbreviation "m" as its units (e.g., 100m), but no explanation as to what this unit represented was provided, and a few participants also found this confusing. Upon investigation, it was determined that "m" represented milli or (0.1). Inclusion of such units resulted from a setting that had inadvertently been applied and resulted in the display of SI units. This setting has since been turned off.

In relation to layout, a few participants indicated that overlap occurred when multiple graphs were accessed. The text of graph legends and/or the x-axis of graphs also appeared to be "cut off" (i.e., not visible) for a couple of participants. As participants used their own laptops for the usability study, it was helpful to get a sense of minor changes to settings or codes that could be made to optimize viewing of the learning analytics tool for a broad variety of computer models. One participant recommended additional functionality—namely, including a "back button" so that users could view previously generated graphs. This very practical function has been added to the current version of the eClass learning analytics tool.

Finally, as suggested in the tutorial video, several participants confirmed that, when first adding a new graph, it would be valuable for the default settings of that graph to reflect only the current semester's data. At the time of the study, default graphs provided the previous 365 days' worth of data (from the date of access). This proposed modification was echoed later on by several other students in response to question 3 of the survey and has been implemented.

Data interpretation

A number of participants raised concerns about the potential for misinterpretation of the data. For instance, two participants indicated that potentially incorrect conclusions about success could be drawn from the data. They suggested that engagement would not necessarily representatively "signify success" since, for instance, offline use of resources that had been downloaded would not be captured as engagement by the tool.

Affordance Strength Question 2: Please enter any comments regarding the grid (Part 2 quantitative affordance strength questions) above

Part 2 of the affordance strength quantitative questions focused on whether students would want to use the eClass learning analytics tool, whether they thought it would helpful to their study planning, how motivated they would be to use it to support their learning, and whether they thought this tool would be capable of supporting their learning. Participants were asked to provide any comments they wished to share in relation to the Part 2 quantitative questions. The most prominent themes derived from the comments were: (1) potential uses and (2) potential for the instructor/professor to misinterpret the data. Detailed analysis of these themes is provided below.

Potential uses

The quantitative questions that preceded this open question referenced the potential uses of the eClass learning analytics tool and seemingly prompted participants to provide some preliminary ideas about use. One participant thought that the tool "could help students see where they are spending most of their time and which classes they have been dedicating more time to." Another participant indicated that they might use this tool to plan how to spend their study time for their next semester by reviewing where they spent most of their time and how this correlated with the academic (grades) outcomes. As was shared by many others later in the survey, two participants also indicated that the eClass learning analytics tool could be useful for monitoring their participation in the online forums component of eClass, particularly when participation was required for a course. Two participants simply indicated that they were unsure as to how it would inform their study planning in the future, while two others thought that access to the information would not impact their behaviour. Finally, one student insightfully suggested that one specific course of action should naturally flow from this study's work; they proposed that it would be helpful to investigate in the literature to determine what metrics have been identified as beneficial to student success, which, in turn, could inform potential uses.

Six participants echoed comments from questions 1, stating that they mainly downloaded and used resources offline. However, these comments were made in the context of potential use, rather than in relation to concerns about how the data might be interpreted. These participants stated or implied that because they use eClass more sparingly, for instance, by downloading and accessing resources offline, the gaps in data that may result could prevent them from taking advantage of what the eClass learning analytics tool has to offer. Somewhat relatedly, one student pointed out that a limitation of the tool is that it "does not account for the time you spend offline." As a final note related to potential uses, a few participants indicated that they generally found the tool to be "interesting" and "fun." These and related descriptors were repeated throughout the open questions of the survey. While such feedback does not speak to ways in which the tool could be used per se, they suggest quite positively that using it is an enjoyable experience, which may, in turn, promote students' desire to use it moving forward.

Potential for instructor/professor misinterpretation of data

Some participants expressed earlier in the survey that their offline access would not be captured and, therefore, that conclusions about how online engagement impacts success may be limited. Along the same lines, some participants were concerned that the data, when considered by instructors or professors, may lead to misinterpretation of student participation/engagement. The downloading of resources to use offline was again mentioned. Participants were concerned that this practice could affect how they might be perceived or assessed. One participant commented that "[the] number of views doesn't necessarily mean much if, at the beginning of the term, students look through all of the material and download relevant PDFs and links to their personal computers and then refer to those, not eClass, throughout the term." Another remarked that they were concerned that instructors might make "quantitative judgments about success without full knowledge of … [a student's] learning style or strategies." Also related to the potential for misinterpretation, two participants commented (in response to this question and to question 1—included here for convenience) that if this tool was used for assessment by in-structors/professors, they might be motivated to "appear" to be engaged, rather than actually be engaged. Arguably, such behaviour could also result in misinterpretation of the data.

General Question 1: Do you have any comments about improving the look and feel of the interface?

Participants were generally particularly complementary of the look and feel of the eClass learning analytics tool. As in response to Affordance Strength Question 1, it was described again as "easy to use," "straightforward," and understandable. It was also toted as "user friendly" and "easy to navigate." In reference to its visual design, the tool was complimented as being "simple," "appealing," and "aesthetically [and] visually appealing," and the graphs were felt to have an "intuitive layout." Also, as mentioned earlier, the instructional video that introduced and guided students through the tool was described as being valuable for its clarity and facilitating understanding of the tool. Participants also provided some specific and valuable points for improvement. The following four requests echoed responses to Affordance Strength Question 1 and have all been addressed in a newer version of the eClass learning analytics tool: (1) request to set the default period for graphs to display the latest semester (this was by far the most frequent request); (2) request for clarification around the units used on the axis of the graphs; (3) request for reformatting of graph legends; and (4) request to include a back button. Additional layout improvements were also suggested. Two participants indicated that the checkbox located next to the time period selection tool—which had to be selected prior to selecting a time period for a graph—was cumbersome and confusing and should be removed (see figure 2, to left of "dates" selector). A few participants also requested the ability to close or remove generated graphs. Finally, an option to be able to hover over some key terms in order to access definitions was requested. This feedback was extremely valuable, and the tool has been modified accordingly.

General Question 2: Can you describe how you would use this tool?

While some participants indicated at various points in the survey that they were unsure how to use this data to affect positive academic outcomes, many others offered a variety of interesting ideas about potential uses for instructors/profes-sors and students in response.

Potential uses for instructors

Participants thought that instructors and faculty (professors) could use the eClass learning analytics tool mainly to (1) determine student participation and overall engagement; (2) determine with which elements of an eClass site students most commonly interacted; and (3) assess the overall "utility of their eClass for students." Regarding the use of data by instructors and professors, several participants cautioned that instructors and professors could misinterpret the data, especially when site resources were downloaded and repeatedly accessed offline. One student, however, thought that they might actually shift their behaviour to access sources online (i.e., leverage the data-recording capabilities of the eClass learning analytics tool) if they knew that their information about their activity would be captured. Arguably, this suggestion reveals a potential benefit of the tool–namely, it provides students with a means by which they can evidence their participation in a measureable and quantifiable way. Participants also articulated concerns about access to student information and impacts on stress. Two students commented that knowing that professors had access to their information would cause them stress/anxiety. In contrast, however, one participant indicated that the ability to "visualize" the effort they put into a class (i.e., access their own data) would actually reduce their stress and would serve as a motivator that could show them how productive they could be. Finally, two participants expressed concerns about their professors being able to "surveil" their actions. One participant proposed that the data should be made exclusively available to students, while a couple others wanted the option to opt out.

Potential uses for students

Participants offered a number of creative and enthusiastic suggestions in relation to how students might use the tool. Most prominently, participants felt that the eClass learning analytics tool would be useful to track and monitor their participation on eClass forums when such forums formed part of the class assessment. Eight of 39 participants identified this as a potential function. Several participants also mentioned that they would use the tool to either track or reflect on their participation generally. Another handful of participants offered that they might also use the visualized data to motivate online participation in some shape or form. On a related note, one student indicated that they would use it to "keep from procrastinating," and another indicated that it would help them "gauge how active" they were and determine whether they needed to adjust their activities accordingly. Others were hopeful that the tool could reveal to them those resources that they had "missed" out on, especially if their peers' data was available and signalled activity in relation to mentioned resources. In fact, several participants indicated that seeing their peers' interaction with the tool might serve to prompt them to increase or change participation. A number of other more general uses were noted, including the comparison of participation to class averages, time planning, goal setting, time management, and informing study habits. A few concerns regarding potential student uses were also raised. Two participants felt that this tool would not be useful, either because they believed they had a good sense of their degree of participation or because the courses they were taking were primarily face-to-face and were not online courses. As was stated earlier in this article, because the learning analytics tool is incorporated into the University's learning management system, it can be used for any course that is available within the system, regardless of it being offered face-to-face, blended, or completely online. Also, one student indicated that they were prone to anxiety and that the tool might cause them stress because they would be concerned with having a record that demonstrated the irregular times at which they accessed the information.

Conclusion and further research

Learning analytics applications have the potential to provide insight into how students interact with, and make use of, learning objects in their learning process. This insight can provide instructors with new knowledge about the ways in which students interact with the course website and how the instructor can provide advice and help to improve teaching and learning. This article reports a study that has adopted a student-centred and empirically sound usability evaluation of a learning analytics tool within an operational learning management system and a real graduate course setting. The response to the eClass learning analytics tool was generally very positive. As for usability, both the quantitative and qualitative data reflect that an overwhelming majority of participants found the tool to be visually appealing and easy to use. In addition, and perhaps most encouragingly, a few participants indicated that they actually found the tool to be fun. It appears that using the tool is an enjoyable experience; this may motivate students to explore it moving forward. Another very positive outcome of the study was that participants were forthcoming with excellent constructive feedback. Many of their suggestions were practical and specific and have been incorporated into a newer, more refined version of the tool. An abundance of data was provided in relation to perceived potential instructor/professor uses of the data presented by the tool. Participants acknowledged that instructors and professors would be able to get a holistic sense of eClass site use by students and would benefit from being able to identify which students engaged online, with which activities, and to what extent.

These realizations were tempered with the potential for misinterpretation of student data, which counsels considerate (i.e., cautious) use of the tool, especially by instructors/professors. Misinterpretation of data could be minimized by using a broad range of sources of evidence for assessing students' performance and success, such as instructor-student interaction, engagement data, usage data, grades, as well as an individualized feedback mechanism. Students were eager to point out that records of their activity may not accurately represent their efforts and experience in relation to a given class. In particular, participants noted that downloading and use of materials offline would generate data that does not necessarily reflect student engagement. One student suggested, however, that they saw an opportunity to leverage the tool to demonstrate their engagements—they would be willing to move more of their activity online if they knew it would be accounted. Ultimately, several students cautioned that drawing conclusions about student success or assessing students based on usage data should be done with care. It was further noted that students seemed to focus mostly on the direct impacts of the assessment of their work and less so on the ability of instructors or professors, for instance, to enhance the learning process or experience based on data or to identify students who are struggling or require additional challenge. As was noted earlier, it is particularly important that learning analytics tools be used as one of the many sources of evidence for the assessment of student learning and not as the sole source and that the analytical results should be closely triangulated with various other data sources, such as students' performance on different assignments, their success rate throughout the course, and their engagement throughout the term.

In relation to potential student uses of the data, students indicated mostly popularly that the ability to track interactions required by courses—forum postings being the example repeatedly mentioned—would be particularly valuable. Beyond that, various students reflected on the variety of uses for the tool, such as planning for studying, getting a sense of important course resources, reflecting on engagement and associated implications, using its results for motivation, and even, in one case, using the data as reassurance of one's participation. A few students also mentioned that it could be stressful to know that instructors and professors could "watch" (and judge) your online academic activity. Although a number of interesting ideas and considerations were generated, some students reported being at a loss as to how they might use the tool. Further research into impactful uses for student data in analytics tools would be valuable to help guide and support students and their learning. The eClass learning analytics tool was designed to be flexible and organic rather than prescriptive and to invite users to let their circumstances, needs, and imagination guide them in using and reflecting on the online engagement captured by the tool. The findings of this study have practical implications for designing visualization tools for learning management systems. This study has social and ethical implications for instructors who would like to use learning analytics tools in their courses. The findings from this study will be useful for the designers of learning analytics tools, instructors, and educational administrators who make use of learning analytics tools for assessment as well as for researchers who have an interest in conducting usability evaluation of learning analytics tools.