Data Dashboard: a scenario-based prototype evaluation

Managing personal data (from files and documents to photos and mobile applications) is a challenging task that requires time, patience, and the right tools. With people using an ever-increasing number of devices and cloud platforms, it’s easy to get lost.

In this study, we designed a new prototype system for managing personal data and evaluated it in a lab study with 18 participants. We used interviews, a set of scenarios, and two card sorting exercises to understand what participants thought of the prototype. The results were encouraging: most participants had positive reactions. But we also found some usability issues and aspects of the system to improve.

Read the detailed study plan:

My role as a User Researcher

  • Designed and developed the interactive prototype.
  • Planned and led the user evaluation with 18 participants (and 4 pilots): wrote the study plan and script, coordinated recruitment, conducted most of the interviews, analyzed the data, wrote the usability report and research paper with the results of the study.

Team: I worked with Janet Chen, an undergraduate research assistant who helped with the project under my supervision + William Odom and Joanna McGrenere, senior researchers in HCI.

The project resulted in a research paper that we presented (virtually) at the DIS 2020 conference.

The problem

People struggle with managing their personal data across devices and platforms. They also have different preferences for how to do it.

Most people who use technology have to manage a growing amount of personal data, from photos, documents, and files to mobile applications and location history data. Often, personal data is spread across many different platforms and devices.

Managing data becomes a challenging task because it’s hard to know what data is stored where and how to best organize it. Some people like to keep everything and organize all their data neatly, others might be messier or might want to get rid of unnecessary items. So, how can we support people’s needs and individual preferences for managing their data?

Managing personal data spread across devices and platforms can be challenging. (Illustration by Icons8.)

Research goals and questions

Can centralization and customization help users better manage their data?

In this study, we wanted to understand if two design approaches could help people in better managing their data.

The first design approach is to centralize data in a single location, providing an aggregated overview of items from different platforms and devices. We wanted to know whether centralization could help people.

The second design approach is to use customization to create a system that can better adapt to people’s individual preferences for how to manage their data. We wanted to know whether customization might help to create a system that resonates with a wide range of users.

To answer these two key questions, we designed Data Dashboard, a centralized and personalized system for managing data. Then, we evaluated a prototype of the system in a user study with 18 participants.

Can providing a centralized overview of personal data help users?

The Data Dashboard prototype

Data Dashboard is a centralized system that gives users an overview of personal data and a set of personalized filters for sorting through their items

We designed Data Dashboard as a centralized system that shows an aggregated overview of data stored on different devices or cloud platforms (e.g., Dropbox, Google Drive, iCloud) and also provides a set of customizable filters for curating data. There are four sections in the system:

  • Activity shows data based on recent activity, grouped by type.
  • Explore Your Data shows all data broken down by type and also has a set of filters for sorting through data.
  • Quick Actions provides a set of recommendations for quick management actions and also has the same filters found in Explore Your Data.
  • Settings let users change how the filters work and add or remove platforms and devices.

We designed and developed the prototype in the first half of 2019, using AngularJS and Bootstrap. Because we wanted it to be quick and easy to implement, we decided to keep the visual design simple and make the interactions illustrative only. The prototype doesn’t actually connect to any platform or device. This is because at this stage we wanted to test the viability of the idea, understanding what works well and what doesn’t, before developing it further.

Evaluation methodology

We structured the prototype evaluation in three parts:

  1. An introductory interview, where we asked participants about their general attitudes and behaviors in data management.
  2. A scenario-based interaction with the prototype.
  3. A debriefing interview with two card sorting exercises.

Scenarios of use

After the introductory interview, we asked participants to interact with the prototype and prompted them to think out loud as they went through five different scenarios of use:

1 The space on your computer is running out. You want to find some data to discard. You are not sure where to start looking, but you know that you do not care too much about old documents.

2 It is a rainy day. You have set aside some time for doing a regular cleanup of your devices. You usually do this every few months. You want to review your data and make sure everything is organized in your preferred way.

3 You have 5 to 10 minutes in between meetings and errands. You decide to take a look at your recent data to get a sense of anything that needs taking care of.

4 You have heard about a data leak from a popular cloud storage platform that exposed personal information to hackers. You want to review what data you have stored on different cloud platforms that might pose a privacy risk in the future.

5 You are in the process of buying a new computer. You want to make sure that you are not going to lose any of the data you care about. You want to ensure that everything is stored in more than one place.

We used these scenarios to cover possible data management behaviors and situations.

Debriefing and card sorting

Then, after going through the scenarios, we had a debriefing interview where we asked participants clarifications and overall impressions about the prototype. In this last part of the study, we also had participants go through two short card sorting exercises:

  • In the first card sorting exercise, we asked participants to rank the scenarios based on how relevant they were to their experience. We used this exercise to understand what parts of the prototype worked best in different situations.
  • In the second card sorting exercise, we asked participants to rank the Data Dashboard prototype against other similar tools that they had used in the past. Here, we wanted to understand how the prototype would integrate into their existing flows and how it compared to existing tools.
Examples of participants card sorting results
We had two card sorting exercises: in the first, participants ranked the scenarios of use; in the second, they ranked the prototype against other similar tools.

Take a look at the study materials we used in the evaluation:

Key results and design recommendations

The majority of participants (12/18) had positive reactions to the prototype, saying that it was smart, intuitive, user-friendly, and would save their time. Several participants also preferred the system when comparing it to other tools they had used in the past.

The rest of the participants had mixed or negative reactions, thinking that some aspects of the system were unclear or unnecessary, although they still thought that in some specific situations and with some changes it could be useful.

Overall, participants thought that centralizing data in a single location presented some privacy risks, but the customization options offered by Data Dashboard could help offset these risks. The different sections of the system proved useful in different scenarios of use, with the first three scenarios being the most relevant for participants.

An overview of discoverability and clarity issues in the Data Dashboard interface for each participant in the study.
An overview of discoverability and clarity issues in the Data Dashboard interface.

Based on participants’ interactions during the scenarios, we noticed that the prototype had some issues in discoverability, usability, and clarity. Some of our key design recommendations focus on making key functions more discoverable, simplifying how the different sections work, and promote consistency within similar sections or functions:

  1. To improve discoverability, we recommend using direct manipulation to sort and hide different data types instead of a pop-up. We also think that redesigning the customization options might be necessary: instead of having them in a separate Settings section, these options should be embedded within the sidebar filters found in Explore Your Data and Quick Actions.
  2. To simplify the interface, we recommend to merge Activity and Explore Your Data, remove the Shared Data section, and provide a wider array of default customization options, possibly curated from an online repository.
  3. To promote consistency within the system we recommend giving more emphasis to the location of different items throughout the sections (for example, indicating whether an item is stored on a device or on a cloud platform) and always including an indication of how much storage space different items or categories of items take.

In the full usability report that we wrote after the study, we provide additional recommendations for each section and scenario of use:

The recommendations we provide should help in redesigning specific parts of the prototype and make it more user friendly. Overall, the results of the evaluation show that our approach of combining centralization and customization in a single system has promise. That said, there is more work to do in this space to fully support people’s needs and Data Dashboard can provide a good jumping point for future system development.

Even though I have focused mostly on usability issues here, our study also had a broader and more theoretical focus. In the paper we presented at the DIS 2020 conference, you can read more about the design decisions that went into the prototype and the reactions from participants framed against the conceptual metaphor of data boundaries.

Below, you’ll find our virtual presentation for the DIS conference:

Our presentation of the Data Dashboard paper at DIS 2020.