Observing the user experience of software upgrades

Summary

A study where I used multiple methods (field observations, diary study, interviews) to better understand the upgrade experience on common operating systems (e.g., Windows, MacOS). It turns out, installing an OS upgrade can be a long and sometimes frustrating experience, with most changes noticed only in the immediate aftermath. But there are opportunities to improve the experience.

My role as a User Researcher

  • I was the lead researcher for the project: I recruited participants, arranged all the logistics for the study, moderated the observations and interviews, collected and analyzed the data, collaborating with the rest of the team.
  • I also took the lead on writing the paper based on the project, presenting the results at the CHI 2017 conference.

Team: I worked with a team of senior researchers in HCI and Computer Science at Inria, the French National Institute for Research in Computer Science.


Context and research questions

How many times has your computer asked if you want to install an update, just for you to say, “Not today?” You’re not alone. It is common for many technology users to skip or postpone software updates.

In a first attempt to understand this phenomenon, we ran a survey with 65 Mac users and found that they delayed major upgrades by an average of 80 days. But why? We decided to follow up the survey with a second study, taking a closer look at the actual process of downloading and installing operating system upgrades to see how users experienced it.

We focused on two key research questions:

  • How do users experience the upgrade installation process?
  • What are the costs and benefits, immediately after and over a longer period?

Methods

To capture the user experience of upgrading, we used observations, interviews, and a diary study

Because we wanted to build a complex understanding of the issue and go beyond the time-limited experience of the upgrade itself, we decided to use multiple methods to collect data: field observations, interviews, and a diary study.

In the field observations, we observed 14 participants during the installation process of a software upgrade or update on one of their own devices and asked them questions about what was going on. Then, we followed the participants for a month in an online diary asking them to report any changes they noticed.

Day 1
Field observation

We observed participants during the installation process.

Day 1
Day 2
Diary study starts

From this day and until the end of the study, participants complete a daily diary entry (online) describing changes they notice in their devices.

Day 2
Day 14
Mid-study interview

We meet again with participants to check on their progress with the diary and do a short interview.

Day 14
Day 28
Final interview

The diary study is over. We meet with participants for a final debriefing interview.

Day 28

From the observations, diaries, and interviews, we had a lot of data to untangle. I decided to use thematic coding to make sense of it.

  • For observations and interviews, I grouped the participants’ data by phase (before, during, and after the upgrade).
  • For the online diary, instead, I decided to systematically categorize positive, neutral, or negative changes to give a sense of the effect of the upgrade on participants’ perceptions.

Key insights

Before the upgrade, participants approached the process with mixed feelings

Most participants did not see security as a top concern or reason to upgrade. Instead, they mostly worried about upcoming changes to their system (e.g. losing their data or having to learn again how to do something). But even though they had mixed feelings, they did not prepare for the process (for example, by doing a backup of their data—this really surprised us!) and had confused mental models about what to expect from it.

A photo from a scene in Sex and the City, where Carrie Bradshaw asks her friend Miranda why no one talks about backing up
Several participants, to our surprise, did not back up their systems, just like Carrie Bradshaw in a memorable episode of “Sex and the City.”

During the upgrade: a long and frustrating user experience

The long duration of the upgrade process was a key issue for participants: the long waiting times made the experience unpredictable and frustrating.

“I was supposed to meet a friend, I have a social life!” said one participant, exhausted.

😤🤬😴

What made the long duration of the process feel worse, was not knowing what was going on in the process. Participants complained about the poor visual feedback indicating the progress, often consisting of vague and minimal progress bars that did not feel very informative.

Participants complained about the visual feedback during the upgrade process.

After the upgrade: was the long process worth it?

Participants noticed most of the changes in the first week after the upgrade. But these changes were not necessarily worth the frustrating experience because participants noticed more negative than positive changes in their systems. For example: lost settings, bugs, crashes, unwanted features, or a worse user interface. They also noticed some positive changes (for example, new features or better performance), but these were fewer in number and milder in quality compared to negative changes.

In the end, one month after the upgrade, most participants were ok with having gone through the process, but they did not look forward to doing it again any time soon.

An illustration of reported changes after the upgrade: participants noticed most changes in the first week after the upgrade.
Participants noticed most changes in the first week after the upgrade.

Takeaways & opportunities

Based on these insights about user behaviors and perceptions, we identified several opportunities to improve the user experience of OS upgrades:

  1. How to keep users informed during the download and installation of the upgrade? Better feedback and explanations of what is happening at each step could help here.
  2. How to counteract the long duration of the process? If not possible to shorten it, making it feel shorter than what it is might make the experience less daunting.
  3. How to inform users of security changes? Making any invisible but important security changes more visible to users could help, as well as reminding users about the positive benefits of upgrading in the first week after the process.

In the paper I presented at CHI 2017, you’ll find more details about the project, which also included an additional set of interviews with three security managers, giving us a different take on software upgrades.