Observing the user experience of software upgrades

How many times has your computer asked if you want to install an update, just for you to say, “Not today?” You’re not alone. Most people deal with these notifications far too often and, whether it’s a small update or a big upgrade, it’s common to just skip them. But why? In this study, we wanted to understand how regular people experience operating system upgrades and how to improve the process.

What is the user experience of software upgrades?

What are the costs and benefits, immediately after and over a longer period?

(Key research questions for the project)
OS upgrades: who has time for them?

To answer our questions, we ran a field study, observing 14 participants while they installed an operating system upgrade on one of their own devices. Then, we followed the participants for a month in an online diary asking them to report any changes they noticed. We also ran a log analysis using an online survey with 65 Mac users to quantify upgrade installation delays.

We found that Mac users in our survey delayed major upgrades significantly. In the field study, we saw how the upgrade installation process was a frustrating experience for participants, with poorly designed feedback. After the upgrade, participants noticed more negative than positive changes. Based on these insights, we identified some design recommendations that technology companies can use to improve the user experience of upgrades.

My role as a User Researcher

  • I was the lead researcher for the project: I recruited participants, arranged all the logistics for the study, moderated the observations and interviews, collected and analyzed the data, collaborating with the rest of the team.
  • I also took the lead on writing the paper based on the project, presenting the results at the CHI 2017 conference.

Team: I worked with a team of senior researchers in HCI and Computer Science at Inria, the French National Institute for Research in Computer Science.


Methods

To capture the user experience of upgrading, we used observations, interviews, and a diary study

Because we wanted to build a complex understanding of the issue and go beyond the time-limited experience of the upgrade itself, we decided to use multiple methods to collect data.

We decided to focus the field study on asking participants to upgrade the operating system on one of their own devices. We observed as the upgrade installation process took place. Then, we followed the aftermath in a four-week diary study.

During the diary, we also did two additional interviews: a short check-in halfway through and a slightly longer interview to conclude the study and reflect back on the upgrade experience.

Day 1
OS upgrade observation

Upgrade observation + contextual interview.

Day 1
Day 2
Diary study starts

From this day and until the end of the study, participants complete a daily diary entry (online) describing changes they notice in their devices.

Day 2
Day 14
Mid-study interview

We meet again with participants to check on their progress with the diary and do a short interview.

Day 14
Day 28
Final interview

The diary study is over. We meet with participants for a final debriefing interview.

Day 28
Take a look at some of the materials used in the study:

From the observations, diaries, and interviews, we had a lot of data to untangle. I decided to use thematic coding to make sense of it.

  • For observations and interviews, I grouped the participants’ data by phase (before, during, and after the upgrade).
  • For the online diary, instead, I decided to systematically categorize positive, neutral, or negative changes to give a sense of the effect of the upgrade on participants’ perceptions.

Online survey and log analysis

As a complement to the field study, we also ran an online survey to take a mixed-methods approach and contextualize the results of our observations.

In the survey we wanted to know how long people delay the installation of upgrades from the date of the release, to quantify the effect of frustrating experiences with the installation process. We asked 65 Mac users to upload a log of updates and upgrades installed through the App Store and then performed a log analysis on the files we collected (InstallHistory.plist, example below).

	<dict>
		<key>contentType</key>
		<string>critical-update</string>
		<key>date</key>
		<date>2014-06-12T11:04:50Z</date>
		<key>displayName</key>
		<string>OS X Update</string>
		<key>displayVersion</key>
		<string>10.9.3</string>
		<key>packageIdentifiers</key>
		<array>
			<string>com.apple.pkg.update.os.10.9.3.13D65.delta</string>
		</array>
		<key>processName</key>
		<string>softwareupdated</string>
	</dict>

From a total of 394 installation events, we found that participants delayed major upgrades by 80 days on average and minor updates by 16 days on average. What’s more, we found that the delay for major upgrades increased over the last three versions of macOS at the time.

These results around long delays for installing an upgrade further motivated us to take a close look at the installation process.

Take a look at the survey form we used:


Key results and insights from the field study

Before the upgrade, participants approached the process with mixed feelings

Most participants did not see security as a top concern or reason to upgrade. Instead, they mostly worried about upcoming changes to their system (e.g. losing their data or having to learn again how to do something). But even though they had mixed feelings, they did not prepare for the process (for example, by doing a backup of their data—this really surprised us!) and had confused mental models about what to expect from it.

A photo from a scene in Sex and the City, where Carrie Bradshaw asks her friend Miranda why no one talks about backing up
Several participants, to our surprise, did not back up their systems, just like Carrie Bradshaw in a memorable episode of “Sex and the City.”

During the upgrade: a long and frustrating user experience

The long duration of the upgrade process was a key issue for participants: the long waiting times made the experience unpredictable and frustrating.

“I was supposed to meet a friend, I have a social life!” said one participant, exhausted.

😤🤬😴

What made the long duration of the process feel worse, was not knowing what was going on in the process. Participants complained about the poor visual feedback indicating the progress, often consisting of vague and minimal progress bars that did not feel very informative.

Participants complained about the visual feedback during the upgrade process.

After the upgrade: was the long process worth it?

Participants noticed most of the changes in the first week after the upgrade. But these changes were not necessarily worth the frustrating experience because participants notice more negative than positive changes in their systems. For example, lost settings, bugs, crashes, unwanted features, and a worse user interface. They also noticed some positive changes (for example, new features or better performance), but these were fewer in number and milder in quality compared to negative changes.

In the end, one month after the upgrade, most participants were ok with having gone through the process, but they did not look forward to doing it again any time soon.

An illustration of reported changes after the upgrade: participants noticed most changes in the first week after the upgrade.
Participants noticed most changes in the first week after the upgrade.

Design recommendations to improve upgrades

Based on these insights about user behaviors and perceptions, we came up with several recommendations that could help improve the user experience of OS upgrades. These are some of them:

  1. Provide clear feedback on progress during the download and installation of the upgrade, clearly describing what is happening.
  2. Make the upgrade process take less time. Or, at least, make it feel shorter than it is by taking advantage of times that do not require user input to describe changes or introduce new features.
  3. Make any invisible but important security changes more visible to users, possibly reminding them about the positive benefits of upgrading in the first week after the process.

In the paper I presented at CHI 2017, you’ll find more details about the project, which also included an additional set of interviews with three security managers, giving us a different take on software upgrades.