Summary
A study where I used multiple methods (survey, field observations, diary study, interviews) to better understand the upgrade experience on common operating systems (e.g., Windows, MacOS). It turns out, installing an OS upgrade can be a long and sometimes frustrating experience, with most changes noticed only in the immediate aftermath. But there are opportunities to improve the experience.
My role as a researcher
- I was the main researcher for the project from start-to-finish: I recruited participants, arranged all the logistics for the study, moderated the observations and interviews, collected and analyzed the data, collaborating with the rest of the team.
- I also took the lead on writing a paper based on the project, presenting the results at the CHI 2017 conference.
Team: I worked with a team of senior researchers at Inria, the French national research institute for digital science and technology.
Context and research questions
How many times has your computer asked if you want to install an update, just for you to say, “Not today?” You’re not alone. It is common for many technology users to skip or postpone software updates.
In a first attempt to understand this phenomenon, we ran a survey with 65 Mac users and found that they delayed major upgrades by an average of 80 days. But why? We decided to follow up the survey with a second study, taking a closer look at the actual process of downloading and installing operating system upgrades to see how users experienced it.
We focused on two key research questions:
- How do users experience the upgrade installation process?
- What are the costs and benefits, immediately after and over a longer period?
Methods
To capture the user experience of upgrading, we used observations, interviews, and a diary study
Because we wanted to build a complex understanding of the issue and go beyond the time-limited experience of the upgrade itself, we decided to use multiple methods to collect data: field observations, interviews, and a diary study.
In the field observations, we observed 14 participants during the installation process of a software upgrade or update on one of their own devices and asked them questions about what was going on. Then, we followed the participants for a month in an online diary asking them to report any changes they noticed.
Day 0 – Field observation
We observed participants during the installation process.
Day 1 – Diary study starts
From this day and until the end of the study, participants complete a daily diary entry (online) describing changes they notice in their devices.
Day 14 – Mid-study interview
We meet again with participants to check on their progress with the diary and do a short interview.
Day 28 – Final interview
The diary study is over. We meet with participants for a final debriefing interview.
From the observations, diaries, and interviews, we had a lot of data to untangle. I decided to use thematic coding to make sense of it.
- For observations and interviews, I grouped the participants’ data by phase (before, during, and after the upgrade).
- For the online diary, instead, I decided to systematically categorize positive, neutral, or negative changes to give a sense of the effect of the upgrade on participants’ perceptions.
Key insights
Before the upgrade, participants approached the process with mixed feelings
Most participants did not see security as a top concern or reason to upgrade. Instead, they mostly worried about upcoming changes to their system (e.g. losing their data or having to learn again how to do something). But even though they had mixed feelings, they did not prepare for the process (for example, by doing a backup of their data—this really surprised us!) and had confused mental models about what to expect from it.
During the upgrade: a long and frustrating user experience
The long duration of the upgrade process was a key issue for participants: the long waiting times made the experience unpredictable and frustrating.
“I was supposed to meet a friend, I have a social life!” said one participant, exhausted.
What made the long duration of the process feel worse, was not knowing what was going on in the process. Participants complained about the poor visual feedback indicating the progress, often consisting of vague and minimal progress bars that did not feel very informative.
After the upgrade: was the long process worth it?
Participants noticed most of the changes in the first week after the upgrade. But these changes were not necessarily worth the frustrating experience because participants noticed more negative than positive changes in their systems. For example: lost settings, bugs, crashes, unwanted features, or a worse user interface. They also noticed some positive changes (for example, new features or better performance), but these were fewer in number and milder in quality compared to negative changes.
In the end, one month after the upgrade, most participants were ok with having gone through the process, but they did not look forward to doing it again any time soon.
Takeaways & opportunities
Based on these insights about user behaviors and perceptions, we identified several opportunities to improve the user experience of OS upgrades:
- How to keep users informed during the download and installation of the upgrade? Better feedback and explanations of what is happening at each step could help here.
- How to counteract the long duration of the process? If not possible to shorten it, making it feel shorter than what it is might make the experience less daunting.
- How to inform users of security changes? Making any invisible but important security changes more visible to users could help, as well as reminding users about the positive benefits of upgrading in the first week after the process.
In the paper I presented at CHI 2017, you’ll find more details about the project, which also included an additional set of interviews with three security managers, giving us a different take on software upgrades.
