Background

This is a long one, because it was a pretty complex study that elicited a lot of attention from UXD leadership at Fidelity. I presented it to the group at an All-Hands Meeting (300 attendees!) in March 2016, and received an On-the-Spot Award for my efforts.

At Fidelity, I worked on a new product launch for a tool that would help users through a complex financial-planning process.

The introductory page of this tool would have to make the process approachable, motivate users to move forward, and communicate both what the tool does and what it doesn’t do. The design team opted for a flat design with parallax animation to engage users and tell the story of the complex process.

Some team members (myself included) were concerned that the design and animation were not appropriate for the expected user group. We decided to run a study that would compare preference and performance data between the flat, parallax design and a more traditional, static design with the same content.

Process

Because this introductory page of the tool was so critical, the team wanted to get statistically significant findings. I set it up as a within-subjects quant study using UserZoom. My goal was to get feedback from 400 participants who are representative of the expected user group: older, with considerable assets.

The research questions were:

1. Which of the 2 versions results in greater comprehension of what’s to come?
2. Which of the 2 versions do users feel is more appropriate to the topic?
3. Which of the 2 versions is better at driving users into the tool?

There was a critical challenge. The designs differed in two ways—the visual style, and the parallax animation. In a perfect world, we would be testing four designs (flat w/ animation, flat w/o animation, traditional w/ animation, traditional w/o animation) so we could account for multiple variables. There wasn’t enough time, however, to build four versions of the design. I built the study in a way I hoped would enable me to tease out the effects of the two variables during analysis.

The study included three sections:

1. Modified Microsoft Product Reaction Cards: I wanted to give participants an opportunity to describe the two designs while addressing the study goals, and PRCs were a solid starting point. Microsoft’s list of 118 adjectives was excessive for a quant study—I didn’t want the participants to disengage on the first page—so I cherry picked from the Microsoft list and added a few that met the study goals. I landed on 45 adjectives. At the start of the study, I established a baseline by asking participants to select adjectives they’d use to describe an ideal website on this topic. Then, after showing each design, they selected adjectives from the same list to describe the designs.
2. Comprehension: As I mentioned previously, this tool handles only part of a complex process, and we wanted to ensure that the introductory page effectively communicated what was and was not included in Fidelity’s service. The order in which each participant saw the two designs was randomized. After each participant had seen only one design, I asked a single comprehension question: What does this tool do? This was the only piece of performance data I collected, so it was arguably the most important question in the study.
3. Thunderdome: This is where the within-subjects design counted. I pitted the two designs against each other with a series of questions that spoke directly to the study goals. My hope was that the follow-up verbatim question would help isolate out the effects of visual style vs. the effects of the parallax animation.

Results

The two designs performed roughly the same on comprehension (the only performance metric recorded), but in all of the preference measures, the traditional/static design won by a significant margin.

1. Modified Microsoft PRCs: There was remarkable consensus regarding what participants wanted in a website on this topic:

Reactions to the flat/parallax design, in contrast, varied widely. More than 10% of participants described it as “distracting.”

The distribution of terms for the traditional/static design looked more like the one for the ideal website. The only negative term with more than 20 votes was “dull.”

2. Comprehension: This was the only place in the study with virtually identical performance for the two designs. About ¾ of participants understood what the tool would and would not do, regardless of which version they had seen.

3a. Thunderdome // appropriateness: The traditional/static design won this by a landslide. In the follow-up question, participants reported that the topic was too serious for a “cartoonish” treatment.

3b. Thunderdome // ease of understanding: Again, the traditional/static design won by a healthy margin. 10% of participants reported that the animation in the flat/parallax design was “distracting.”

3c. Thunderdome // motivation: the difference in preference here was less drastic than the previous two battles, but still statistically significant.

In conclusion, the more traditional approach was highly preferred over the flat/parallax design, and eliminating the story-telling scrolling animation does not appear to have harmed comprehension. The team adopted the traditional/static approach for this tool, but parallax animation is still under consideration for future design projects at Fidelity.