Date: Spring 2022

Background

Coinbase Pro is Coinbase’s legacy advanced retail trading tool, and it operates as a wholly separate product from the rest of the company’s retail-investor offerings. In order to give advanced traders access to the robust features of Coinbase’s main app, the team built a new advanced trading tool within the main app that had (mostly) feature parity with Pro plus a frequently expanding set of new features. 

After building and launching the new advanced trading tool, however, users were slow to migrate from Pro. We needed to understand why so we could remove the friction points and migrate millions of users before Pro’s scheduled deprecation. 

Process

The team needed at least notional research results fast as this was an extremely high-profile, high-revenue-impact project with a short timeframe for migrating users. But I also understood that hesitation and motivation are nuanced. A deep-dive would be necessary to ensure that any changes we made to product and/or messaging would get to the heart of the user problem.  

I opted for a mixed-methods approach with an initial survey + follow-up interviews.

For the survey, I targeted users who had been invited to the new trading tool via a banner announcement in Pro (the legacy tool) but who had not taken any action. The banner has two call-to-action buttons (CTAs): “Transfer funds” and “Check it out”. I hypothesized that the “Transfer funds” CTA was too committal. To tease out user reactions to the CTAs separately, I made the survey between subjects, with half of the respondents responding to each of the CTAs. 

The very first survey question blatantly violates survey best-practices: it’s leading, and it asks respondents to self-report what they would do. 

That’s right – I violated cardinal survey-design rules! Here’s why:

1. The data for each call-to-action (CTA) could not be taken at face value, but they would be useful for comparing perception of the two CTAs.
2. There was little risk of anyone taking the individual responses at face value, because all stakeholders knew that these users had, in fact, seen the invitation banner in real life and not acted.      
3. It enabled me to branch the rest of the survey based on whether the response to the first question leaned positive or negative.
4. Bonus: The results serve as a great reminder to stakeholders that people don’t do what they say they’ll do.

I followed up the horrible question with open-ended and multi-select questions on what motivated a positive response or caused a hesitant response. We received more than 500 responses and shared out results immediately so the team could start strategizing fixes.

In the follow-up moderated interviews with 9 Pro users, I opted to target users who had not been exposed to the new advanced trading tool invitation banner, because I wanted to capture their first impression. I also took the opportunity to get high-level usability feedback on the new tool. 

Results

As suspected, users in the survey and interviews weren’t ready to commit to transferring their funds from one platform to another after seeing only the banner invitation in Pro. Additionally, while survey responses to pick-list questions indicated that users were concerned about the potential of higher fees in the new tool, open-ended responses and interviews revealed that inertia was the first problem we needed to solve. This was the nuance we needed to lower barriers to migration.  

The team acted swiftly to launch a follow-up experiment. Half of Pro users saw a new and improved invitation banner, while a control group saw the old one. After only 3 days, revenue for the new trading tool increased 120% among the experimental group.