Background 

This was a project for The User Experience Center at Bentley University, where I was a Research Associate in 2014 – 2015.  I led the project under the mentorship of Bill Albert, Executive Director of the UXC.

The client is a media company that wanted to conduct 15-minute in-person usability studies with committed sports fans.  I needed to recruit 70 participants that fit a strict set of criteria, schedule them for a single day of testing (with multiple concurrent sessions), and manage the intake of participants on testing day.

Process

Once I was clear on the logistics of testing day (70 sessions, 7 moderators, 15 minutes + 5-minute break for each session, etc.) I worked with the client to understand the kind of participants they wanted to recruit: smartphone users, 18-24 years old, mostly male, die-hard NBA fans who frequently checked scores and news on their devices.

While I awaited feedback on the recruit screener, I sought scheduling software that would enable participants who passed the survey to schedule themselves, which would save me time and (hopefully) reduce errors.  I decided on Flexbooker, which offered a 30-day free trial and provided an account manager to help if I had any questions.

Once the screener questions were approved, I set about creating the survey logic in Qualtrics.  I decided to have all respondents answer all questions in case we had difficulty finding participants who fit the criteria and needed to ease the requirements at a later date.  To automate the scheduling, there would be two different end-of-survey messages: one with a link to the scheduler for respondents who passed, and one with a generic note of thanks for respondents who did not pass.

Part of the recruit requirements included quotas: we needed no more than five female participants, at least twenty tablet users, and at least twenty participants who were familiar with the client’s current UI.  I created quotas in Qualtrics for 65 men, 50 non-tablet users, and 50 not familiar with the client’s UI and incorporated them into the survey flow logic.

I distributed the survey via a Bentley University distribution list and held my breath.  In the first few hours, we had 420 respondents… but only one appointment booked.

I pulled a list of respondents from Qualtrics and filtered for passing responses, which resulted in 80+ qualified participants.  Was it possible that 80 qualifiers were not available on testing day?  Probably not.  Clearly something was wrong in the survey logic.

Fortunately I had captured all the respondents’ email addresses, so I reached out to them directly with a link to the Flexbooker scheduler.

I still wanted to get to the bottom of the problem, so I examined and re-examined the survey.  After verifying that the survey flow, end-of-survey messages, and links to the scheduler were correct, I finally discovered the error.  The quotas were active no matter how the rest of the questions were answered, so that once 65 men (or 50 non-tablet users, or 50 people unfamiliar with the client’s UI) took the survey–regardless of whether they passed–no man would receive the message with the link to the scheduler.

Mystery solved!

Results

I recruited 68 participants who fit the client’s criteria, and participant management on test day went smoothly.  The client was happy with the results of the usability study and with the quality of the recruit.

More importantly, I learned valuable lessons about survey quotas, automation, and planning for hiccups.