How IBM uses Respondent to drive insight discovery

Eric Mahlstedt is the kind of executive who sees things differently. “I consider it a privilege to think about people as the inputs and outputs of the company,” he says.

His innovative approach has transformed IBM Cloud, Data, and AI research into the sophisticated growth engine it is today. While legacy enterprise organizations conduct design research as a development phase exercise, Eric’s B2B research team is integrated with the product management function itself, through the long tail of the product lifecycle.

They didn’t get there by following the status quo.

The High Cost of High-Tech Recruiting

Recruiting technical professionals for research studies was once “a struggle,” Eric says. A robotics process automation developer who’s been using an out-of-the-box solution for at least one year? How about 10 a month?

The significant resources required to find niche participants made dropout rates even more painful. “The longer it takes to set up your study, the longer it takes to get your findings and recommendations to the product team,” says Eric.

Eric, an experienced research expert, understood the limitations of traditional research recruiting for an incumbent like IBM: “Our competitors would eat our lunch.” His vision of a modern research operation would require higher agility, and less risk.

Modern Cloud Research 

Respondent’s cloud-based enterprise research platform was the natural solution for a cloud-based enterprise like IBM. Eric tested Respondent and similar solutions before scaling it to his team. 

Ultimately, he stuck with Respondent for B2B research interviews for three reasons: 

• Participant quality: IBM researchers easily find (and screen) specific participants, like competitors’ customers. Since participants are verified by Respondent and IBM only pays for completed studies, dropout headaches have been cured.
• Speed: Sourcing specialized participants like data scientists and systems engineers used to take 2-4 weeks of product managers’ time; with Respondent, IBM can find them in 2-3 days — and recruit past participants at no additional cost.
• Simplicity: Processing incentives through the Respondent platform bypasses the complex payment process required for a massive enterprise like IBM to pay participants directly.

Priya Noel, research lead on Eric’s team, says that their ability to rate participants on Respondent adds continuous value: “We have a lot of cross-collaboration among teams. Rating allows us to avoid low quality participants in future studies.”

The Impact

Eric’s team, and their product management peers, are delivering greater value to IBM and IBM’s customers with Respondent.

“Our executives are looking at product health funnel metrics like NPS, monthly recurring revenue, and conversion,” Eric says. “When they see underperforming trends, they want to know why. I love that we can answer them.”

IBM product teams get the benefit of qualitative user feedback throughout the entire product lifecycle. IBM researchers on more than 30 teams use the Respondent platform to derive insight beyond the traditional usability use case, like market opportunity sizing and segmentation.

Eric’s initial team has grown from five researchers to over 80 in the last five years, doubling staff on a yearly basis since 2016. “Accelerating organizational growth requires resource ownership and a clear operating model,” says Eric. “Most user researchers execute studies that help optimize design quality at the expense of delivery speed. Leveraging services like Respondent can increase efficiency so researchers can focus on their real job after delivering findings and recommendations. This requires political capital to pass the benefit of their work all the way downstream to end users through the actions and commitment of their stakeholders. At that point we learn that products don’t succeed or fail, people do.”

IBM’s Expert Tips

“A good screener can get you access to customers within two to three days,” says Priya. Below, she and Eric share their success tips.

Screening for Quality

• Evaluate candidates’ product knowledge by using (and looking for) acronyms, like WAS for WebSphere Application Server.
• Plant red herrings; ask if candidates use fake products that no expert would claim to know. If they say yes, disqualify.
• Include a screener question to disqualify consultants. 
• Request an additional five-minute phone screen. Only qualified participants will opt in.
• Ask open ended questions.
• Check the time it took for a respondent to complete a screener. If it took 30 minutes, they were probably copy/pasting search results or learning about your product while taking it.
• Ask open ended questions to (eg. describe a typical day) to evaluate their communication abilities and discriminate role differences based on wide job titles.
• Review respondent’s past participation history to find participants that may be completing multiple studies with different target profiles.

Filling Niche Studies

• Review respondent profiles (job title, company, LinkedIn profile) for the respondents that were disqualified by your screener to potentially increase your sample size for your current or future studies. 
• Favorite respondents that you think are most qualified (ideally with your stakeholders) and review later. 
• Out of respect for candidates’ time, ask the most critical qualifying questions first. Candidates will opt out sooner if they’re not a match.
• Include secondary screening questions to identify backup participants if primary qualifiers aren’t met.
• If you don’t see a LinkedIn profile for a respondent, ask for it and a link to their company website.
• Ask respondents to provide a piece of advice that only someone in their industry would know.
• Don’t forget to disqualify employees of your competitors.

Conducting Interviews

• Sign an NDA in advance.
• Inform the participant you’ll be recording the conversation.
• Leave time at the end for participant questions.
• Ask quality participants if they’re interested in additional studies so you can add them to your panel.