User research panel manager Jason Stockwell reflects on what happens when a user research project grows suddenly, and quickly, and how this needs to be managed to keep things on track.
Back in November, I wrote about why we have research panels in Defra. Since writing that post both the team and the programme have changed significantly. The programme and the panel have gone from being used for six researchers to over 20 researchers. This doesn’t mean the panel is now impractical, but it does mean there are extra considerations for the panel and booking participants from it.
Of course, with this change comes a change in the processes associated with managing a panel. What once was a full-service recruitment for all user researchers has had to scale back. With that comes a “luxury” list of what panel management could be, and a series of tasks panel managers must do to do their job.
I want to share some of the learnings from this work about growing and managing a panel within a complex programme. These are some of the elements that can change in a growing programme:
Managing the panellists
Let’s talk about the most important thing on the panel, the people. We were in a lucky position where we have 2,500 participants who are (mostly) keen to take part in research and signed up on a panel to do this. There are still considerations we need to take with these panellists before we get them involved in research,
- Panel exhaustion. The first challenge might be the most obvious, when booking over 130 sessions a month this panel is going to get exhausted very quickly. We currently combat this by monitoring interactions with participants and starting a “waitlist” for any participants who are keen to get involved as soon as the next research session becomes available. This limits the volume of participants we send each research round to which means members of the panel are not bombarded with irrelevant research pieces.
- No shows. No shows happen, there is only so much you can do when inviting participants to research. What we do is follow up post-session and understand why the participant failed to show and understand what part of the process didn’t work. We are still likely to get participants that have urgent issues come up, but at least we can mitigate against tech issues or confusion with joining instructions.
Managing the data
Beyond the people themselves, there’s the data produced by these research sessions and how we deal with the volumes of data needs more thought given the complexity of some major projects.
- FOI Requests. Pseudonymising participant data and including that information in recordings and insights makes it easier to understand where that participant has had input on research within the programme. This way, when a request comes in, we can match who that participant is on the panel, search by the previous research sessions they have been involved in, when they signed up, view any consent forms and retrieve any data we have on that participant.
- Data storage. The next element is being consistent in where data is stored. This should be in a password-protected and restricted area of a server. This makes it easier to track permissions and delete the data in time. Going back to the previous point on FOI requests, consistent file structure makes it easier to understand what data the programme holds on a specific participant. We work a lot with external suppliers, there needs to be an understanding on how we process recordings and any other data collected that this should follow processes defined by the rest of the team. This will likely look like a restricted area of SharePoint, with defined permissions on when this data should be deleted.
Sometimes people are hard to find, but that’s not your fault
Finding the right people for research opportunities need time. Some participants you might never find, or even if you do find them, they might have had a difficult experience with the department previously so might not want to get involved again.
- Finding niche participants. Getting niche profiles to take part in research can be tricky, especially with short timeframes and working in two-week sprints. The tactics and methods to find these participants don’t change. Internal networks, trade associations and external campaigns, these are consistent. It’s the variables in user recruitment that are more around the screening process to ensure the research sessions are with the right participants and being more flexible with research dates and staggering this objective over multiple sprints when we find the right participants.
There is only so much we can do when finding some people. The people managing the panel are limited by other research requests coming in. That’s just a reality.
Jason Stockwell is the user research panel manager on the Collection and Packaging Reforms and Waste programme.
In Part Two of Jason’s blog, we’ll hear about the challenge of constant recruitment, how to bring teams together and the very real prospect of personal burnout.
If you are interested in becoming a User Researcher in Defra Digital Data and Technology, check our jobs page to see the latest vacancies. You can also follow our LinkedIn page for all the latest blog posts and job openings.
1 comment
Comment by Tim Kleinschmidt posted on
Really interesting reflections Jason. I really like how you are managing participants, especially how you've addressed the challenge of participant exhaustion. Looking forward to hearing more in Part Two.