How many responses is enough

There is no recommended maximum or minimum number of responses. Survey responses from previous PSPs have varied from 100s to 1,000s. Steering Group members should be mindful of the volume of responses that they can manage and think about the response they would like and are likely to achieve knowing their clinical area and the numbers of patients and clinicians involved. Attracting a high number of respondents may seem desirable, but this can create problems if a PSP does not have the resources to process a high number of submitted uncertainties. To put it into perspective, if 1,000 people take part in the survey and submit around three questions or comments each, that will produce 3,000 lines of narrative data to assess.

In addition, it should be noted that this is a qualitative survey aimed at generating questions and themes: a high number of respondents may not necessarily result in more or better uncertainties and the range of themes needed may come from smaller numbers of responses. When thinking about numbers of responses, PSPs should consider aiming for quality not quantity, whilst ensuring that there has been reasonable representation from the range of possible stakeholder groups.

It can be more effective to aim for a diverse range of respondent types by taking a targeted approach when promoting the survey, rather than simply looking for high numbers. Although of course if the numbers are too low, it may be difficult to say that they adequately represent a community's concerns. Some PSPs have created a target list of the groups of people they would ideally need to hear from. The PSP Steering Group needs to be confident that responses represent the community effectively, cover a broad range of issues, and are from a good balance of patients, carers and clinicians. If the range of topics in the submissions is too narrow, there could be important evidence gaps that are missing.

Some Steering Groups have taken active measures to ensure they receive a manageable number of responses while still consulting with a wide range of constituent groups. These measures include:

• placing a limit on the number of uncertainties each respondent can submit (for example limiting it to three or fewer)
• agreeing the maximum number of responses that can be processed and closing the survey on achieving that - it is advised that Steering Groups identify the range of groups they want responses from, and that targeted communications are planned to ensure they reach those people, rather than using solely mainstream routes of communication, which may generate a large response but without hearing from some important groups
• agreeing in advance to stop the analysis once saturation point is reached (when no new themes are emerging).