A sound understanding of audiences or visitors is fundamental to any cultural organisation’s mission and business. Surveying audiences enables us to understand what kinds of people do and don't engage, how often, with whom, what they hope to get from their visit, and how they can be reached. High quality information on these issues helps us to make better decisions. To do so with confidence, the information needs to be suitably robust. We need to be sure it is representative of any audience and any activity for which we are planning or reporting.
This guide explains how and why you can use Audience Finder data for planning and reporting with confidence, giving details on sampling, dealing with respondent bias, and the weighting of results.
Overview
Audience Finder includes a standardised, national survey framework which cultural organisations can use to collect information about their audiences. As well as enabling comparison and a sectoral understanding of audiences and common engagement issues, it guarantees a reasonable quality of information for decision-making at organisational level. It includes well designed, comparable questions, and is set up to ensure that data is representative and robust, accounting for issues of sampling, respondent bias and weighting. However, it is important to understand how these issues affect the data when drawing certain conclusions.
Survey sampling
Constructing a sound audience sample is key to getting representative and reliable audience data. That means getting the right number and type of people to answer the survey. The “sample frame” is the plan to get the right sample across the full range of events and activities you need to understand. To make a good sample frame, you should consider how many people will need to respond to the survey in order to produce meaningful results; how many are likely to respond; and how, where and when you’ll conduct the survey (interviewer-led, on exit, during visit, online, rolling or one-off); and how audience members will be selected to participate. For the survey results to be reflective of the whole audience for the offering under exploration, you need a sample which is a) randomly selected and b) of suitable size.
To get a random sample, you need a structured process – eg every tenth person entering the foyer – to remove interviewer bias (ie some types of people to be more represented than others). Getting the right process will help mitigate over-representations of, say, a particular age, gender or ethnic background.
The sample size appropriate for any aspect of activity (eg particular event, or type of event, exhibition, collection, late night opening etc) is determined by the size of the population you’re trying to describe (eg all visitors, or just those who go to a particular exhibition) and the level of detail you will want to go into in analysis.
This can be quite complicated which is why we always work with organisations to help them construct the most appropriate sample frame when setting up Audience Finder surveys, including giving them a target number of responses to achieve to ensure robust results.
If you want to understand how different types of visitor compare to one another you will need to increase your overall sample size. This will ensure you collect enough data from those from each type of visitor to allow you to determine differences (and similarities) with a good degree of confidence. This is particularly important for monitoring diversity and understanding the experiences of those from different demographic groups. For example, if you want to understand the experiences of those who have a disability, you will need to speak to a large enough number of visitors that the results from those with a disability are based on a big enough sample to be robust in and of themselves.
Audience Finder resources also help organisations to get this right: they include fieldworker guidelines, webinars and FAQs, giving advice on constructing sample frames, managing data collection and other general survey tips. The Audience Agency also offers fieldworker training.
Respondent bias
Even with a good sample frame, other factors can introduce bias into the results of a survey. For example, families with small children might be less at leisure to respond to a survey and could be under-represented in an organisation’s results. Language barriers might put off non-English speaking visitors. First-time visitors may feel less entitled to share their opinions. If you have such concerns, especially about groups who are important to your organisation, there are a number of things you can do. Firstly, try to gain an understanding of the issues you suspect. You could do this by talking to a small group of people with those characteristics. Talk to The Audience Agency too, we may be able to advise you on similar situations elsewhere, or carry out a cross-venue exploration.
Then think about how you could reset your data-collection method, or sample frame to mitigate the effect. For example, to address the family issue, there might be means of planning the point of invitation to undertake the survey (while children are participating in activities?), or the means of completing the survey (by online, post visit survey) around family neutral factors. Or in the case of non-English speakers, surveys could be made available in a range of relevant languages. Such measures would need to be mindful to not over-compensate, and introduce an opposite effect of over-representation of these groups.
Again, The Audience Agency can advise, and our fieldworker and sampling resources and training explain how to avoid respondent bias.
Weighting
The construction of the sample, and controlling for bias in the sample frame, are perhaps the strongest means by which organisations ensure representative information, but where circumstances do not allow these approaches, then weighting can address bias. Weighting is an allowance or adjustment made in order to take account of special circumstances or compensate for a distorting factor.
Weighting in Audience Finder benchmarks
Audience Finder survey data results are NOT weighted. So if you are looking at the averages and benchmarks in Audience Finder, bear in mind that they:
- represent all audiences in that benchmarking group, but are not weighted to take account of – say – disproportionately small numbers of family respondents.
- however, we can drill down into the data to look at how the responses of a particular sub-group - like families – differ from those of all audiences. Ask The Audience Agency team.
- represent the audiences of all venues in a benchmarking group, but are not weighted by the size/resources/audience numbers for each venue. So results will be biassed towards the venues that submit the greatest amount of data – rather than those that have the largest audiences.
- in future reports, we plan to look at this form of weighting in more detail. We will look at how different audiences are for museums of different sizes.
- for the most part, the larger the venue, the larger the data-set they tend to submit, so there is a kind of natural weighting applied, and the benchmarks in Audience Finder are therefore biased towards describing the larger audiences of larger venues.
- if venue size or type is an important factor in how you want to look at the data, however, The Audience Agency can weight particular data sets to give you an adjusted picture.
The particular bias which may concern one organisation will not obtain to all. For example, one organisation might be minded to take steps to ensure a representative sample of responses from family audiences, whilst for another the focus might be around trying to ensure the representation of particular ethnic groups. The number of potential biases are potentially endless, and a mandatory control for a number of them would make the logistics of conducting the survey too onerous for a majority of organisations. Moreover, the sheer scale and variety of participating organisations in Audience Finder do mitigate such biases in the aggregate to a large extent.
Weighting the results for your organisation
For individual organisations that perceive a particular bias important, there is the option to keep a “refusal log” and to then weight responses accordingly. The Audience Agency can advise you on recording refusals to control for bias on features such as family, age, ethnicity, sex etc. It can be built into your ongoing Audience Finder survey, or run for a discrete period, to identify the levels of respondent bias and therefore the weighting to be applied to your results. These are paid-for options in addition to the free Audience Finder survey service.
As part of our Quality Assurance procedure and with feedback from museums, we now proactively and regularly monitor response rates and data quality for key groups such as families. This means that we can check in with organisations to ensure their sample frame is on track.
How The Audience Agency is innovating to help art galleries find out more about their visitors in a fun and interactive digital way.
Harnessing the power of Audience Spectrum to map out a strategic, data-driven approach to embracing new audiences and encouraging anxious returners in South Wales's altered post-pandemic landscape.