Representability

We encounter the problem of representability when conducting surveys. Web surveys are relatively inexpensive as long as a nonprobability sample is used. If a closed population is studied (e.g. associations, students of a faculty, employees of a company) and email addresses are accessible that is not an issue. The problem arises when we wish to draw conclusions about the general population on the basis of web surveys where the probability of an individual being recruited is unknown.

Statistical inference – which usually includes a confidence interval as a measure of the risk that we are wrong – is based on the fact that the probability of an individual being included in the survey is known a priori and for the entire population.

Of course, we cannot apply a generalization to the entire population for an online survey in which we invited friends and random visitors to some websites. Similarly, we cannot do this by walking the two streets of one city and asking people for their opinion. Since we do not know the probability of inclusion, we do not know the probability of being wrong.

If such a probability is unknown, one can at best presume – based on past experience – that the findings apply to the entire population. However, past experience usually varies greatly. This problem is similar to television or radio polls, where the opinion of the general population can be completely different from the opinion of a few thousand individuals that decided to participate.

If one wishes to draw conclusions about the entire population based on web surveys, there are three options:

1. When interpreting the results, it is important to openly admit to the use of a nonprobability sample, so that the users are not misled;
2. Recruitment must be dispersed as much as possible. Web surveys RIS which recruited from a very broad selection of internet users matched telephone survey results for all variables. WageIndicator surveys that are carried out in multiple countries show that web surveys can be representative of a wide array of sociodemographic characteristics;
3. Data weighting can help to some extent (but not necessarily).

What can a student or a researcher without significant funds do if they want to infer on the whole population?

• Unfortunately, times of cheap web surveys where several hundred answers were gathered by just posting on a random more popular website are gone;
• The times of sending massive amounts of invitations through email are also gone. Expert codices now prohibit such activities;
• Clicking on banners with invitations is also less and less effective – even research MOSS inviting people to a survey with advertising on more than a hundred most popular websites in Slovenia can only gather a couple of thousand responses.

To carry out a high-quality web survey based on the general population, one requires:

1. A first-rate internet panel that is supported by an organisation that follows professional standards;
2. Or sufficient advertising resources;
3. Or a very expensive system of traditional recruitment (mail, telephone).

In case of a limited budget there are two possibilities:

1. Either to obtain help from a commercial internet panel (if there are free capacities);
2. Or to publish the survey link on web portals, as well as publish texts on as many websites, forums and blogs as possible, send e-mails to business contacts and acquaintances, try to get published in traditional media, etc.

All this time we have been talking about the general population and not about members of a certain club, society, organisation..., where the problem is much smaller; we only need to obtain permission to invite all potential respondents (members, employees) into a survey.

More information about web surveys can be found on the WebSM webpage.

<< Back to Web Surveys