We encounter representativity problem when conducting surveys. Web surveys are relatively inexpensive as long as a nonprobability sample is used. If a closed population is studied (e.g. associations, students of a faculty, employees of a company) and email addresses are accessible that is not an issue. The problem arises when we wish to draw conclusions about the general population on the basis of web surveys where the probability of an individual being recruited is unknown.
Statistical inference – which usually includes a confidence interval as a measure of the risk that we are wrong – is based on the fact that the probability of an individual being included in the survey is known a priori and for the entire population.
If such a probability is unknown, one can at best presume – based on past experience – that the findings apply to the entire population. However, past experience usually varies greatly. This problem is similar to television or radio polls, where the opinion of the general population can be completely different from the opinion of a few thousand individuals that decided to participate.
If one wishes to draw conclusions about the entire population based on a web surveys, there are three options:
- When interpreting the results, it is important to openly admit to the use of a nonprobability sample, so that the users are not misled.
- Recruitment must be dispersed as much as possible. Web surveys RIS which recruited from a very broad selection of internet users, matched telephone survey results for all variables. WageIndicator surveys that are carried out in multiple countries show that web surveys can be representative for a wide array of sociodemographic characteristics.
- Data weighting can help to same extent (but not necessarily).
What can a student or a researcher without significant funds that wants to infer about whole population do?
- Unfortunately times of cheap web surveys where several hundred answers were gathered by just posting on an random more popular website are gone.
- Times of sending massive amount of invitations through email are also gone. Expert codices now prohibit such activities.
- Clicking on banners with invitations is also less and less effective - even research MOSS inviting people to a survey with advertising on more than a hundred most popular websites in Slovenia can only gather a couple of thousand responses.
To carry out a high quality web survey based on the general population, one requires:
- a first-rate internet panel that is supported by an organisation that follows professional standards,
- or a sufficient advertising resources,
- or a very expensive system of traditional recruitment (mail, telephone).
In case of a limited budget there are two possibilities:
- either to obtain help from a commercial internet panel (if there are free capacities),
- or to publish the survey link on web portals, as well as publish texts on as many web sites, forums and blogs as possible, send e-mails to business contacts and acquaintances, try to get published in traditional media, etc.
All this time we have been talking about general population and not about members of a certain club, society, organisation..., where problem is much smaller: we only need to obtain permission to invite all potential respondents (members, employees) into a survey.
More information about web surveys can be found on WebSM webpage.