Questionnaires: Procedures and Factors
3. Overall research question or hypothesis
4. Type and size of sample
5. Terms and concepts
6. Question structure and response options
7. Constructing questions
8. Layout and design
9. Measuring concepts
ANALYSIS, INTERPRETATION AND PRESENTATION OF QUESTIONNAIRE DATA
10. Data processing and analysis
11. Presenting your data
12. Interpreting and discussing your findings
1. Survey
research
Questionnaires, with interviews, represent the two most common modes of data collection in
the social sciences.
It is almost impossible to grow up and function in modern society without being asked to complete a questionnaire or respond to interviews on a fairly regular basis. As students of the social sciences, you are regularly asked to accept interpretations of social, political or economic behaviour that derive from questionnaire or interview survey data.
To be able to fully participate in the academic community, it is necessary to understand how such surveys are conducted and the techniques and conditions of interpretation of survey data.
2. What is a
questionnaire?
A questionnaire is a data-gathering device that elicits from a respondent the answers or
reactions to pre-arranged questions presented in a specific order. Questionnaires are
flexible and adaptable to a variety of research designs, populations and purposes.
Questionnaire surveys are a form of research which depend on the frankness of the subjects' responses. They need to be designed and carried out carefully so that they provide a genuine reflection of the attitudes and beliefs of a group of people.
Pitfalls to avoid
There is always the danger that, as questions become more searching, respondents will
either try to comply with what they think the questioner wants to read/hear, or to project
what they think will result in a positive image of themselves.
It is well known, for example, that respondents tend to agree with questions, whether they are positively or negatively constructed:
For example:
Do you agree that most people are inherently peaceful/aggressive?
People tend to provide an affirmative, agreeing response to whichever option is given. Overcoming the desire to present a positive image requires a subtle approach on the part of the questioner.
A word of caution
While Questionnaire surveys may be the most popular form of social science
research, they are also difficult to do well. There are a many factors to
consider in designing and conducting a successful questionnaire survey.
This Academic Grammar program uses authentic and textbook examples to alert you to the most important factors to consider in questionnaire design.
DESIGNING A SURVEY-BASED RESEARCH PROJECT
3. Overall
research question or hypothesis
As Dillon (1990) puts it, the first step is not to write questions but to define the
purpose for asking. This means you should formulate questions the information from the
survey is designed to answer.
Inexperienced researchers are often tempted to rush into formulating survey questions without first having defined the overall research question. This mistake is apparent only after the survey, when the collected responses are found to be unusable, incomplete or even incoherent.
Problem definition
Investigations are goal-oriented. This means the goals must be specified and the
problem(s) to be investigated must be defined.
What is a problem?
In operational terms, people tackle problems to which solutions are possible. Generally,
the easier a problem is to define, the easier it is to solve.
The more difficult problems are those which are really problematic situations. This refers to a set of conditions which is unsatisfactory, whose underlying causes are what need identifying if a practical solution is to be found.
Here is a medical example:
While reading, you get a headache, which makes it painful to read. You can define the problem as:
I have a headache or I feel pain when I read
These are both symptoms.
Even setting up a hypothesis: "If I read for long, I get a headache" is not very helpful. It shows that even if something is true, it does not offer a solution. It only provides a superficial explanation.
You expect to be able to read, and the situation is abnormal and problematic. To get relief requires an examination of the conditions under which you are reading. This is likely to involve examining the lighting conditions, the quality of your eyesight, the print size of the reading material, and so on.
4. Type and size of
sample
The second step is to determine your population and the type of sampling you intend to
carry out.
Beware of the cliche random sampling - pure random sampling is rare, expensive and very difficult to guarantee. More usual at this (HKU student) level is accidental or systematic non-random sampling, where you might stand in the foyer of a building and end up with a sample of those students who agree to be interviewed.
Type of population
In a situation where you only have the resources to look at a small number of
respondents, and to cover a small geographical area, it is best to select a subject for
study and a population which will yield interesting results.
In the case of the first EAC project, the subject and the population are well matched. University students are well-qualified to express opinions on the subject, and, an additional bonus, there are fewer risks of misunderstanding or misinterpreting the questions (scroll down to take a look at them).
It is well known that a lot of claims about human behaviour and attitudes are based on research conducted with either students or female homemakers, because of their easy availability: one close to the site of most research - universities, and the other as the easiest accessible population in the field.
Size of sample
A guiding rule is: the more homogeneous the population, the smaller the sample needs
to be to accurately reflect the characteristics of that population. How narrow you
make your population is a function of the nature and generality of your questions, and
vice versa.
Another factor determining sample size is the degree of precision or number of variables you want to test. These factors account for variation in response:
- social class
- religion
- race
- sex
- age
If you intend to divide up your sample based on sex or age differences, then you need to calculate sample size in terms of the smallest sub-division of your population. This means a sample of four girls and twenty-four boys is not representative of variability according to sex.
In the past, Social Science students have compiled questionnaires which have ignored vital differences even within a small, focused population.
For example, the question: Which Faculty? Is not as revealing as asking respondents about:
- the degree stream within that Faculty
- the year of study
- subjects taken
- their major and minor subjects
- reasons for their choices, and so on.
CONSTRUCTING THE QUESTIONNAIRE
5. Terms and concepts
Surveys involve the negotiation of concepts and terms with respondents who might not know
what you mean by a term or a question, but are pressured by the situation to comply.
This can result in conflicting data which is difficult to interpret.
Take a project on the Medium of Instruction in H.K.. schools. What do you mean by
- medium of instruction?
- bilingualism?
- English-medium education?
Will all your respondents, even in an exclusively university-educated population, understand the same thing by these terms?
This is a clear illustration of the way in which terms may be constructed differently by people who seem to share the same background, experience and situation.
6. Question
structure and response options
Questionnaires generally comprise a combination of open and closed questions,
providing balance between depth and authenticity of information, and fixed-option data
which are more easily quantifiable.
Each type has advantages:
- for research exploring feelings, attitudes or types of behaviour; and where resources are plentiful, open-ended questions are preferable
- for demographic or performance data, and where time, subject or topic sensitivity, objectivity and ease of scoring and analysis are important, closed questions are more practicable
Within the closed question range, there is are a number of response options, from the simple Yes/No choice to the so-called Likert 5-point scale (Strongly agree--> Strongly disagree) or the ‘cafeteria-style’ checklist, offering a range of options for selection.
Examples of these, and a description of the advantages of the various formats, are included in Chapter 10 of M. Youngman. Designing Questionnaires (1978).
7. Constructing
questions
When you compile a questionnaire it is important that you use simple language in order to
convey the meaning of your questions clearly. Your questions should be easy to read and,
in some cases, clearly relevant to the subject under investigation.
A good questionnaire should create a feeling of importance in the respondent, a feeling that the research is relevant, and that cooperation is vital.
Here are some specific things to avoid:
- double-barrelled questions, for example: Do you like school and do you study regularly?
- leading or suggestive questions, such as: Do you agree that most Chinese prefer X?
- a pattern of positively or negatively constructed questions which are targeting opposite values. There is a tendency for respondents to want to comply, to agree. This is one reason questionnaires should feature a balance between positively and negatively constructed questions, and why it is wise not to have a string of agreement-oriented questions (Strongly agree —> Strongly disagree)
- ideologically-loaded expressions, called red flag words (using terms such as ‘communist government’ to elicit negative responses which are then interpreted as indicating negative feelings about socialistic values)
- long, complex questions; aim at short, clear and direct questions
- putting the main idea at the beginning of the question; respondents will tend to expect end-focus of the key information you are seeking
8. Layout and design
Basically, the questionnaire should not be too long, too complex, or too confusingly
varied in format. See Youngman (1978) for advice on questionnaire layout and design.
9. Measuring concepts
This is the trickiest part of questionnaire design: how do you measure abstract
qualities/concepts? Use of indicators; a series of more concrete questions
can be used to determine opinions/beliefs on more abstract phenomena.
For example, on attitudes to L1-medium schooling:
- First question: Do you agree with it? (Yes)
- Second question: Would you send you child to an L1-medium school? (No)
Such tensions are common if an investigation is on a rich, problematic issue with no easy answers or positions, people exhibiting paradoxical positions, and so on.
ANALYSIS,
INTERPRETATION AND PRESENTATION
OF QUESTIONNAIRE DATA
10. Data
processing and analysis
Data does not speak for itself. Raw data must be classified and/or
coded before it is useful.
It is important to think ahead to this phase when you are constructing your questionnaire and numbering and coding the questions. Keep each response as a discrete item, separately numbered; this certainly makes computerised analysis and sorting much easier.
You are strongly advised to explore Windows options for handling data and representing them in graphic form.
11. Presenting your data
Once you have your responses, how can you best organise and present them to an audience:
your tutor, colleagues, simulated professional audience?
Graphs and tables
These can save a hundred words, but can also be wasted if poorly
exploited.
Verbal exploitation of graphics
A hopeless task if the graphics are obscure.
Percentages vs. absolute figures
The reader or audience finds it easier to expect all data to be expressed as
a proportion of 100%. In a presentation, to give only absolute figures (33 of the 69
males, but only 46 of the 136 females felt that ....) is an insult to the audience.
The solution is either to:
- give both (this can make your graph look too crowded), or
- place the absolute figures for the population at the top (or foot) of the graph, and then deal in percentages.
Rank ordering responses
This is a simple procedure, but surprisingly often ignored.
For example:
There are four closed option responses given to the question:
What's your favourite language?
A series of languages are listed alphabetically, with an additional Other option.
You might have the series, and responses, as follows:
English 28
French 7
German 4
Italian 13
Japanese 18
Others: Spanish 8, Russian 2(from a sample of 80 students)
Which of the following looks easier to process?:
|
|
|||||||||||||||||||||||||||||||||||||
With experience, you should develop an understanding of how you want to present the data, and this will help you with the design of the questionnaire. It is important to think ahead and plan through to the interpretation stage before actually constructing your questionnaire.
Using a computer will not improve the quality of your data, but it will enhance the clarity of your presentation.
12. Interpreting
and discussing your findings
This is a large area, but there is one major principle you should bear in mind:
Only make claims in proportion to the size and scope of your population, and on the specific questions you were asking.
This means that the language you use needs to carry the tentativeness or assertiveness to reflect the strength or weakness of the claim.
For example, you can talk of supporting or tending to confirm a hypothesis. Also, be careful about claiming changes in opinion on the basis of a single survey (for example, people increasingly believe...).
In terms of the broader implications of your findings, you need to consider, and discuss carefully how they relate to the broader questions you started out with. What do your local population's responses contribute to the wider debate?
In an investigative project, this will feature the full mirroring of the literature review in the introduction.
For example, on the medium of instruction issue, you would be expected to return to the question of whether or not local (student?) opinion tended to support the mother tongue medium protagonists in Hong Kong as a whole.
This is part of the hourglass pattern, of moving back from
- the specific to the general
- the concrete to the abstract
- the local to the universal
You ultimately want to ask yourself where your findings fit into the big picture.
Note: A video: ‘The Investigative Project’ is available in
- the Main Library, Audio-Visual section
- the Language Centre's Practice Lab (K.K.Leung Building, Room 605)