Questionaires     

Graham Tall      research@grahamtall.com     September (2003)

Introduction

This web-page is organised into the following sections:

I. General Guidelines for Designing and Using Questionnaires
II Effective Questions
III Types of Questions
IV. Strengths of Questionnaires
V. Weaknesses of Questionnaires
VI. Response Rate
VII. Organising Questionnaire Data for Analysis
VIII.. Analysing Questions

I.     General Guidelines for Designing and Using Questionnaires

Questionnaires and interviews share a common purpose and can be considered as alternative ways of collecting the same information. Indeed in their simplest form the only differences between them are that:
i)        A questionnaire is ‘written’ and the interview a ‘spoken’ series of questions.
ii)       Limited, if any, interaction can take place in questionnaires.
iii)      Greater confidentiality is possible with questionnaires - the researcher can organise the
          questionnaire so that she/he does not know who gave a particular response.

The information given below recommends an eclectic questionnaire sharing aspects of both empirical research
paradigms:

 

Scientific Paradigm

Socio-Anthropological Paradigm

Questionnaires

Multiple-choice/Attitude Closed questions*

Open-ended-questions*, Essays Diaries etc.

* The differences between open-ended and closed-ended questions are two-fold. Open-ended questions:
   i) allow the interviewee to reply using their own words and ii) enable a wider range of possible responses to be given.

Since questionnaires are inflexible and  a relative bore to complete it is essential to encourage the respondent by both explaining why it is worth filling in the questionnaire and making it attractive, easy to fill in and quick to complete.

Hence, before any questions are written, consider carefully:

a)     the purpose of the questionnaire i.e. it's research aims
b)     list ideas in order of priority
c)     how the questionnaire is to be analysed.

Hence many of the guidelines below are related to making questionnaires easy to fill in and quick to complete.

a) 

 

 Make the questionnaire as short as practicable: Unless the responders are very committed, or captive (a class of students) aim at a maximum of 5 to 10   rather than 20 to 30 minutes.   It is  self evident that there must be an inverse relationship between the length of the questionnaire and the percentage of returns obtained.
b)

 

 Number the questions:   N.B. The collation of responses, on to a database, is quicker and  more accurate, when one person reads the information and another enters it - coding the questionnaire makes this and later cross-checking much easier.
c)


 Pilot the questionnaire: There is no such thing as a perfect questionnaire. Put the draft on one side for a few days and then try to answer it. Ask your dissertation/thesis supervisor and relevant colleagues to read the questionnaire to help ensure that you have asked relevant questions. Pilot it with a group of comparable individuals to check that the questions are clear and provide the range of feedback that you expect.
d)
 Don’t give respondents too much time to fill in the questionnaire:   i.e. don’t distribute it and ask for it back in a month. If the questionnaire is not answered quickly the probability is that it will be put down and lost.
e) 

 

 Ensure the questionnaire looks attractive and is easy to fill in:   For example, with open-ended questions don’t draw lines telling the respondent how much they should write.  The additional advantage of a blank space is that respondents can write small if they wish to give a very full answer.
f) Introduce the questionnaire clearly:  Explain its purpose and make it interesting to the potential respondent.
g) 

 

 Give clear information on how the questionnaire is to be returned:   If it is posted either arrange collection or include a stamped addressed envelope. Try and keep a check on who has replied so that if the response rate is low, you have some chance of discovering why. - But, as a point of ethics, if you have promised TOTAL confidentiality, then:
i) 
 you mustn’t write the names of those who have responded on their questionnaires - somebody else might see them.   Keep a totally separate list which solely indicates who has replied.
ii) 

 

 Don’t attempt to identify who gave a particular response, e.g. by checking their writing etc. The real advantage of a questionnaire over an interview, is  that it allows  respondents to give their views ‘without fear or favour’ - be scrupulous to retain that strength!
h) 
 Report the findings back:    Apart from being simply good manners, it is also essential research practice if you subsequently wish to question/interview those same individuals again.   Top of the Page

  II.    Effective   Questions

a)

 

 Questions Must ReferTo A Single Aspect.  The major underlying weakness of questionnaires is that neither the researcher nor the respondent can ask for clarification, this means that it is crucial for questions to be as clear and simple as possible.  The normal mechanism to ensure this to use short multiple choice questions wherever practicable.
b)

 

 

 Questions Must Be Unambiguous: if a question can be misunderstood or mis-interpreted it will be!   Whereas in an interview it can be productive to start by asking questions which are very general, the danger in a questionnaire is that such questions will produce an equally general :responses   What do you think of the XXXXX? Responses such as ‘not much’, ‘OK’  might let the course organiser know if there are any problems, but they don’t  provide the information needed to improve the course.
c)

 

 

 

 Consider The Type of Question Which Will Best Obtain The Information Required: many questions which appear to be open-ended can be converted into a multiple choice format:  Questions such as ‘How many students attend your institution?’ rarely need a detailed response like ‘1,563’.   Instead begin the analysis process.  How is the information going to be used?  All that might be required is a decision on   whether the institution is small medium or large; hence think about size and provide a range of answers such as:

Less than 600 children               601 to 1200 children        More than 1200 children
d)

 

 

 

 

 

 


 Multiple Choice Questions Must Have Exclusive Categories.   With the use of number categories such as those given above and  age, this is widely recognised eg.  age: 7 to 11 years old     11 to 16 years old   an eleven year old could select either category.  Be particularly careful of descriptive categories, like those illustrated by Gilbert (1981 p10),  where the categories may not be mutually exclusive (compare 2 and 6, and 3 and 5):

.The major reason for mothers with children choosing to go out to work

(1) For the money
(2) To get me out of the house
(3) For the company
(4) To use my training
(5) Because I enjoy working
(6) Because I hate housework
(7) Other
The underlying problem here is asking for the 'major reason', possible solutions, in addition to redefining the categories, would be to ask the respondent to put the categories in rank order OR  rate each statement on, say, a 1 to 10 scale.
e)
 Multiple Choice Questions must be Exhaustive.  This is normally done by including an 'other' category and asking for it to be clarified.                                                       Top of the Page

III.       Types of Questions

a)

 

 

 Multiple choice identification questions:  Remember, if you intend to compare the responses of different groups of people, then it is essential to include questions which will identify the groups  i.e. consider:
i) who is going to be questioned:-  their age, linguistic ability etc.
ii) how the questionnaire is to be analysed:-  If you intend to compare the responses of different groups of people, then it is necessary to include questions which will identify them.
b)

 

 

 Multiple choice information questions.   Consider which type of question will  give you the information required.  Many questions which appear to be open-ended can be easily  converted into a multiple choice format.  Questions such as  
                                        ‘How many students attend your institution?’
appear to be expecting a detailed response like ‘1,563’; if all that  is required is information  on whether the institution is large or small it is better to offer a range of classifications.
c)

 

 

 Multiple choice attitude statements/questions.    The value of these questions is that  they can quickly obtain information on a wide range of topics.   Essential questions in factor or cluster analysis is going to be considered.  These questions are generally described as having a Likert-style response.  Strongly recommend a six category rather than a five category response,   i.e.  from either:      Strongly Disagree to Strongly Agree,      or      NO! to YES!,   because both can be analysed with  two and three categories.  See tables below.

Table 1

Numerical Value

1

2

3

4

5

6
Attitude Statement

Strongly Disagree

Disagree

disagree?

agree?

Agree

Strongly Agree
Attitude Question

NO!

No

no?

yes?

Yes

YES!


Table 2

   

6 Categories

3 categories

 2 Categories

Question

 SD D d? a? A SA FD% ?% FA% D% A% Mean

N
Preparation for inspection was well planned and managedin school SMT
MM
Staff

0
1
1

0
1
5

0
0
7

1
13
23

5
16
17

3
5
2

0
6
11

11
36
55

89
58
35

0%
6%
24%

100%
94%
76%

5.2
4.6
4.0

9
36
55
The information from OFSTED (Framework for inspection) was helpful. SMT
MM
Staff

0
2
1

0
3
5

1
11
15

1
13
20

5
4
13

2
3
1

0
14
11

22
67
64

78
19
25

11%
44%
38%

89%
56%
62%

4.9
3.6
3.8

9
36
55
The information and support from the LEA was very positive SMT
MM
Staff

0
3
2

0
5
13

0
3
19

2
15
14

6
9
7

1
1
0

0
22
27

22
50
60

78
28
13

0%
31%
62%

100%
69%
38%

4.9
3.7
3.2

9
36
55
Preparation for inspection was well supported by INSET SMT
MM
Staff

1
6
2

0
4
15

0
11
17

2
9
11

4
6
10

2
0
0

11
28
31

22
56
51

67
17
18

11%
58%
62%

89%
42%
38%

4.6
3.1
3.2

9
36
55

 Table 2 has far more information in it than would normally be recommended, but it illustrates how the 6 category responses (lime coloured) can be reorganised into three and two categories (yellow and blue respectively).  The latter being far easier to describe.

  The statistical debate with respect to attitude questions is whether the numerical value is ordinal or interval.  Mathematicans would almost certainly state they were ordinal, but the linguistic logic is that they have interval aspects.   The range of responses can be considered as a continuous range of views, from Strongly Disagree to Strongly Agree, or NO! to YES!:  (see table 1 above and discussion ordinal or interval).    Indeed, in some questionnaires the interval aspect is emphasised by representing the continuum as a line, divided  into points of equal length.  Statistically, the present writer, has regularly  found that they can be meaningfully interpreted effectively using factor and cluster analyses, both of which theoretically require interval data.

d)

 

 

 

 

 Always ask at least one open-ended question: (socio-anthropologists would request far more).  Give the responders the opportunity to tell you what they think. Incidentally, apart from invariably providing new insights quotations from such questions also help make the final dissertation/thesis much more interesting to read.

However, whilst  it can be productive in an interview to initially ask very general questions, the danger in a questionnaire is that such questions will produce an equally general responses:

What do you think of the XXXXX? Responses such as ‘not much’, ‘OK’ or even ‘its fantastic’  might let the course organiser know if there are any problems, but they won’t provide the information needed to improve the course.
e)


 

 

 Other types questions e.g.
Osgood's semantic differential - respondent is presented with a range of alternative words  and asked to circle the one(s) which 'best' answer the question.    e.g.   A teacher is like a:   

        parent,     policeman,    friend,     boss,     helper,    colleague,    dictator,    sergeant-major

In the alternatives offered in this question there appear to be three groups of answers:
                policeman,     boss,     dictator,    sergeant-major
                friend,     helper,    colleague,                                  and,
                parent,    
hence, at least three codes (spreadsheet columns) need to be used:    
                AUTHORITy,      FRIEND,       &     PARENT

Order questions - rank order the following statements, giving 1 to the most important.  If the statement is irrelevant leave the order box blank, if two statements are equally important give them the same order number...          

                                                                                                                                      Top of the Page

IV.     Strengths of Questionnaires:

They SAVE TIME by allowing:

a) a large number of individuals to be questioned   without excessive use of the evaluator’s time.
b) 

 

 each individual to quickly provide detailed information of their views.  Both questionnaires and interviews provide the individual feedback that is inevitably lost at staff meetings, where information from particular individuals can be non-existent
c) rapid analysis and feedback.

   

They ENCOURAGE OBJECTIVITY:

d)

 

 Questionnaires are the only data collecting method where total confidentiality, even from the researcher, can be guaranteed. - this means that individuals can give their views strongly without affecting relationships - or career prospects!    
e) All questions are presented in an identical fashion.
Researcher bias can be checked in advance through procedures such as:
i) showing the questionnaire to others                   and,
ii) by piloting it.
f) The researcher is encouraged to plan each question carefully. With such in-depth planning researchers are more likely to consider how the information gained is to be analysed.

                                                                                                                                    Top of the Page

V.    Weaknesses of Questionnaires:

a)

 

 

 

 

 Questionnaires are inflexible:
        It is virtually impossible to ‘interrogate’ the respondent.    Difficult to clarify some question responses. The researcher sometimes has a problem   in analysing an open-ended questionnaire response: Single words can be particularly difficult to classify: sarcastically could "FASCINATING!"   mean the converse?.   Questions have to be very carefully worded - As a regular user of questionnaires I have found that,  where it is possible for a question to be misunderstood, it will be! (In an interview such errors  can usually be clarified.)   Questionnaires suffer from user-unfriendliness. Most individuals seem to find questionnaires  boring, they would much rather answer such questions face to face. Questionnaires cannot  smile, answer new questions or apologise!   Hence:
i) Make the questionnaire appear to be relevant to the responder. If the questionnaire is posted then it must be ‘sold’ by a short covering letter.
ii) Carefully word each question. Don’t upset the recipient!
b)

 

 

 

 

 

 

 Incorrect Information:
i)

 


 Mis-truths Is the responder putting themselves in a particl;uarly favourable light?   If the respondent feels that this is to their  advantage this would occur in both interviews and questionnaires. With interviews the interviewer may  identify the lie.  Conversely if the questionnaire is fully confidential, without the researcher knowing the respondent, then there is little to gain from lieing and, even if odd individuals do lie, it is much less important  in questionnaires than interviews because their overall effect is reduced by the larger number of individuals questioned.     
ii)

 

 Misunderstandings Those are more likely to be found in poorly designed questionnaires.  Can be identified and reduced through improved techniques: Show draft questionnaire  to supervisor/colleagues etc.         Pilot it.
iii)

 

 Poor Memory If the questionnaire/interview is carried out some time after the event it is highly likely that some things will be forgotten and others will be selectively remembered.   Carry out the questioning as soon as possible.

                                                                                                                                          Top of the Page

VI.    Response Rate    Acceptable response rate = 60%

The literature is over-critical with respect to the response rate to questionnaires because they fail to state that the questionnaires they are referring to are of a survey nature.  The response rate to a clearly defined group, particularly one known to the researcher, is commonly high:  how many children in a class, colleagues in a school are likely to refuse to fill in a questionnaire for a teacher or colleague?   Even with some surveys, the response rate can be very high:

Postal Questionnaires Zimbabwe 1993/94
M.Ed. student who worked for Government:

Schools where PE taught

Population

Sample

Response

Qu 1a: Heads of Schools where PE taught
Qu 1b: Qualif. PE Teachers where PE taught

28
28

24
24

100.0%
100.0%

Schools where PE not taught

     

Qu 2a: Headteachers with Qualif. PE Teachers
Qu 2b: Qualified PE Teachers not teaching PE
Qu 3: Heads of Schools where no Qualif. PE staff

150
150
546

15
15
55

93.3%
100.0%
87.2%

Total:

  

133

94.3%

Of 133 Questionnaires distributed 125 were returned. A response rate of 94%
A Ministry of Education and Culture covering letter explaining the purpose of the survey and asked for the completion and return of the questionnaires not later than 30 June 1994 accompanied the questionnaires. In each package of the questionnaires had an ‘On Government Service’ officially stamped envelope addressed to the officer appointed to assist with the administration of the questionnaires. The returned questionnaires were received by the researcher in mid-July 1994.

Comments: Confidentiality guaranteed  and,   majority of questions multiple choice.

A low response rate is likely when:

a)

 

 

 

 

 

 Too many open-ended questions are asked and the questionnaire is long.  Open-ended questions are particularly appealing to recipients because they demonstrate that any response is acceptable.    However, too many of them and the likelihood will be that the  questionnaire is put on one side to be returned to later or never!    Hence, when possible, use a majority of   Closed-ended questions.
From the writer’s experience a high response rate is likely if:
i) the researcher has ‘power’
iui) the responders are ‘captive’ and unable to avoid it
iii) the responders are friendly to the researcher,
iv) the responders are passionate about the issue
b)


 Unless a questionnaire is filled in immediately it is VERY EASY TO MISPLACE.     If the response rate is low, try to discover why.  Study the identification questions to see if the  sample appears to be a fair one.  Send out reminder letters.  IF the response remains low, use  the data BUT state clearly the problem and be very wary of making any generalisations.

                                                                                                                                        Top of the Page

VII.  Organising Questionnaire Data for Analysis

The writer strongly recommends the use of  Microsoft EXCEL  for entering and initially analysing questionnaire responses.  The major reasons for this are that the database:
 

 

 

 

 

 

 

 

 

 

 

 

 

 

N.B. the descriptions ‘record’ and ‘field’ do not appear on the screen

Fields can be altered in size (use mouse or Alt F & W). In the example below, field A has been made slightly smaller, fields B to K reduced and field L increased in size. Some spreadsheets/databases, e.g. EXCEL, can also make the records narrower or deeper.

FIELDS

RECORDS

A

B

C

D

E

F

G

H

I

J

K

L

 M

1

                          

2

                          

3

                          
4                          


With spreadsheets like EXCEL the Fields cannot be labelled, in spreadsheets the first row is used to identify the columns AND the ‘Split’ command (ALT W and S) to create the following impression:

FIELDS

RECORDS

 A B C D E F G H I J K

L

1

ID

Class

Sex

Ethnic

A1

A2

A3

A4

A5

A6

A7

Preference

2

                        

3

                        

4

                        

 Because of the possibility of using SPSS, it is worth also doing this with EXCEL .

Finally, in case you ever intend to analyse your data using SPSS,   ensure that the field names in spreadsheets  begin with a LETTER, have a maximum of 8 characters and don't include hyphens, full stops etc.                      Top of the Page

VIII.     Analysing Questions

Initial analysis of closed ended questions (multiple choice/attitude questions/statements) can be done using specially prepared spreadsheets see QuestAnal-6.xls  and  Questanal-5.XLS both of which  provide a  rapid detailed overview, Dataanal.xls can be used but is more cumbersome,  all have been prepared by Graham Tall and can be found on the following web page:  Chi test .  For detailed analyses at MPhil/PhD level  SPSS will often be necessary.   See   Analysis Closed Ended Data

Analysing open ended questions can be done using Microsoft WORKS database or, NUDIST.   See:
                                                                            Open Ended Analysis                             Top of the Page

References:
Ellis, Lee (1994)  Research Methods in the Social Sciences.  Oxford: Brown & Benchmark. 
                           Discusses Questionnaire construction guidelines.                           
Hopkins, David  (1993) A teacher's Guide to Classroom Research   Buckingham: Open University 
                           Short section on questionnaires and young children.
Keeves, John P. (Ed.) (1988) Educational Research, Methodology and Measurement.
                            An International Handbook. Pergamon & Kegan Paul
                            (This book is very expensive to buy, but is worth dipping into.)
Fink, Arlene (1995) The Survey Handbook  London:Sage  Plus other books: 'How to Design
                             Surveys' and 'How to ask Survey Questions' in the series
Gilbert, G.N. (1981)  Modelling Society:  An Introduction to Loglinear Analysis for Social
                             Researchers.   London:Allen&Unwin
Tall, Graham  (1988)  Why Use a Questionnaire? - The Questionnaire as an Evaluation Tool in Schools.
                              Pastoral Care 4, 6, 33-36  Use of questionnaire in article below.
Tall, G. & Langtree, G. (1988) Pastoral Education, Evaluation and GRIST.
                              Pastoral Care 2, 6, 17-22

Home Page    Research Introduction   Quantitative Advice    Index    Statistical Tests   
Research and Statistics Courses