Composing a Written Questionnaire

This document was originally written for private purposes. Along the line it got linked from a number of sites. I guess that is what the internet is for. It emphasizes printed questionnaires which are used less and less often in professional research, but that is because of the original context of these notes (viz. surveys to be conducted by non-professionals on behalf of religious congregations). By no means do I claim this as the last word in survey methodology. You are welcome to take or leave my advice.

Dr. Paul Riedesel
Action Marketing Research (Minneapolis)

A questionnaire is a very special kind of document. While it might seem like any reasonable person could type out a bunch of questions, creating a good, user-friendly instrument is very much a craft. You cannot separate the composition of questions themselves from the physical formatting (word processing and/or on-screen layout). We recommend that you compose the document in questionnaire format from the very first draft.

The most user-friendly form is a "booklet" of 4 or 8 pages. That literally means printing on both sides of 11x17 paper and folding in the middle. Any commercial printer can do this, as can better office copiers. Single sheets in the middle (say for a 6-page survey) have a way of getting lost.

There is no end to the rules-of-thumb and good habits used by professionals. Here are some of the major ones.

Use questions with fixed answer categories as much as possible. Insofar as your main goal is to provide summary tables, you will want to avoid the necessity of analyzing and coding open-ended remarks. On a printed questionnaire, the easiest thing for people to do is to check boxes. This is better practice than having them circle numbers. You need to figure out how to insert nice looking check boxes in whatever word processor you are using.

The answer categories must be exhaustive. That is, they need to cover all imaginable possibilities. That does not mean you have to offer dozens of choices, but at least leave an "other" option.

If you only want one answer, say that explicitly. If several answers may be checked, say that explicitly. Instructions, by the way, should be in a different font than the questions and answers. Italics or bold will do it. (Good online survey software lets you control this).

Pre-number the answer categories. This aids greatly in correctly tabulating the results. The numbers should be in a small, subscripted font.

It is customary and useful to number the questions. This is essential if there are to be any "skips" in the questionnaire. It will also make it easier for you when you are tabulating the answers.

Never split the answer categories across two pages!

Place the easier-to-answer questions at the beginning of the survey. Ones that require more effort to answer, or that are more sensitive in nature (sexuality, or worst of all, money), should appear later.

Questions that are related to each other should be grouped together as much as possible. Subject headings and brief introductions to the various sections can help.

Note that these categories are exhaustive and exclusive, that multiple answers are explicitly invited,
and that the check boxes are pre-numbered.

Here is a really bad question! Many of us could check several boxes, but is that what is wanted?

Here is a much better version.

Certain data you wish to report may describe an entire household, rather than individual members. You must take care to ask questions in such a way that you are accurately describing households.

Here is a tidy solution.

Question Skips
It is sometimes necessary to ask that only certain people answer certain questions. All others would be asked to skip ahead to another question. While it is almost impossible to ensure that everyone will follow the instructions, leave nothing to chance. State explicitly what the "rules" are, and use graphic devices to emphasize what they are to do. (Good online survey software lets you control this).

 The skip instruction is bolded, and the contingent questions are numbered differently (4a, 4b) and indented.
Note that 4a really asks two questions at once--that's OK.

Attitude measurement is a specialized field unto itself. Even the experts have yet to agree on the best methods. You may wish to include a section on attitudes, but make sure you know how you are going to present the results and how this data will either 1) help you with your decision making, or 2) help a prospective minister better understand you.

To begin with, be crystal clear what the underlying scale is that you are measuring. There are other dimensions, but you will probably be best off asking about the degree of agreement-disagreement with a series of statements. Again, there are many rules for good research practice. Among them:

You can do worse than offering the five familiar answer categories shown below. Some statisticians would be squeamish, but many other researchers are quite comfortable assigning numeric values of "5" for strongly agree and "1" for strongly disagree. My example uses check boxes, though some would have survey participants circle numbers instead. What you do not want to do is ask people to write in numbers (1-5) or codes like "A" or "SD."