Survey Construction Techniques

Piloting Survey Items

Piloting a survey is a critical step if you want to be sure that the survey will be understandable and will produce valid data. There are a number of ways of piloting a survey, and most researchers will use some combination of these approaches to make sure that their survey is ready for data collection.

Have an Expert Read the Survey

Even when you know material inside out, you may miss something subtle in a survey instrument. Asking one or more experts on the topics that you are investigating will often identify these omissions. There are many types of experts. Some may be knowledgeable about the content of the survey. Some may be knowledgeable about constructing and analyzing surveys. Some may be excellent at writing, and can rephrase items to be more clearly understood. If you have the time and resources, it is good to have experts in all of these areas read your material.

Have a Few Non-Experts Read the Survey

Experts provide an important opinion, but often experts have blind spots. Ask a few people who know little or nothing about the area that you are surveying to read the survey and identify what things they found unclear. You will be surprised how frequently these non-experts will find things that the experts missed.

Give it to a Small Pilot Sample

Giving your survey to a small pilot sample can be helpful in finding the rough spots. During the survey process, issues that are unclear will often trigger questions or comments that are written on the questionnaire next to the participant's answer.

Give it to Small Groups Whose Performance Can be Predicted

Often you may want to survey issues that should reliably differentiate certain groups. For example, if you are interested in how one's level of knowledge of the stock market affects certain attitudes, you can pilot the items that tap knowledge of the stock market by giving them to a small group of brokers and a group of unselected participants. If the brokers do not do substantially better on each item than your unselected sample, the item may have to be reworked.

This approach can also be used by giving a skills test to a group of experts only. In this way, you can verify that the experts agree with you on what the correct answers are. If they do not, you may have written the item in a confusing way, misunderstood the concept yourself, or may have simply incorrectly keyed the item. Regardless, the item will have to be changed before the survey is ready for general release.