List of Contributors |
|
xvii | |
Preface |
|
xxiii | |
Part I Assessing the Current Methodology for Questionnaire Design, Development, Testing, and Evaluation |
|
1 | (116) |
|
1 Questionnaire Design, Development, Evaluation, and Testing: Where Are We, and Where Are We Headed? |
|
|
3 | (22) |
|
|
1.1 Current State of the Art and Science of QDET |
|
|
3 | (8) |
|
1.2 Relevance of QDET in the Evolving World of Surveys |
|
|
11 | (5) |
|
1.3 Looking Ahead: Further Developments in QDET |
|
|
16 | (3) |
|
|
19 | (1) |
|
|
20 | (5) |
|
2 Asking the Right Questions in the Right Way: Six Needed Changes in Questionnaire Evaluation and Testing Methods |
|
|
25 | (22) |
|
|
2.1 Personal Experiences with Cognitive Interviews and Focus Groups |
|
|
25 | (4) |
|
2.2 My 2002 Experience at QDET |
|
|
29 | (4) |
|
2.3 Six Changes in Survey Research that Require New Perspectives on Questionnaire Evaluation and Testing |
|
|
33 | (9) |
|
|
42 | (1) |
|
|
43 | (4) |
|
3 A Framework for Making Decisions About Question Evaluation Methods |
|
|
47 | (28) |
|
|
|
|
|
|
47 | (1) |
|
|
48 | (3) |
|
|
51 | (4) |
|
|
55 | (4) |
|
3.5 Statistical Modeling for Data Quality |
|
|
59 | (4) |
|
3.6 Comparing Different Methods |
|
|
63 | (4) |
|
|
67 | (2) |
|
|
69 | (6) |
|
4 A Comparison of Five Question Evaluation Methods in Predicting the Validity of Respondent Answers to Factual Items |
|
|
75 | (16) |
|
|
|
|
75 | (1) |
|
|
76 | (3) |
|
|
79 | (5) |
|
|
84 | (1) |
|
|
85 | (6) |
|
5 Combining Multiple Question Evaluation Methods: What Does It Mean When the Data Appear to Conflict? |
|
|
91 | (26) |
|
|
|
|
91 | (1) |
|
5.2 Questionnaire Development Stages |
|
|
92 | (1) |
|
5.3 Selection of Case Studies |
|
|
93 | (2) |
|
5.4 Case Study 1: Conflicting Findings Between Focus Groups and Cognitive Interviews |
|
|
95 | (2) |
|
5.5 Case Study 2: Conflicting Findings Between Eye-Tracking, Respondent Debriefing Questions, and Interviewer Feedback |
|
|
97 | (3) |
|
5.6 Case Study 3: Complementary Findings Between Cognitive Interviews and Interviewer Feedback |
|
|
100 | (4) |
|
5.7 Case Study 4: Combining Qualitative and Quantitative Data to Assess Changes to a Travel Diary |
|
|
104 | (6) |
|
5.8 Framework of QT Methods |
|
|
110 | (1) |
|
5.9 Summary and Discussion |
|
|
110 | (4) |
|
|
114 | (3) |
Part II Question Characteristics, Response Burden, and Data Quality |
|
117 | (170) |
|
6 The Role of Question Characteristics in Designing and Evaluating Survey Questions |
|
|
119 | (34) |
|
|
|
|
|
|
119 | (1) |
|
6.2 Overview of Some of the Approaches Used to Conceptualize, Measure, and Code Question Characteristics |
|
|
120 | (7) |
|
6.3 Taxonomy of Question Characteristics |
|
|
127 | (5) |
|
|
132 | (9) |
|
|
141 | (6) |
|
|
147 | (1) |
|
|
148 | (5) |
|
7 Exploring the Associations Between Question Characteristics, Respondent Characteristics, Interviewer Performance Measures, and Survey Data Quality |
|
|
153 | (40) |
|
|
|
|
|
|
|
153 | (4) |
|
|
157 | (17) |
|
|
174 | (8) |
|
|
182 | (9) |
|
|
191 | (1) |
|
|
191 | (2) |
|
8 Response Burden: What Is It and What Predicts It? |
|
|
193 | (20) |
|
|
|
|
|
193 | (4) |
|
|
197 | (5) |
|
|
202 | (4) |
|
8.4 Conclusions and Discussion |
|
|
206 | (4) |
|
|
210 | (1) |
|
|
210 | (3) |
|
9 The Salience of Survey Burden and Its Effect on Response Behavior to Skip Questions: Experimental Results from Telephone and Web Surveys |
|
|
213 | (16) |
|
|
|
|
|
213 | (3) |
|
9.2 Study Designs and Methods |
|
|
216 | (3) |
|
9.3 Manipulating the Interleafed Format |
|
|
219 | (5) |
|
9.4 Discussion and Conclusion |
|
|
224 | (2) |
|
|
226 | (1) |
|
|
227 | (2) |
|
10 A Comparison of Fully Labeled and Top-Labeled Grid Question Formats |
|
|
229 | (30) |
|
|
|
|
229 | (7) |
|
|
236 | (7) |
|
|
243 | (10) |
|
10.4 Discussion and Conclusions |
|
|
253 | (1) |
|
|
254 | (1) |
|
|
255 | (4) |
|
11 The Effects of Task Difficulty and Conversational Cueing on Answer Formatting Problems in Surveys |
|
|
259 | (28) |
|
|
|
|
259 | (3) |
|
11.2 Factors Contributing to Respondents' Formatting Problems |
|
|
262 | (5) |
|
|
267 | (1) |
|
|
268 | (7) |
|
|
275 | (3) |
|
11.6 Discussion and Conclusion |
|
|
278 | (3) |
|
11.7 Further Expansion of the Current Study |
|
|
281 | (1) |
|
|
282 | (1) |
|
|
283 | (4) |
Part III Improving Questionnaires on the Web and Mobile Devices |
|
287 | (184) |
|
12 A Compendium of Web and Mobile Survey Pretesting Methods |
|
|
289 | (26) |
|
|
|
|
289 | (1) |
|
12.2 Review of Traditional Pretesting Methods |
|
|
290 | (4) |
|
12.3 Emerging Pretesting Methods |
|
|
294 | (14) |
|
|
308 | (7) |
|
13 Usability Testing Online Questionnaires: Experiences at the U.S. Census Bureau |
|
|
315 | (34) |
|
|
|
|
|
|
315 | (1) |
|
13.2 History of Usability Testing Self-Administered Surveys at the US Census Bureau |
|
|
316 | (1) |
|
13.3 Current Usability Practices at the Census Bureau |
|
|
317 | (3) |
|
13.4 Participants: "Real Users, Not User Stories" |
|
|
320 | (3) |
|
13.5 Building Usability Testing into the Development Life Cycle |
|
|
323 | (4) |
|
|
327 | (4) |
|
13.7 Measuring Efficiency |
|
|
331 | (4) |
|
13.8 Measuring Satisfaction |
|
|
335 | (2) |
|
13.9 Retrospective Probing and Debriefing |
|
|
337 | (2) |
|
13.10 Communicating Findings with the Development Team |
|
|
339 | (1) |
|
13.11 Assessing Whether Usability Test Recommendations Worked |
|
|
340 | (1) |
|
|
341 | (1) |
|
|
341 | (8) |
|
14 How Mobile Device Screen Size Affects Data Collected in Web Surveys |
|
|
349 | (26) |
|
|
|
|
349 | (1) |
|
|
350 | (2) |
|
14.3 Our Contribution and Hypotheses |
|
|
352 | (3) |
|
14.4 Data Collection and Method |
|
|
355 | (6) |
|
|
361 | (7) |
|
|
368 | (1) |
|
|
369 | (1) |
|
|
370 | (5) |
|
15 Optimizing Grid Questions for Smartphones: A Comparison of Optimized and Non-Optimized Designs and Effects on Data Quality on Different Devices |
|
|
375 | (28) |
|
|
|
|
375 | (1) |
|
15.2 The Need for Change in Questionnaire Design Practices |
|
|
376 | (2) |
|
15.3 Contribution and Research Questions |
|
|
378 | (2) |
|
15.4 Data Collection and Methodology |
|
|
380 | (6) |
|
|
386 | (6) |
|
|
392 | (5) |
|
|
397 | (1) |
|
|
397 | (6) |
|
16 Learning from Mouse Movements: Improving Questionnaires and Respondents' User Experience Through Passive Data Collection |
|
|
403 | (24) |
|
|
|
|
|
|
|
|
|
403 | (1) |
|
|
404 | (5) |
|
|
409 | (1) |
|
|
410 | (5) |
|
|
415 | (5) |
|
|
420 | (3) |
|
|
423 | (4) |
|
17 Using Targeted Embedded Probes to Quantify Cognitive Interviewing Findings |
|
|
427 | (24) |
|
|
|
427 | (4) |
|
17.2 The NCHS Research and Development Survey |
|
|
431 | (2) |
|
|
433 | (12) |
|
|
445 | (3) |
|
|
448 | (3) |
|
18 The Practice of Cognitive Interviewing Through Web Probing |
|
|
451 | (20) |
|
|
|
|
451 | (1) |
|
18.2 Methodological Issues in the Use of Web Probing for Pretesting |
|
|
452 | (1) |
|
18.3 Testing the Effect of Probe Placement |
|
|
453 | (2) |
|
18.4 Analyses of Responses to Web Probes |
|
|
455 | (4) |
|
18.5 Qualitative Analysis of Responses to Probes |
|
|
459 | (1) |
|
18.6 Qualitative Coding of Responses |
|
|
459 | (3) |
|
18.7 Current State of the Use of Web Probes |
|
|
462 | (3) |
|
|
465 | (1) |
|
18.9 Recommendations for the Application and Further Evaluation of Web Probes |
|
|
466 | (2) |
|
|
468 | (1) |
|
|
468 | (1) |
|
|
468 | (3) |
Part IV Cross-Cultural and Cross-National Questionnaire Design and Evaluation |
|
471 | (100) |
|
19 Optimizing Questionnaire Design in Cross-National and Cross-Cultural Surveys |
|
|
473 | (20) |
|
|
|
473 | (1) |
|
19.2 The Total Survey Error Paradigm and Comparison Error |
|
|
474 | (3) |
|
19.3 Cross-Cultural Survey Guidelines and Resources |
|
|
477 | (1) |
|
|
478 | (2) |
|
19.5 Developing Comparative Scales |
|
|
480 | (3) |
|
19.6 Focus Groups and Pretesting in Cross-National/Cultural Surveys |
|
|
483 | (1) |
|
19.7 Tools for Developing and Managing Cross-National Surveys |
|
|
484 | (1) |
|
19.8 Resources for Developing and Testing Cross-National Measures |
|
|
485 | (1) |
|
19.9 Pre- and Post-Harmonization |
|
|
486 | (2) |
|
|
488 | (1) |
|
|
488 | (5) |
|
20 A Model for Cross-National Questionnaire Design and Pretesting |
|
|
493 | (28) |
|
|
|
|
493 | (1) |
|
|
493 | (2) |
|
20.3 The European Social Survey |
|
|
495 | (1) |
|
20.4 ESS Questionnaire Design Approach |
|
|
496 | (1) |
|
20.5 Critique of the Seven-Stage Approach |
|
|
497 | (1) |
|
20.6 A Model for Cross-National Questionnaire Design and Pretesting |
|
|
497 | (4) |
|
20.7 Evaluation of the Model for Cross-National Questionnaire Design and Pretesting Using the Logical Framework Matrix (LFM) |
|
|
501 | (11) |
|
|
512 | (2) |
|
|
514 | (7) |
|
21 Cross-National Web Probing: An Overview of Its Methodology and Its Use in Cross-National Studies |
|
|
521 | (24) |
|
|
|
|
|
|
521 | (2) |
|
21.2 Cross-National Web Probing - Its Goal, Strengths, and Weaknesses |
|
|
523 | (3) |
|
21.3 Access to Respondents Across Countries: The Example of Online Access Panels and Probability-Based Panels |
|
|
526 | (1) |
|
21.4 Implementation of Standardized Probes |
|
|
527 | (5) |
|
21.5 Translation and Coding Answers to Cross-Cultural Probes |
|
|
532 | (1) |
|
|
533 | (3) |
|
21.7 Cross-National Web Probing and Its Application Throughout the Survey Life Cycle |
|
|
536 | (2) |
|
21.8 Conclusions and Outlook |
|
|
538 | (1) |
|
|
539 | (1) |
|
|
539 | (6) |
|
22 Measuring Disability Equality in Europe: Design and Development of the European Health and Social Integration Survey Questionnaire |
|
|
545 | (26) |
|
|
|
545 | (1) |
|
|
546 | (2) |
|
22.3 Questionnaire Design |
|
|
548 | (5) |
|
22.4 Questionnaire Development and Testing |
|
|
553 | (7) |
|
22.5 Survey Implementation |
|
|
560 | (3) |
|
|
563 | (3) |
|
|
566 | (1) |
|
|
567 | (1) |
|
|
567 | (4) |
Part V Extensions and Applications |
|
571 | (198) |
|
23 Regression-Based Response Probing for Assessing the Validity of Survey Questions |
|
|
573 | (20) |
|
|
|
|
|
573 | (1) |
|
23.2 Cognitive Methods for Assessing Question Validity |
|
|
574 | (3) |
|
23.3 Regression-Based Response Probing |
|
|
577 | (2) |
|
23.4 Example 1: Generalized Trust |
|
|
579 | (1) |
|
23.5 Example 2: Fear of Crime |
|
|
580 | (1) |
|
|
581 | (5) |
|
|
586 | (2) |
|
|
588 | (5) |
|
24 The Interplay Between Survey Research and Psychometrics, with a Focus on Validity Theory |
|
|
593 | (20) |
|
|
|
|
593 | (2) |
|
24.2 An Over-the-Shoulder Look Back at Validity Theory and Validation Practices with an Eye toward Describing Contemporary Validity Theories |
|
|
595 | (7) |
|
24.3 An Approach to Validity that Bridges Psychometrics and Survey Design |
|
|
602 | (4) |
|
|
606 | (2) |
|
|
608 | (5) |
|
25 Quality-Driven Approaches for Managing Complex Cognitive Testing Projects |
|
|
613 | (26) |
|
|
|
|
|
613 | (1) |
|
25.2 Characteristics of the Four Cognitive Testing Projects |
|
|
614 | (1) |
|
25.3 Identifying Detailed, Quality-Driven Management Approaches for Qualitative Research |
|
|
615 | (1) |
|
25.4 Identifying Principles for Developing Quality-Driven Management Approaches |
|
|
616 | (1) |
|
25.5 Applying the Concepts of Transparency and Consistency |
|
|
617 | (1) |
|
25.6 The 13 Quality-Driven Management Approaches |
|
|
618 | (14) |
|
25.7 Discussion and Conclusion |
|
|
632 | (2) |
|
|
634 | (5) |
|
26 Using Iterative, Small-Scale Quantitative and Qualitative Studies: A Review of 15 Years of Research to Redesign a Major US Federal Government Survey |
|
|
639 | (32) |
|
|
|
639 | (2) |
|
26.2 Measurement Issues in Health Insurance |
|
|
641 | (4) |
|
|
645 | (15) |
|
|
660 | (3) |
|
|
663 | (1) |
|
|
664 | (7) |
|
27 Contrasting Stylized Questions of Sleep with Diary Measures from the American Time Use Survey |
|
|
671 | (26) |
|
|
|
|
|
671 | (1) |
|
|
672 | (2) |
|
27.3 The Present Research |
|
|
674 | (4) |
|
27.4 Study 1: Behavior Coding 67S |
|
|
|
27.5 Study 2: Cognitive Interviews |
|
|
678 | (4) |
|
27.6 Study 3: Quantitative Study |
|
|
682 | (4) |
|
27.7 Study 4: Validation Study |
|
|
686 | (3) |
|
|
689 | (3) |
|
27.9 Implications and Future Directions |
|
|
692 | (1) |
|
|
692 | (5) |
|
28 Questionnaire Design Issues in Mail Surveys of All Adults in a Household |
|
|
697 | (26) |
|
|
|
|
|
|
697 | (1) |
|
|
698 | (1) |
|
28.3 The NCVS and Mail Survey Design Challenges |
|
|
699 | (5) |
|
28.4 Field Test Methods and Design |
|
|
704 | (2) |
|
|
706 | (2) |
|
|
708 | (8) |
|
|
716 | (1) |
|
|
716 | (3) |
|
|
719 | (1) |
|
|
720 | (3) |
|
29 Planning Your Multimethod Questionnaire Testing Bento Box: Complementary Methods for a Well-Balanced Test |
|
|
723 | (26) |
|
|
|
723 | (2) |
|
29.2 A Questionnaire Testing Bento Box |
|
|
725 | (8) |
|
29.3 Examples from the Census of Agriculture Questionnaire Testing Bento Box |
|
|
733 | (10) |
|
|
743 | (1) |
|
|
744 | (5) |
|
30 Flexible Pretesting on a Tight Budget: Using Multiple Dependent Methods to Maximize Effort-Return Trade-Offs |
|
|
749 | (20) |
|
|
|
|
|
|
|
|
|
|
|
749 | (3) |
|
30.2 Evolution of a Dependent Pretesting Approach for Gender Identity Measurement |
|
|
752 | (7) |
|
30.3 Analyzing and Synthesizing Results |
|
|
759 | (5) |
|
|
764 | (2) |
|
|
766 | (1) |
|
|
766 | (3) |
Index |
|
769 | |