(Mostly) Plain language research results

Final Comments

Near the end of a web survey respondents are often asked whether they have additional comments. Such final comments are usually ignored, partially because open-ended questions are more challenging to analyze. Our research mainly focused on final comments in the LISS panel and Dutch immigrant panels, two well-known long running web survey panels in the Netherlands. Only 5.7% of the surveys contained final comments in the immigrant panel as compared to 3.6% in the LISS panel. In both panels there are slightly more neutral than negative comments, and very few positive comments. Members of the immigrant panel were much more likely to complain about unclear questions as compared to those of the (regular) LISS panel. Researchers might want to consider additional pretesting of questions when fielding a questionnaire in the immigrant panel.

We next explored whether final comments are associated with next-wave attrition in survey panels. In the Immigrant panel we found shorter final comments (<30 words) are associated with increased next-wave attrition, and longer final comments (>55 words) with decreased next-wave attrition relative to making no comment. Comments about unclear survey questions quadruple the odds of attrition and “other” (uncategorized) negative comments almost double the odds of attrition; however this finding was not confirmed in the LISS panel. To our knowledge this was the first project to systematically analyze final comments.

[Image of straightlining]

Straight lining

As part of this work we noticed an unusual increase in straightlining over time. Straightlining refers to giving the same answer in a series of questions arranged on a grid. This is undesirable as it may indicate low data quality. We found that for the first three years of respondents’ membership in the LISS panel straightlining increased. The findings were confirmed in the immigrant panel. Interestingly, this result that we stumbled upon is generating the most interest (as measured by google scholar citations).

The figure shows the percentage of straight lining in the LISS panel over time. The left panel measures time as months past since joining the panel, the right panel as the number of previously answered surveys.

References

Return to Home Page
Remove navigation bar on the left