Moving to Survey Self-Completion: Mode Choice and Sensitive Questions

Insights from the ESS Ireland Self-Completion Pilot

Introduction

Research and evidence used to understand changes in our societies, both over time and comparatively, relies heavily on survey data. The European Social Survey (ESS) has long been recognised as a pioneer in high-quality cross-national survey research. Since its inception, the ESS has relied on rigorous face-to-face interviewing to ensure comparability across countries and over time. However, like many large-scale surveys, the ESS has been challenged by rising fieldwork costs, declining response rates, and changing communication habits among respondents (Cernat & Revilla, 2021; DeLeeuw, 2018; Wolf et al., 2021). To address these challenges, the ESS is transitioning to self-completion modes of data collection, first through mixed-mode designs in Round 12 (2025/26), and moving to full self-completion in Round 13 (2027/28) (Fitzgerald, 2025). This represents a major methodological shift and an important innovation in comparative survey research.

As part of this transition, during summer 2024, Ireland participated in a Round 12 self-completion pilot study, which offered respondents a choice between completing the ESS questionnaire on paper or online. This pilot provides valuable evidence for understanding both the opportunities and challenges of adopting self-completion modes (Li and Collins, 2025). Previous analyses of the Irish pilot have examined response rates, sample composition, and respondents’ preferences for web versus paper (Collins and Li, 2025). In this paper, we extend this line of inquiry by focusing on how survey mode shapes responses to sensitive questions.

Our core research questions are:

1. Are responses to sensitive questions different in self-completion compared to face-to-face surveys?
2. Within self-completion, are there systematic differences between web and paper?
3. Are differences driven by mode effects (greater freedom, anonymity) or cohort effects (differences in respondent composition)?

Theory

Survey questions are considered sensitive when they lead to higher nonresponse or measurement error, often due to their content. For example, questions exploring drug use, sexual behaviour, income, or voting (Tourangeau and Yan, 2007). Tourangeau, Rips, and Rasinski (2000) identify three distinct meanings of sensitivity.

First, some questions are seen as intrusive, touching on taboo or private topics like religion, which may offend respondents regardless of their actual answers. These may be considered an invasion of privacy and are sensitive due to their content rather than the context in which they are asked.

Second, sensitivity can arise from social desirability, where questions elicit responses that may be socially undesirable, such as admitting to not voting. This form of sensitivity depends on social norms and the respondent’s behaviour. The literature diverges on whether socially desirable responding is a stable personality trait (Crowne & Marlowe, 1964) or a situational strategy (Sudman & Bradburn, 1974). Paulhus (2002) bridges both views, distinguishing between consistent response styles and temporary response sets.

Finally, the threat of disclosure refers to respondents’ fear that their answers might be revealed to third parties, leading to potential consequences. Even with confidentiality assurances, respondents may not be willing to share their personal information, as well as report behaviours they are ashamed of or perceive as socially undesirable (Singer, van Hoewyk & Neugebauer, 2003; Singer, von Thurn, & Miller, 1995).

Drawing on the ESS round 12 self-completion pilot study, we identified three groups of questions that correspond to these three dimensions of sensitivity in the Irish context. First, the intrusive questions are likely to include those related to attitudes towards the LGBTQ+ individuals. While Ireland has made significant progress in LGBTQ+ rights, including the legalisation of same-sex marriage, some respondents may still view such topics as deeply personal or culturally sensitive, particularly in more conservative communities. Second, we expect that social desirability concerns are relevant for questions about attitudes toward migrants. Ireland has positioned itself as a welcoming society, and public discourse often emphasises inclusivity and diversity. As a result, respondents may feel pressure to express pro-migrant views, even if their private attitudes differ. Finally, the threat of disclosure is especially salient in questions about income. Respondents with high incomes may be reluctant to disclose them due to privacy concerns or fear of judgment, while those with low incomes may feel embarrassed or ashamed. Despite assurances of confidentiality, these concerns can lead to underreporting or misreporting, particularly in interviewer-administered surveys.

We therefore expect the mode of data collection to influence how respondents answer sensitive questions. For intrusive questions, respondents may provide more socially acceptable answers in face-to-face interviews but be more candid in self-completion formats. For socially desirable questions, self-completion is also expected to reduce pressures to conform to social norms. As to threat of disclosure questions, we anticipate fewer missing responses in self-completion formats than in face-to-face interviews, as the absence of an interviewer may reduce concerns about privacy and judgment. Within self-completion, web surveys may further enhance perceived anonymity compared to paper.

Data and Methods

This study draws on data from two ESS rounds: the Irish pilot study for Round 12, which was conducted in a self-completion format, and the previous round (Round 11), which relied on face-to-face interviews. The pilot in Ireland offered respondents the choice between completing the questionnaire on paper or online. Thus, we can compare the data between three groups: face-to-face (ESS11), paper self-completion, and web self-completion.

Our analysis focuses on the three types of sensitive questions identified in the literature. For intrusive items, we use questions on attitudes toward LGBTQ+ individuals. For socially desirable items, we include attitudes toward migrants. For confidential items, we examine self-reported total household income, as well as feeling about household income.

The analytical strategy is twofold. First, we provide descriptive statistics comparing mean responses and nonresponse rates across modes, highlighting how patterns differ between face-to-face, web, and paper respondents. Second, using the ESS 12 self-completion data, we estimate regression models to test whether observed differences between web and paper mode can be explained by cohort effects (i.e., the demographic composition of respondents across modes) or whether they reflect genuine mode effects. The models include controls for gender, age, education, domicile, and income, which are key predictors of both survey participation and attitudes toward sensitive topics.

Results

The descriptive results provide a first indication that survey mode matters for responses to sensitive questions (see Table 1). For intrusive questions on LGBTQ+ individuals, mean responses were stable across modes, with a slightly higher mean for “Gays and lesbians free to live life as they wish” in face-to-face and for “Ashamed if close family member gay or lesbian” in the web mode. This suggests that responses are not strongly influenced by the mode of data collection. However, the rate of item nonresponse was lower in the self-completion modes (0-0.7%) than in face-to-face (1-4%), indicating that respondents may have felt more comfortable engaging with these questions when no interviewer was present.

The results are different for socially desirable questions. In the case of attitudes toward migrants, respondents in the paper mode expressed more conservative views compared to both web and face-to-face respondents. This may indicate that in the presence of an interviewer, the respondents want to show more positive attitudes towards migrants, while the anonymity of the paper mode allows for more truthful answers. At the same time, respondents in the web mode may express more positive attitudes due to their age or education.

For confidential questions such as household income, the mode of data collection clearly shaped response patterns. As expected, the proportion of missing data was highest in the face-to-face survey (23.6%), lower in the paper mode (18.8%), and lowest in the web mode (17.2%). This pattern suggests that the absence of an interviewer reduces concerns about privacy and disclosure, particularly in online settings where respondents may feel the highest degree of anonymity. However, this does not apply to a question about the feeling about household’s income.

Table 1: Descriptive Statistics for ESS Sensitive Questions by Mode

Notes: ESS Round 11 fieldwork took place between June 2023 and January 2024; the ESS 12 pilot fieldwork took place between June 10 to August 2, 2024. Both ESS 11 and ESS 12p results are based on the unweighted sample data. For ESS 11 n=2,017, for ESS 12p n=631. NA=no answer/nonresponse.

Table 2 presents the results of regression models with one of the sensitive questions from each group as the dependent variable. Regression analyses confirm and refine the descriptive findings. For the intrusive question, differences across modes disappear once demographic controls are added, pointing to a cohort effect. Respondents were asked if they would feel ashamed if a close family member were gay or lesbian. The response options range from 1 = “strongly agree” (would feel ashamed) to 5 = “strongly disagree” (wouldn’t feel ashamed). The coefficient estimates suggest that females, as well as younger, more educated, and respondents with higher household income tend to hold more liberal attitudes regardless of mode.

For the socially desirable question, however, both mode and cohort effects play a role. Respondents were asked if immigrants make their country a worse or better place to live with 0 indicating “worse place to live” and 10 a “better place to live”. Regression results show that respondents in the paper mode express more negative attitudes toward migrants even after controls are introduced (p<0.05). At the same time, older, more educated, and those in urban areas and with higher income express more positive attitudes towards migrants.

Finally, for the confidential item, decile of total household’s net income, both mode and cohort effects are evident. Respondents in the paper mode reported lower income than those in the web mode. While females and older respondents reported lower income, highly educated respondents reported higher income.

Taken together, the results suggest that self-completion formats, and particularly web surveys, yield more candid responses on sensitive questions. At the same time, part of the observed differences reflect cohort effects, since the demographic composition of web and paper respondents differs in systematic ways. These differences are explained further in Li and Collins (2025).

Discussion

The findings of this study shed light on how survey mode shapes responses to sensitive questions, with important implications for the transition from interviewer-administered to self-completion formats for the ESS and other surveys. Across the three dimensions of sensitivity (intrusive, socially desirable, and confidential), we find that self-completion tends to reduce barriers to candor and lower nonresponse, though the extent of the effect varies by topic and by whether the self-completion was online or on paper.

For intrusive questions concerning LGBTQ+ attitudes, differences between modes were modest once sociodemographic composition was accounted for. This suggests that responses on such topics are largely driven by cohort characteristics, such as age, education, and income, rather than by the survey mode itself. However, the lower rate of item nonresponse in self-completion formats indicates that the absence of an interviewer may encourage respondents to engage with questions that they might otherwise skip, even if their substantive answers do not change.

In contrast, responses to socially desirable items, particularly attitudes toward migrants, show clear evidence of mode effects. Paper respondents consistently expressed more negative views of migrants compared to web or face-to-face respondents, even after controlling for demographics. This finding points to the role of perceived anonymity: when respondents feel less observed, they may answer more freely, reducing the tendency to give normatively acceptable answers.

Confidential items such as household’s income showed the most striking difference by mode. The proportion of missing data was substantially lower in self-completion compared to face-to-face, with the web mode producing the lowest nonresponse. This pattern aligns with expectations that concerns about privacy and disclosure are mitigated in settings where no interviewer is present, and especially when the survey is completed online. At the same time, regression models revealed that demographic composition also plays a role, as web respondents differ systematically from paper respondents in their socioeconomic profile. Thus, both mode effects and cohort effects contribute to the observed differences.

Taken together, these results suggest that self-completion modes can enhance data quality on sensitive questions by fostering more truthful answers and reducing nonresponse. Yet they also underscore the need to disentangle true mode effects from compositional differences between respondents who opt for paper versus web formats.

Conclusion

The Irish pilot study of ESS Round 12 provides important lessons for the ongoing transition of the ESS to self-completion. Our analysis shows that responses to sensitive questions differ across survey modes, with self-completion, and web in particular, yielding more candid answers and lower levels of nonresponse compared to traditional face-to-face interviews. For intrusive items, differences are largely attributable to respondent composition, whereas for socially desirable and confidential items, genuine mode effects emerge alongside cohort effects.

These findings have two key implications. First, they provide reassurance that moving to self-completion will not compromise, and may even improve, data quality for sensitive questions. Respondents appear more willing to disclose potentially uncomfortable or private information in settings where anonymity is greater. Second, the results highlight the importance of monitoring demographic differences between paper and web respondents. While both formats reduce interviewer effects, their respondent profiles are not identical, and these differences can influence observed outcomes.

As the ESS transitions fully to self-completion in Round 13, careful attention to mode-specific patterns of response and nonresponse will be essential to ensure comparability across waves and countries. Further methodological work is needed to disentangle mode and cohort effects more systematically. Nevertheless, the evidence from Ireland suggests that the move to self-completion offers an opportunity to reduce measurement error on sensitive topics, thereby strengthening the ESS’s capacity to capture attitudes and behaviours across Europe.

References

Cernat, A., & Revilla, M. (2021). Moving from Face-to-Face to a Web Panel: Impacts on Measurement Quality. Journal of Survey Statistics and Methodology, 9(4), 745–763. https://doi.org/10.1093/jssam/smaa007

Crowne, D., & Marlowe, D. (1964). The approval motive. New York: John Wiley.

Collins, M.L., & Li, O. (2025). Moving to Self-Completion: Challenges and Opportunities for Survey Data Collection in Ireland. Available at: https://publicpolicy.ie/papers/moving-to-self-completion-challenges-and-opportunities-for-survey-data-collection-in-ireland/

de Leeuw, E. D. (2018). Mixed-Mode: Past, Present, and Future. Survey Research Methods, Vol 12, 75-89 Pages. https://doi.org/10.18148/SRM/2018.V12I2.7402

Fitzgerald, R. (2025, April 4). The move to self-completion interviewing on Europe’s cross-national general social survey – balancing innovation and continuity. International Methodological Conference “The Future of Survey Research,” Warsaw, Poland.

Li, O., & Collins, M.L. (2025, April 8). Implementing the ESS Self-Completion Mode in Ireland: Results from the Pilot Study. 2025 International Workshop on Comparative Survey Design and Implementation, Skopje, North Macedonia.

Li, O., & Collins, M.L. (2025). Moving to Survey Self-Completion: Exploring Mode Preferences. Insights from the ESS Ireland Self-Completion Pilot. Available at: https://publicpolicy.ie/papers/moving-to-survey-self-completion-exploring-mode-preferences/

Paulhus, D. L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braun, D. N. Jackson, & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 49 69). Mahwah, NJ: Erlbaum.

Singer, E., Van Hoewyk, J., & Neugebauer, R. (2003). Attitudes and behavior: The impact of privacy and confidentiality concerns on participation in the 2000 Census. Public Opinion Quarterly, 65, 368 –384.

Singer, E., von Thurn, D., & Miller, E. (1995). Confidentiality assurances and response: A quantitative review of the experimental literature. Public Opinion Quarterly, 59, 66 –77.

Sudman, S., & Bradburn, N. (1974). Response effects in surveys: A review and synthesis. Chicago: Aldine.

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge, England: Cambridge University Press.

Tourangeau, R., & Yan, T. (2007). Sensitive questions in surveys. Psychological Bulletin, 133(5), 859–883. https://doi.org/10.1037/0033-2909.133.5.859

Wolf, C., Christmann, P., Gummer, T., Schnaudt, C., & Verhoeven, S. (2021). Conducting General Social Surveys as Self-Administered Mixed-Mode Surveys. Public Opinion Quarterly, 85(2), 623–648. https://doi.org/10.1093/poq/nfab039