Darragh Flannery, Kemmy Business School, Economics Department, University of Limerick
The impact of school closures in terms of learning loss or other education or labour market outcomes will most likely take years to assess accurately. However, a recent working paper by Zsuzsa Blaskó, Patricia da Costa and Sylke Schnepf of the European Commission’s Joint Research Centre (JRC) attempts to get a handle on this using TIMMS data1 from 2019 for a range of European countries, including Ireland. I have summarised a few points of note related to Ireland (Flannery, 2021), with the results highlighting a relatively low incidence of those without resources at home that may help home learning (having internet, your own room etc.) but a strong negative association between lacking these resources and performance in the TIMMS maths test.
However, this study uses data relating to 9/10 years for their analysis and given that school closures in Ireland have lasted longer for those in secondary school, it may be useful to examine this cohort. Furthermore, the JCR study restricts their analysis to maths performance at the mean, with some type of distributional analysis potentially useful. So using the most recent wave of the OECD’s PISA data (2018) for Ireland, I have tried to replicate their study somewhat for both maths and reading performance at the mean but also at the top and bottom ends of the performance distribution. PISA data is relatively well known (for good and bad reasons) and loads of info on it is found (Klieme, 2016; OECD, 2021; UCL, 2019).
A few things important to note about PISA; it sampled 15 year olds in Ireland in 2018 and as well as including PISA test scores in maths, reading and science it includes a host of socioeconomic information at the student level. For example, it captures data on parental education levels as well as whether students have certain resources at home for schoolwork purposes such as a computer or desk. It also has a large range of school level information as answered by the school principal. For our purposes here it is also relevant to note that in each PISA test subject, there is theoretically no minimum or maximum score; rather, the results are scaled to fit approximately normal distributions, with means for OECD countries around 500 score points and standard deviations around 100 score points.2
The overall aim in this exercise was simply to examine the association between having access to resources that may be helpful to learning from home and maths/reading achievement as indicated by PISA scores. To that end, table 1 below shows the proportion of students in the data that lacked some potentially important resources for home learning, as asked in the PISA survey.
Table 1 – Proportion without home learning resources
As we can see, within the PISA sample of over 5,000 students, very few lacked access to the internet at home, but a sizeable minority lacked either a computer to use for schoolwork at home or a desk to use for study at home.3 Table 2 below shows the mean and standard deviation for both maths and reading scores for groups with different access to home resources. Unsurprisingly, we see a stark difference with a raw gap of ~25 ‘points’ in maths performance and ~30 ‘points’ in reading with no computer at home particularly affected in a negative way; a 30 ‘point’ gap in reading scores is roughly 1/3 of a SD.
Table 2 – Raw PISA scores for mathematics and reading by selected home learning resources
To get a better idea of this association Table 3 below presents OLS estimates for maths and readings PISA scores regressed against a range of home learning resources. These estimates include controls for factors such as gender, having at least one parent with a degree level education and an ‘attitude to learning’ index captured in the PISA data to help proxy for motivation to learn. A lack of internet access at home is associated with a strong decrease in both math and reading performance and given the proportion of students without a desk or computer at home for study purposes outlined in Table 1, it is interesting to see the relatively large and significant negative association between not having these resources and maths/reading scores.
Table 3 – OLS estimates of PISA test performance for mathematics and reading
Table 4 presents the results of a distributional analysis with the same variables seen above used. Using an unconditional quantile regression approach4 for the 20th and 80th percentiles of maths and reading performance we see large differences in the coefficients for different home learning resources for the ‘low’ and ‘high’ performing students. Specifically, we see that those closer to the bottom end of the distribution affected more by not having resources such as a computer at home or a desk for school work at home.
Table 4 – Unconditional Quantile Regression Models of PISA Result
The usual caveats obviously apply here in terms of causality and in terms of how the PISA scores, as a low stakes test, actually relate to how students may perform in more high stakes educational outcomes.5 Furthermore, there were policy interventions such as the purchase of laptops for schools between the sample period and the closure of schools that may have helped mitigate any potential problems. However, the scale of the differences in performances in maths and reading for those with and without certain home learning resources does add to the evidence base of how school closures may have affected some groups of students in a particularly significant manner and underline the importance of measures to help address any gaps.
Avvisati, Francesco, and François Keslair. 2020. REPEST: Stata Module to Run Estimations with Weighted Replicate Samples and Plausible Values. Statistical Software Components. Boston College Department of Economics. https://ideas.repec.org/c/boc/bocode/s457918.html.
Blaskó, Zsuzsa. 2021. ‘Learning Loss and Educational Inequalities in Europe: Mapping the Potential Consequences of the COVID-19 Crisis’. IZA DP No. 14298. Available at: https://www.iza.org/publications/dp/14298/learning-loss-and-educational-inequalities-in-europe-mapping-the-potential-consequences-of-the-covid-19-crisis.
Flannery, Darragh. n.d. ‘Twitter Post. May 10, 11:32 AM. Https://Twitter.Com/Darraghflanner1/Status/1391702642276118529’. @darraghflanner1.
Kelly, Emma O. 2020. ‘Schools to Be given €10m in Technology Funding’. RTÉ, April. https://www.rte.ie/news/2020/0422/1134053-education-coronavirus/.
Klieme, Eckhard. 2016. ‘How Are They Related on the Country Level ?’ German Institute for International Educational Research, Working Paper published online. Available at: https://pisa.dipf.de/de/pdf-ordner/Klieme_TIMSS2015andPISA2015.pdf.
OECD, Organisation for Economic Co-operation and Development. 2021. ‘What Is Pisa?’ 2021. https://www.oecd.org/pisa/.
UCL, University College London. 2019. ‘The PISA Results Are Coming! But Should the Findings Be Trusted?’ Institute of Education. 1 October 2019. https://www.ucl.ac.uk/ioe/events/2019/dec/pisa-results-are-coming-should-findings-be-trusted.
1 See: https://timssandpirls.bc.edu/
2 See PISA FAQs: https://www.oecd.org/pisa/pisafaq/ 3 Importantly, both of these resources were not included in the JCR analysis.
3 Importantly, both of these resources were not included in the JCR analysis.
4 This was done using the RIFREG command in STATA.
5 PISA aims to measure functional literacy in maths and reading in the sense of how students can apply their skills to function in their societies