Multilevel Design Parameters and Effect Size Benchmarks for Student's Competencies
Aim
The present project is the first to rigorously examine design parameters and effect size benchmarks of intervention studies in education that are targeted to the multilevel framework of Germany’s school system. Therefore, MULTI-DES is to make substantial contributions to evidence-based education.
The MULTI-DES project is led by the University of Potsdam (Prof. Dr. Martin Brunner, Chair of Quantitative Methods in Education Sciences). Prof. Dr. Oliver Lüdtke (IPN Leibniz Institute for Science and Mathematics Education, Kiel) and Prof. Dr. Cordula Artelt (LIfBi Leibniz Institute for Educational Trajectories, Bamberg) are also involved in the project.
Background
The advancement of evidence-based educational practices and policies hinges on the availability of rigorous research and its effective communication to a broad audience. Particularly, educational intervention studies need to be conducted in ecologically valid settings to probe whether the interventions actually work in the field. Such studies are often designed as so-called group-randomized or cluster-randomized trials where entire groups (e.g., entire schools) are randomly assigned to experimental conditions.
To ensure sufficient statistical power, several design parameters are essential to determine the sample size (i.e., number of students, classes, and schools) of group-randomized trials. Several design parameters are essential for the quality of these studies:
- Design parameters comprise estimates of between-school and between-class differences (in terms of intraclass correlations)
- as well as the amount of variance explained by covariates (e.g., pretest scores) at the individual, class or school level.
- Moreover, to effectively communicate effects of educational interventions on students’ competencies requires empirical effect size benchmarks that serve as vital aids to interpret empirical results.
- Crucially, both design parameters and effect size benchmarks should correspond to the target population.
Approach
The present project is the first to rigorously examine design parameters and effect size benchmarks that are targeted to the multilevel framework of Germany’s school system. To this end, we will conduct four studies that capitalize on data from three German longitudinal large-scale assessments: the National Educational Panel Study (NEPS), the longitudinal extension of the German year 2003 cycle of PISA (PISA-I+), and the Assessment of Student Achievements in German and English as a Foreign Language (DESI).
- Study 1 will examine design parameters in cluster-randomized studies for the general student population as well as for different school types.
- Study 2 investigates how various covariates affect the precision/statistical power of group-randomized trials.
- Study 3 examines average academic-growth as effect size benchmark for the general student population and different school types.
- Study 4 investigates performance gaps between weak and average schools as effect size benchmarks.