4  Clustering Variables Examined but Not Included

We explored the particular class section that the participants were recruited from as a potential covariate. There were many aspects of the classes that were homogeneous: all instructors were required to adhere to the same set of pre-defined learning objectives, and beginning in cohort 3’s semester (during which over half our sample participated), the department instituted measures which required classes to synchronize exam schedules, recitation quizzes, homework systems, and textbooks. Nevertheless, each class has its own variation in terms of the number of students enrolled, days and times they meet, the amount of synchronous vs. asynchronous activities, and the students that select to be in those classes. Furthermore, all instructors have idiosyncratic aspects to their teaching methods and can have different demands and resources in the class.  The inclusion of class by instructor as a covariate was explored, but ultimately not included. Seven instructors taught the physics classes represented in our sample. Most instructors taught one class except one instructor who taught two, and another who taught 4 (total of eleven classes).

The number of students associated with each class ranged from 5 to 21 (M = 13.5, SD = 5.01). Theoretically, it made sense to include class and instructor as nested random intercept terms because we wanted to account for clustering by class and instructor, but we did not have any predictions about specific classes or professors. However, the ICC for class and instructor was at or very close to zero in all the models. This indicates that statistically, observations within classes and instructors were no more similar to each other than to observations from different classes and instructors. We also conducted a visual inspection of the all the focal study variables by instructor and did not detect any differences that appeared systematic. Based on these analyses, we did not include class or instructor in the reported models.