Study Habits, Lecture Attendance, and Exam Performance

- - posted in data, teaching

Now that I have a fair bit of data on my students’ performance and participation in the class, I’ve been champing at the bit to start analyzing it. I figured this would have to wait until the end of the semester since my plate is pretty full with class prep and other responsibilities, but the other day over lunch, my friend Edward and I had a great idea: Why not combine the two and analyze the data during lecture? It would make a great introduction to multiple regression and hopefully teach students how to use their study time more efficiently.

I started by collecting even more data with yet another survey of the class:

On Tuesday night, while I waited for the data to come in, I prepared Wednesday’s lecture. I would start by showing the correlation between study hours and exam scores using bivariate regression. I expected the relationship to be weakly positive because it was confounded with prior background. That is, those students who had seen some or most of the material before (about half the class) would study less but still perform well on the exam. Then I would show them how to control for prior background and we’d see a decent estimate of the causal effect of studying and everyone would say “Oh! I should study more!”

Wednesday morning at 10am, I closed the survey with a 74% response rate and by 11am all my data was set up nicely. The lecture was completely ready–I just needed to run the regressions and drop them into my slides. Here’s what I got with the first one:

. reg midterm midterm_study_hrs

      Source |       SS       df       MS              Number of obs =     107
-------------+------------------------------           F(  1,   105) =    0.62
       Model |  55.3687445     1  55.3687445           Prob > F      =  0.4343
    Residual |  9436.22004   105  89.8687623           R-squared     =  0.0058
-------------+------------------------------           Adj R-squared = -0.0036
       Total |  9491.58879   106  89.5432904           Root MSE      =  9.4799
-----------------------------------------------------------------------------------
          midterm |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
midterm_study_hrs |  -.2579158   .3285868    -0.78   0.434    -.9094427     .393611
            _cons |    81.5407   2.494516    32.69   0.000     76.59454    86.48687
-----------------------------------------------------------------------------------

Those of you familiar with regression will immediately see a problem. For each hour of studying, the average exam score drops by a quarter point. It’s not statistically significant, but still. I took a deep breath and controlled for prior exposure to the material:

. reg midterm midterm_study_hrs nostats

      Source |       SS       df       MS              Number of obs =     107
-------------+------------------------------           F(  2,   104) =    0.63
       Model |  113.430728     2   56.715364           Prob > F      =  0.5352
    Residual |  9378.15806   104  90.1745967           R-squared     =  0.0120
-------------+------------------------------           Adj R-squared = -0.0071
       Total |  9491.58879   106  89.5432904           Root MSE      =   9.496
-----------------------------------------------------------------------------------
          midterm |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
------------------+----------------------------------------------------------------
midterm_study_hrs |  -.2160917   .3332468    -0.65   0.518    -.8769325    .4447492
          nostats |   2.363543   2.945505     0.80   0.424    -3.477503     8.20459
            _cons |   80.98032   2.594512    31.21   0.000     75.83531    86.12534
-----------------------------------------------------------------------------------

Uh oh. Studying still has a negative effect and having no background in statistics increases exam scores. Again, nothing’s significant, but God help me if my class interprets these as causal effects. The most interesting result I had was that students who filled out the survey scored a highly significant 5 points higher on the midterm than those who didn’t. After a few more deep breaths, I fleshed out my lecture and walked to the classroom to teach.

I thanked them for filling out the survey and showed them some summary statistics–e.g., There was fairly wide variation in how much time they spend studying regularly, how much they studied specifically for the midterm, and in how much they attended lecture. Then I showed them the results for the first regression and asked for a volunteer to interpret the coefficient on study hours. There was a moment of dead silence followed by nervous laughter. One student eventually stated that more studying was associated with lower exam scores and there was more laughter.

That’s when we stepped back to look at the assumptions that underlie regression and together we identified which ones didn’t hold in this case. We spent the rest of the class extending the regression model and found that there was really nothing we could do with this data to get around the fundamental problem that unobserved confounders prevented us from plausibly estimating a causal effect of studying or lecture attendance. The below model includes everything I had, but results of more parsimonious models look extremely similar:

. reg midterm i.q1 i.q2 i.q3 i.q4 i.q5
 
     Source |       SS       df       MS              Number of obs =     107
-------------+------------------------------           F( 13,    93) =    1.13
       Model |  1292.13126    13   99.394712           Prob > F      =  0.3467
    Residual |  8199.45753    93    88.16621           R-squared     =  0.1361
-------------+------------------------------           Adj R-squared =  0.0154
       Total |  9491.58879   106  89.5432904           Root MSE      =  9.3897
----------------------------------------------------------------------------------
         midterm |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-----------------+----------------------------------------------------------------
           q1_bg |
           some  |  -3.090785   3.193788    -0.97   0.336    -9.433014    3.251444
           none  |  -2.547597   3.235462    -0.79   0.433    -8.972583    3.877389
                 |
     q2_lectures |
            1-5  |  -1.908923   7.591896    -0.25   0.802    -16.98492    13.16708
           6-10  |  -7.650552   7.231432    -1.06   0.293    -22.01074    6.709638
          11-12  |  -6.123254   7.188493    -0.85   0.397    -20.39818    8.151669
             13  |  -2.858527   7.210257    -0.40   0.693    -17.17667    11.45961
                 |
    q3_reg_study |
        3-4 hrs  |  -5.663912   2.648944    -2.14   0.035    -10.92419   -.4036348
        5-6 hrs  |   -5.11663   3.149536    -1.62   0.108    -11.37098    1.137725
         7+ hrs  |  -5.584407   4.220263    -1.32   0.189    -13.96501    2.796198
                 |
q4_midterm_study |
        4-8 hrs  |   .2385502   2.822675     0.08   0.933    -5.366723    5.843823
         9+ hrs  |   -.562619   2.874504    -0.20   0.845    -6.270815    5.145577
                 |
    q5_how_study |
  half and half  |  -1.760089   2.395351    -0.73   0.464    -6.516782    2.996603
 mostly friends  |  -6.778929   5.962874    -1.14   0.259    -18.62001    5.062157
                 |
           _cons |   92.27343   7.876745    11.71   0.000     76.63177    107.9151
----------------------------------------------------------------------------------

During the last twenty minutes of class we broke into groups of 5-6 to talk about what we could do to remedy the situation. My teaching assistants and I wandered the room and facilitated the discussion. The groups came up with some good ideas. We could try to measure and control for raw mathematical ability and how much sleep students got the night before the exam since these were likely confounders. Another group thought we could collect the same measures before the final exam then see how changes in exam scores correlate with changes in study habits. That group invented individual fixed effects and I’m definitely going to do what they suggested.

In hindsight, I wish I had introduced multiple regression with a case study or two where it worked well. At the same time, I think they learned a healthy respect for the difficulty of data analysis and the limitations of seemingly advanced methods.

Comments