Last fall I spent a fair bit of time analyzing the determinants of midterm performance (e.g., attendance and video lecture watching) in my big econometrics class. It was difficult to interpret many of the results because of the classic correlation does not equal causation problem. For example, I really wanted to know how time spent studying affected scores, and found that reported hours spent studying was negatively correlated with scores. I think it is unlikely that the causal effect of an additional hour of studying is negative and it is much more likely that the students having the most trouble with the material were the ones who studied the most. And then there’s the fact that quality of studying matters at least as much as quanitity.
At the end of that semester my friend (and cohost on the Teach Better Podcast) Edward O’Neill and I came up with a new question to try and capture how students study:
What did you do to study for the midterm exam? (Choose all that apply)
a. Read the book
b. Rewatch portions of the video lectures
c. Go over your notes
d. Study the problem sets and solutions
e. Rework problems from the problem sets
f. Work new sample problems
This year I added this question to my midterm survey along with the same questions I asked last year (e.g., How much time did you spend studying for the midterm exam?). I also put the survey right into the exam and gave students two points for filling it out instead of emailing a link to a Google Form survey the next day. This brought the response rate from 74% to 100%. If only all surveys could use this method to combat nonresponse.
Here’s what my 121 students reported:
Type of study  Percent who did it
+
Read the book  31%
Rewatch video lectures  55%
Go over notes  77%
Study problems+solutions  90%
Rework problem sets  26%
Work new problems  57%
I think it’s fascinating how few students read the book or go back and rework problems. Far more seem to be reading problems and solutions.
Here’s how midterm exam scores looked for those that did and did not engage in each type of studying:
 Avg midterm of  and those
Type of study  those who did it  who did not
++
Read the book  64.5  64.6
Rewatch video lectures  62.5  67.1
Go over notes  65.4  61.7
Study problems+solutions  64.6  64.4
Rework problem sets  64.5  64.6
Work new problems  65.0  64.0
The only difference that is even marginally significant (p=0.09) is between students who review and do not review the video. At least part of the reason for these somewhat unintuitive results is that students who differ in how they study likely differ on a variety of other relevant characteristics. For example, those students who reported rewatching video as a study strategy also attended far fewer lectures.
It is slightly more illuminating to look at the results with a regression model that tries to control for background knowledge of statistics, average weekly study hours, number of hours spent studying specifically for the midterm, and number of lectures attended. Next week I’ll analyze data on how many of the lectures they watched on video during the semester, but in the meantime, here’s what I have:
midterm  Coef. Std. Err. t P>t
+
somestats  2.984823 3.310259 0.90 0.369
lotstats  .0202767 4.214091 0.00 0.996
reg_study_3to4  5.449729 5.051488 1.08 0.283
reg_study_5to6  2.721032 5.322659 0.51 0.610
reg_study_7plus  .6499088 6.988505 0.09 0.926
mt_study_4to8  2.496174 6.24482 0.40 0.690
mt_study_9plus  12.36427 6.362831 1.94 0.055

ms_nlectures 
68  .1417638 4.641844 0.03 0.976
910  3.160453 4.600167 0.69 0.494
1113  5.171283 4.372493 1.18 0.240

ms_how_book  1.15838 3.09346 0.37 0.709
ms_how_video  2.205832 2.959473 0.75 0.458
ms_how_notes  2.535813 3.355605 0.76 0.452
ms_how_review_ps  1.288894 4.962381 0.26 0.796
ms_how_rework_ps  5.096169 4.452382 1.14 0.255
ms_how_new_probs  5.07274 4.033747 1.26 0.211
_cons  64.55459 7.971838 8.10 0.000
First the bad news: Not a single coefficient is statistically significant at the 5% level. This is in part due to a relatively small sample size (121 students) and in part due to missing important confounding variables. And the only coefficient that is significant at a 10% level is on studying at least 9 hours for the midterm. That means, holding everything else constant, students who studied 9 or more hours for the midterm scored 12 points lower than those students who studied less than four hours.
The slightly good news is that the coefficients on the variables representing how students studied are consistent with current research in cognitive science on effective studying, even if they aren’t statistically significant. In particular, the effects of actually working problems that will be similar to those found on the exam (about 5 points) are higher than any of the other study strategies.
In his fantastic video series on how students can get the most out of studying, Stephen Chew emphasizes study strategies that involve deep processing of information. He gives many examples, but one is to practice working problems. He also points out there are more and less effective ways to read a textbook and review notes. Next time I’m going to create a more detailed set of questions to better distinguish studying that involves shallow vs. deep processing.
In addition to Dr. Chew’s videos, the following podcast, video, and article also discuss how we can apply insights from cognitive science in a college context:

How to use cognitive psychology to enhance learning (Dr. Robert Bjork on the Teaching in Higher Ed podcast)

Science of Student Ratings (a 40m presentation by Sam Moulton, Director of Educational Research and Assessment at Harvard’s Derek Bok Center)

Applying Psychological Science to Higher Education: Key Findings and Open Questions (an article by Sam Moulton)