I spend a fair amount of time during class extolling the virtues of evidence-based decision making. The tools of data analysis that I teach are applicable to a wide array of fields and questions, and teaching itself should be no different. We should be using available data to tell us how different aspects of a class are working and inform us about the progress of particular students while there is still time to intervene.

This is the first semester I’ve had enough students to learn interesting things from the aggregate numbers, and there’s quite a bit of data to analyze. With some advance planning, I could have even more. Below, I divide the data into two categories: measures of performance and measures of participation.

Measures of Performance

• Exams: My midterm and final exams tend to have 6-8 questions each with 4-6 parts. I track every student’s score on each part, and the parts can be categorized by subject matter. The parts also vary in difficulty level and may allow me to compute rough measures of student mastery.
• Problem Sets: Ideally, I would have measures of student performance for every question on every problem set, but I am having my teaching assistants grade just one randomly selected question from assignment. This frees up the TA’s to spend more time with students, and I expect my students to read the solutions to learn where they stand in the class. I’m not sure this was a wise decision. I miss out on good data and my students would rather be scored on their entire effort.
• Online Quizzes: Every week my students take an online quiz on the key concepts covered in lecture. I let them take the quiz as many times as they want. Their first score is not a bad measure of what they learned from lecture while the last score measures what they learned after being told what the key concepts were.
• In-lecture problems: In most of my lectures, the students work through a few problems and click in their answers. If they’ve registered their clickers, I can record who said what. Right now I’m not tracking this, but I plan to next year and might even start doing so before the end of the semester. As of now, only about 2/3 of my students are using the clickers and it’s definitely not a random subset.
• Big Empirical Project: This project involves analyzing real data to answer about 25 different questions. My students pass in their answers to the first 6 at the semester midpoint, and the rest at the end of the semester. Like the exam questions, these can be divided by subject matter and difficulty.

Measures of Participation

• Lecture attendance: I have the number of students for all the lectures where I remembered to count, but I don’t currently have individual-level attendance data. If I make clickers mandatory, I can have everyone who shows up click in to a question at the very beginning of class. I’m definitely doing this next year. Attendance won’t make up part of the their grade, but I do want to know how it correlates with performance.
• Section attendance: If my TA’s have clicker receivers, they can do the same thing I do in lecture.
• Answering questions in lecture: Not everyone clicks in answers to questions–I can use this as a measure of engagement with the lecture.
• Watching video: Every one of my lectures is automatically recorded. At least a few students are watching these in lieu of attending in person. I’d love to know how these students perform relative to the students who attend and those who don’t watch or attend. Some students review the lectures after attending or before the exams. The Echo360 system tells me who watches, when they watch, and what parts they are watching. I would absolutely love to have this data for my online classes where the video is the lecture.
• Course Web Site Activity: I know Canvas tracks total page views per day per student and this is interesting, but I’d rather have more fine-grained measures so I can see what exactly they are looking at. I could see when they downloaded assignments and use this as a noisy measure of when they started working on them. How does this correlate with their performance on that assignment? What students are downloading the supplementary course materials?
• Weekly Quizzes: The quiz data can be a measure of participation as well as performance. I can track when and how how many times students are taking the weekly quizzes.
• Discussion Forum participation: We use the Piazza discussion forums heavily outside class time to discuss course material. My feeling is that students learn a lot in these forums, and I’m planning to write more about them soon. Even though participation in the forums is often anonymous, Piazza lets me download the number of questions posted, answered, and viewed for each student.

There is plenty of student engagement I don’t observe. I don’t know how much time they spend reading the book, reviewing their notes, or actually working on the course assignments. I don’t know anything about the quality of that studying or who they are studying with. That said, I’ve barely scratched the surface of analyzing what I do have.

What can we do with this data?

One thing I’d like to know is how different types of participation are related to performance. Computing the correlation between these measures is straight-forward, but we all know correlation is not the same as causation. For example, does asking a question (and getting an answer) in the forum improve performance or just signal that a student doesn’t understand something? We need experimental or quasi-experimental variation to really answer these questions, but I believe the correlations are at the very least helpful for hypothesis generation. That is, these correlations can suggest what experiments should be run, and in some cases they can be plausibly interpreted as causal effects.

I think this data could also be useful in providing additional feedback to students. Suppose the students who review the lecture video before the exam do better on the exams. Suppose students who download problem sets early get higher scores on those problem sets. Giving students this information as well as telling them where they stand relative to the rest of the class in the participation measures might induce higher participation and (hopefully) higher performance.

Finally, I would love to be able to see the effects of my activity on student performance and engagement so I can optimally allocate my time between preparing lectures, writing problem sets, meeting with students, answering questions in the forums, or something entirely different. While I doubt the data described above can help me with this issue by itself, I do think the data are a necessary component of any experiments that would.