Michelle Smith is an Associate Professor in the School of Biology and Ecology at the University of Maine, and she’s one of the world’s leading discipline-based education researchers. Among many other things, she studies why and how peer discussion works as an effective teaching tool, collaborates with biology teachers in college and high school settings, and develops concept inventories (standard assessments of learning) at the course and program level. In this episode, we talk about the benefits of using concept inventories in your own classes, and Michelle gives advice for finding, creating and/or giving them.

You can subscribe to the Teach Better Podcast through your favorite podcast app or simply subscribe through iTunes if you don’t have one yet.

Show Notes

0:00 ⏯ Intro

0:35 ⏯ Introducing Michelle Smith. What is a concept inventory assessment? Assessing teaching vs. assessing individual students.

4:24 ⏯ DON’T use concept inventory assessment items piecemeal on your exams. Using a concept inventory assessment with low stakes. Students may solve problems using reasoning, but it doesn’t focus on calculations. These assessments may be used for a course or a major, and they generally range from 30 to 65 multiple choice or true-false questions.

7:43 ⏯ When developing a concept inventory, interviewing students is the most important and also the most fun part. The students read the questions (without the answers) and think out loud about the question. Students should have a range of levels of understanding and be at a range of places in their majors. It’s the growth of student understanding across a course or major that’s the focus. It’s an opportunity to see what students know and don’t know, where you’re helping them and where you’re not.

10:38 ⏯ Standard assessments of students’ attitudes towards or perceptions of the field or discipline. Sometimes students don’t understand how research gets published.

12:19 ⏯ Why 30 minutes? That’s a reasonable amount of class time to give up. Students perform a bit differently when the assessment is online: their motivation goes down a bit. Students take these assessments more seriously when the instructor explains why the tool is being used: it’s to improve their experience. If you assess at the beginning and near the end of the course, you can measure growth. Motivation for the later test is lower, so you can incentivize students.

17:33 ⏯ How is a validated instrument different from a good exam question? Starting with your exam questions is actually a good idea. It’s hard to represent the content of a whole course in a small number of questions, so you have to pick and choose.

20:00 ⏯ Not all concept inventories are multiple choice. There are different question formats keyed to practical and disciplinary questions. Doug recaps McMillan’s Smartbooks quizzes which measure the student’s confidence in her answer.

28:02 ⏯ Where to find a concept inventory for your discipline:

• PhysPort: 82 physics assesments with most based on content knowledge
• db-SERC: The discipline-based Science Education Research Center provides many assignments for several disciplines
• Test of Understanding of College Economics: The one and only standard assessment for economics in higher education

Ultimately, web sites might let students take the concept inventories themselves and feed the information back to instructors.

33:15 ⏯ Guidelines for developing your own concept inventory and tips for getting started. The importance of talking and listening to students. Adams and Wieman (2010) on teaching expert-level concepts and learning goals.

37:00 ⏯ Talk to students earning C’s. Paying for the interview helps. Interview students for an hour, record the interviews, and write up your notes. A common misconception students have in genetics is: each cell has different DNA.

41:39 ⏯ Some students are missing basic pre-requisite knowledge. Different fields may use different validation procedures, and which procedures are best is the subject of on-going discussion.

46:21 ⏯ A record for links in our show notes. Edward says there is very little about pedagogy in media studies. Is this teaching to the test? Teaching to the concepts is exactly what we want faculty to do. It’s a moving target: it’s about finding out where students are and defining what that is. And it’s not just a question of giving them the right answer–or teaching would be a lot easier.

51:45 ⏯ This kind of conceptual inventory is a collaborative enterprise. Be prepared for your favorite questions to be shot down. And not every piece of faculty input survives. You need faculty and pilots at a range of institutions.

55:24 ⏯ Controlling access to the test online. Get ready for students to impersonate faculty members to get the test questions.

57:20 ⏯ Sometimes bad conceptual questions expose the professor’s own conceptual difficulties. The time Michelle had to write a concept inventory in a week. When a gene is not a gene, it’s an allele. Asking students to recognize common conceptual errors in media reporting. We all make conceptual mistakes in teaching: Doug’s examples. Comment below with your own conceptual mistake. The skill of calling bull on media reporting.

1:03:10 ⏯ Signing off.