Monday's Principles class was one of those 'growing' experiences that I know make me a better teacher but aren't all that fun when you're in them. I asked what I thought was a fairly straightforward review clicker question about unintended consequences. Here's the question:
Which of the following is an example of the unintended consequences of people responding to incentives?
A.Feeling sick after over-eating.
B.Sleeping through an exam because you were up all night studying.
C.Using a bigger font because your teacher says your paper must be at least ten pages long.
D.Paying taxes when you win the lottery.
For any non-economist readers, let me explain that when economists refer to the 'unintended consequences of people responding to incentives', we are referring to the idea that people always respond to incentives (changes in costs or benefits) but their response may not be what was intended by the person setting the incentives. So the correct answer here is C since playing with the formatting on a paper is clearly not what a teacher wants you to do to get your paper up to a certain length.
Well, among my students, 12% answered A, 45% answered B, 20% answered C and 23% answered D. Given that distribution, I asked the students to confer with a neighbor and then I re-asked the question. The second time around, 3% answered A, 67% answered B, 14% answered C and 17% answered D! In the ensuing discussion, a couple things became clear:
1) The answers are not well-formed. Many students did not understand the incentives involved in a teacher saying your paper must be a certain length. I am not sure if it would have mattered if I had phrased it differently (e.g., 'your teacher will dock points if your paper is under ten pages') because the problem seemed to be that students did not understand that when a teacher sets a target length for a paper, the intention is for students to include a certain amount of information - that is, they didn't see changing the font as an unintended consequence (I was too scared to ask what they think it means when a teacher says a paper should be ten pages long...). It may also be that all teachers now specify that papers must be in a particular font with particular margins so today's students are less familiar with the whole idea of gaming the format.
2) I had not taught the idea of 'unintended consequences' well. For answer B, students were caught up with the idea that studying hard for an exam is responding to the incentives of wanting to get a good grade, and sleeping through the exam is an unintended consequence of studying so hard. So in a very indirect way, I can understand why they would want to say that this was an unintended consequence of responding to incentives. But it took quite a while to get them to see the difference between the *response* being unintended (i.e., doing something specifically because the costs or benefits changed) and something unintended just happening.
Now, this was a question I just came up with for this class, partly because of a question a student had emailed me about this topic over the weekend, so this was the first time it was being 'vetted' by a class. It was a painful reminder of how hard it can be to come up with good multiple choice questions! But I'm sort of used to students complaining about not understanding questions (this has always been a struggle for me). What was more striking/disturbing to me was when I realized that I had made a newbie mistake in teaching the material - the concept is so familiar to me that I forgot to think about how it could be seen differently by my students. To give myself some slack, I'll say that I've never actually taught this concept so explicitly before (I've mentioned it and maybe given a few fun examples but I haven't spent any real time on it, this specifically, in the past). But it made me think hard about how I will need to change the way I explain it next time...
Related posts:
Dumping content
It's hard to get incentives right
Which of the following is an example of the unintended consequences of people responding to incentives?
A.Feeling sick after over-eating.
B.Sleeping through an exam because you were up all night studying.
C.Using a bigger font because your teacher says your paper must be at least ten pages long.
D.Paying taxes when you win the lottery.
For any non-economist readers, let me explain that when economists refer to the 'unintended consequences of people responding to incentives', we are referring to the idea that people always respond to incentives (changes in costs or benefits) but their response may not be what was intended by the person setting the incentives. So the correct answer here is C since playing with the formatting on a paper is clearly not what a teacher wants you to do to get your paper up to a certain length.
Well, among my students, 12% answered A, 45% answered B, 20% answered C and 23% answered D. Given that distribution, I asked the students to confer with a neighbor and then I re-asked the question. The second time around, 3% answered A, 67% answered B, 14% answered C and 17% answered D! In the ensuing discussion, a couple things became clear:
1) The answers are not well-formed. Many students did not understand the incentives involved in a teacher saying your paper must be a certain length. I am not sure if it would have mattered if I had phrased it differently (e.g., 'your teacher will dock points if your paper is under ten pages') because the problem seemed to be that students did not understand that when a teacher sets a target length for a paper, the intention is for students to include a certain amount of information - that is, they didn't see changing the font as an unintended consequence (I was too scared to ask what they think it means when a teacher says a paper should be ten pages long...). It may also be that all teachers now specify that papers must be in a particular font with particular margins so today's students are less familiar with the whole idea of gaming the format.
2) I had not taught the idea of 'unintended consequences' well. For answer B, students were caught up with the idea that studying hard for an exam is responding to the incentives of wanting to get a good grade, and sleeping through the exam is an unintended consequence of studying so hard. So in a very indirect way, I can understand why they would want to say that this was an unintended consequence of responding to incentives. But it took quite a while to get them to see the difference between the *response* being unintended (i.e., doing something specifically because the costs or benefits changed) and something unintended just happening.
Now, this was a question I just came up with for this class, partly because of a question a student had emailed me about this topic over the weekend, so this was the first time it was being 'vetted' by a class. It was a painful reminder of how hard it can be to come up with good multiple choice questions! But I'm sort of used to students complaining about not understanding questions (this has always been a struggle for me). What was more striking/disturbing to me was when I realized that I had made a newbie mistake in teaching the material - the concept is so familiar to me that I forgot to think about how it could be seen differently by my students. To give myself some slack, I'll say that I've never actually taught this concept so explicitly before (I've mentioned it and maybe given a few fun examples but I haven't spent any real time on it, this specifically, in the past). But it made me think hard about how I will need to change the way I explain it next time...
Related posts:
Dumping content
It's hard to get incentives right
What a great post. I recently talked to an instructor about some stumbling blocks his students constantly had. Because I was in essence a student in his discipline, (we were talking about an intro class) I was able to point out to him my stumbling blocks, namely not understanding certain foundational concepts/vocabulary he assumed the students would/should know.
ReplyDeleteI thought I understood incentives, I guess I don't :)
I picked A. My thinking was: In the US we incentivize overeating, not only in food portions (eat more for less money campaigns) but in the size of containers, dishware, etc.
No need to respond :)
So when you presented the material on unintended consequences (UC), how did you do it? That is, was the problem in student preconceptions that were not sufficiently dealt with, or in the approach? (What do you think would have happened, by the way, if you'd asked the question first, then presented the material on UC, then asked either the same question or a variant of it?)
ReplyDeleteI am sincerely interested in the examples you used in teaching the concept, because it is a difficult one for students. There is a tendency, evident in what you got, for students to think that, because I got an outcome *that I did not intend* from actions I took, that's UC. In this case, thinking as I think students think, "C" cannot be right; the techer wanted 10 pages and got 10 pages (your point about teachers adapting their instructions is also relevant; when I assign papers now, I give a target word count)...It's harder for them to realize that the point is that someone else has developed a structure of incentives designed to get people to do A, and, in the course of doing A, they also do B, which is an undesireable outcome.
I would like the record to reflect that I personally chose C, but I had to think about it! Choice C was the only one that provided a clear and actual response to an incentive.
ReplyDelete@doc: I introduced unintended consequences uses the first chapter of Freakonomics about cheating (what do teachers and sumo wrestlers have in common?) and a 1996 AER article about a Virginia policy about garbage (to encourage recycling the city started charging per bag of trash - fewer bags were picked up but avg weight of bags went up and there was more illegal dumping). I do think that next time, I will build up to UC in smaller steps - I think I need to make sure they solidly understand using incentives to encourage specific (intended) behavior first (i.e., maybe ask some questions to get them to identify simply a response to incentives and/or the intention behind a set of incentives), before moving on to the more subtle idea of people responding in ways that were not intended.
ReplyDelete