[For an overview of TBL, see this post]
In my last post, I described the Readiness Assessment process; by the time students are done with the team RA, they are ready for the fun stuff: the team applications. For the next several class meetings, class time is almost entirely devoted to teams working on problems that require they think through and discuss the kind of data issues that empirical researchers routinely address. During the all-class discussion that follows the team discussions, my contribution tends to be limited to directing the discussion. I may spend a few minutes at the end of class tying together concepts but that's as close as I get to lecturing. TBL 'guidelines' say that good applications should satisfy the 4 S's: Significant problem, Same problem, Specific choice, Simultaneous report.
Significant problem
This seems sort of obvious - of course you want students to be working on problems that are 'significant', which I take to mean both relevant and complex enough to require application and integration of course concepts. But actually coming up with 'significant' problems isn't always easy. It would be great if I could use some canned examples from textbook supplements but an awful lot of those are neither relevant nor complex, particularly because I want students to think and not plug-and-chug. For most economics course, I think current events are probably the best source for applications. I use a number of applications based on headlines and real-world examples, many involving issues that are not actually resolved among experts (more about this in a minute). For example, for discussion of issues with defining vague concepts, students are asked to decide what variable to use in a comparison of whether Americans are 'better off' today than they were four years ago.
Same problem
I've had groups in other classes do 'jig-saw' problems, where each group (or member of the group) works on a slightly different problem and then the parts have to be put together to form an overall conclusion/product. While I think this can work for some purposes, it is better for discussion to have everyone working on the same problem. Both within and between groups, discussion is more lively as students compare answers and then must defend and explain their approach to the problem.
Specific choice
From my perspective, this is the key factor for ensuring productive team discussions. Teams must agree on ONE choice and, as a team, defend that choice to other teams. I think teachers usually tend to prefer open-ended questions for getting students to think critically. Yet for most of my applications, the way I force students to make a 'specific choice' is by giving them multiple choice options. However, a large number of the applications have more than one answer choice that could be 'right', depending on what assumptions you make. For example, one of my favorite applications is about the CPS definition of 'income'. Before class, students go to the CPS website and get data on median household income and have to read about how the CPS defines income. The team application reiterates that income does not include noncash benefits, including in-kind transfers like food stamps and employer-provided work benefits, and asks:
On the worksheets that the teams complete, I ask them specifically to identify any assumptions that they are making but this is always the most difficult part for them - it really takes some work to get them to admit that they are 'assuming' information. But it helps that the answers are always split, both within teams and then among teams, which means there is someone else in the room who is making very different assumptions and coming to a different conclusion.
The biggest challenge for me with these applications is that certain students really hate that there is no one 'right' answer. I've learned that I really have to emphasize to students the 'if-then' nature of the answers; that is, "IF you assume X, THEN you would want to choose answer A; but IF you assume Y, THEN you would want to choose answer B." [I should add that I also point out to them that, usually, both X and Y are assumptions people might have for different reasons, like political beliefs, which is why I'm constantly telling them all to be aware of their own biases...]. I also repeatedly tell students that although some questions may have more than one 'right' answer, there is still a 'right' way to think about the questions, i.e., to identify what assumptions will lead to different answers. I try to point out to them that this is the way the 'real world' works - for example, when they hear politicians saying things that seem to be contradictory, it doesn't necessarily mean that one side is "wrong" but that they are making different assumptions. Of course, then you should next ask whether those assumptions are valid!
Simultaneous report
After the teams have some time to discuss the application and make their choice, every team reports their choice at the same time. This means teams can't change their answer once they see what everyone else chose. The way I do the simultaneous report is to use whiteboards - each team writes the letter corresponding to their choice on their boards and holds it up at the same time. I used clickers the first time I taught the class, which made it easy to see how many teams had chosen each answer but I couldn't see which team had chosen what. I think that's more important because teams need to be accountable for their answer - when I ask Team 6 to explain their choice, they can't just say they chose whatever the most popular answer was. Similarly, I can see immediately if one or two teams choose something different than all the others; that's usually where I start the all-class discussion.
Some logistical issues
Individual responses: I do have students register their choices individually before the team discussion (using PollEverywhere). I may or may not show them the distribution of those answers. The important thing is that they have to give the question at least a little bit of independent thought, and they have to register their own choice, before discussing the problem with their team. This not only gives the teams an easy starting point (they go around and ask everyone what answer they chose) but I think that once people have registered an answer, even if they don't have a ton of confidence in it, they will either try to defend it or want a coherent explanation of why the team should go with a different choice.
Team worksheets: For each application, there is a team worksheet where the teams must write down which answer choice they selected and provide a justification for that choice, including identifying any assumptions they are making. I do grade these worksheets, though on mostly a "plus, check, minus" kind of scale. One of the issues I'm still trying to figure out is how to make those worksheets more productive for everyone. What typically happens is that one person (either the 'smartest' person or the one with the best writing) ends up always writing up the explanation and everyone else just sort of sits around while he or she does that. I'm trying to decide if I want to 'force' teams to rotate this responsibility, since I think whoever does the writing probably learns the most.
Timing: Another thing I am constantly re-evaluating is how much time to give teams for the applications. This has largely been trial and error: the first few times I taught the class, I gave the teams a set amount of time (usually 5 to 7 minutes) and then would monitor the teams and give them additional time as necessary. But I got a lot of complaints from students that they needed more time. This past spring, I decided to make the time open-ended and gave each team a green piece of paper, folded in half. When they were done, they were supposed to put the green 'tent' on someone's desk so I could see they were done. When about half the teams were done, I'd tell everyone else they have 1 minute to finish up. That seemed to work well and although the teams took a lot longer, we never ran out of time either (I'm not entirely sure how that happened but I think it may be that because they had more time to discuss without feeling rushed, I had to do less explaining afterwards).
All-class discussion: When there are one or two teams that select a different answer than everyone else, I will ask someone from that team to explain their answer, and then ask someone from a different team to explain why they made a different choice. I try to learn all the students' names so I can call on individuals - if I don't, there is a tendency for the same people to always answer for a given team. I'm still playing around with this; sometimes I will call on teams randomly by using a deck of cards, or a random student-picker app. If there is a lot of agreement about one particular answer, I will sometimes extend discussion by asking students to identify why someone might choose one of the other options, or I ask someone who chose a different option with PollEverywhere originally to explain their thinking.
In my next post, I'll discuss the final step of the TBL process, team evaluations.
In my last post, I described the Readiness Assessment process; by the time students are done with the team RA, they are ready for the fun stuff: the team applications. For the next several class meetings, class time is almost entirely devoted to teams working on problems that require they think through and discuss the kind of data issues that empirical researchers routinely address. During the all-class discussion that follows the team discussions, my contribution tends to be limited to directing the discussion. I may spend a few minutes at the end of class tying together concepts but that's as close as I get to lecturing. TBL 'guidelines' say that good applications should satisfy the 4 S's: Significant problem, Same problem, Specific choice, Simultaneous report.
Significant problem
This seems sort of obvious - of course you want students to be working on problems that are 'significant', which I take to mean both relevant and complex enough to require application and integration of course concepts. But actually coming up with 'significant' problems isn't always easy. It would be great if I could use some canned examples from textbook supplements but an awful lot of those are neither relevant nor complex, particularly because I want students to think and not plug-and-chug. For most economics course, I think current events are probably the best source for applications. I use a number of applications based on headlines and real-world examples, many involving issues that are not actually resolved among experts (more about this in a minute). For example, for discussion of issues with defining vague concepts, students are asked to decide what variable to use in a comparison of whether Americans are 'better off' today than they were four years ago.
Same problem
I've had groups in other classes do 'jig-saw' problems, where each group (or member of the group) works on a slightly different problem and then the parts have to be put together to form an overall conclusion/product. While I think this can work for some purposes, it is better for discussion to have everyone working on the same problem. Both within and between groups, discussion is more lively as students compare answers and then must defend and explain their approach to the problem.
Specific choice
From my perspective, this is the key factor for ensuring productive team discussions. Teams must agree on ONE choice and, as a team, defend that choice to other teams. I think teachers usually tend to prefer open-ended questions for getting students to think critically. Yet for most of my applications, the way I force students to make a 'specific choice' is by giving them multiple choice options. However, a large number of the applications have more than one answer choice that could be 'right', depending on what assumptions you make. For example, one of my favorite applications is about the CPS definition of 'income'. Before class, students go to the CPS website and get data on median household income and have to read about how the CPS defines income. The team application reiterates that income does not include noncash benefits, including in-kind transfers like food stamps and employer-provided work benefits, and asks:
The exclusions in the CPS definition mean that the measured income gap between rich and poor (assume ‘rich’ and ‘poor’ measured by 10th and 90th percentile of the population), when compared to the "true" income gap, is likely to be:Almost all students quickly recognize that the exclusion of noncash benefits means that the measured income of poor people is likely less than their 'true' income. Some students see that richer people's incomes are also understated because work benefits are not included either; someone always also points out that income is measured pre-tax and rich people pay more in taxes. The point is that through the team discussion, students have to identify not only how different people's incomes are affected but they have to think about how big those effects are likely to be. And since none of them have any real idea what those magnitudes are, they have to make assumptions, and those assumptions will ultimately drive which answer they choose.
A. understated
B. overstated
C. unaffected
On the worksheets that the teams complete, I ask them specifically to identify any assumptions that they are making but this is always the most difficult part for them - it really takes some work to get them to admit that they are 'assuming' information. But it helps that the answers are always split, both within teams and then among teams, which means there is someone else in the room who is making very different assumptions and coming to a different conclusion.
The biggest challenge for me with these applications is that certain students really hate that there is no one 'right' answer. I've learned that I really have to emphasize to students the 'if-then' nature of the answers; that is, "IF you assume X, THEN you would want to choose answer A; but IF you assume Y, THEN you would want to choose answer B." [I should add that I also point out to them that, usually, both X and Y are assumptions people might have for different reasons, like political beliefs, which is why I'm constantly telling them all to be aware of their own biases...]. I also repeatedly tell students that although some questions may have more than one 'right' answer, there is still a 'right' way to think about the questions, i.e., to identify what assumptions will lead to different answers. I try to point out to them that this is the way the 'real world' works - for example, when they hear politicians saying things that seem to be contradictory, it doesn't necessarily mean that one side is "wrong" but that they are making different assumptions. Of course, then you should next ask whether those assumptions are valid!
Simultaneous report
After the teams have some time to discuss the application and make their choice, every team reports their choice at the same time. This means teams can't change their answer once they see what everyone else chose. The way I do the simultaneous report is to use whiteboards - each team writes the letter corresponding to their choice on their boards and holds it up at the same time. I used clickers the first time I taught the class, which made it easy to see how many teams had chosen each answer but I couldn't see which team had chosen what. I think that's more important because teams need to be accountable for their answer - when I ask Team 6 to explain their choice, they can't just say they chose whatever the most popular answer was. Similarly, I can see immediately if one or two teams choose something different than all the others; that's usually where I start the all-class discussion.
Some logistical issues
Individual responses: I do have students register their choices individually before the team discussion (using PollEverywhere). I may or may not show them the distribution of those answers. The important thing is that they have to give the question at least a little bit of independent thought, and they have to register their own choice, before discussing the problem with their team. This not only gives the teams an easy starting point (they go around and ask everyone what answer they chose) but I think that once people have registered an answer, even if they don't have a ton of confidence in it, they will either try to defend it or want a coherent explanation of why the team should go with a different choice.
Team worksheets: For each application, there is a team worksheet where the teams must write down which answer choice they selected and provide a justification for that choice, including identifying any assumptions they are making. I do grade these worksheets, though on mostly a "plus, check, minus" kind of scale. One of the issues I'm still trying to figure out is how to make those worksheets more productive for everyone. What typically happens is that one person (either the 'smartest' person or the one with the best writing) ends up always writing up the explanation and everyone else just sort of sits around while he or she does that. I'm trying to decide if I want to 'force' teams to rotate this responsibility, since I think whoever does the writing probably learns the most.
Timing: Another thing I am constantly re-evaluating is how much time to give teams for the applications. This has largely been trial and error: the first few times I taught the class, I gave the teams a set amount of time (usually 5 to 7 minutes) and then would monitor the teams and give them additional time as necessary. But I got a lot of complaints from students that they needed more time. This past spring, I decided to make the time open-ended and gave each team a green piece of paper, folded in half. When they were done, they were supposed to put the green 'tent' on someone's desk so I could see they were done. When about half the teams were done, I'd tell everyone else they have 1 minute to finish up. That seemed to work well and although the teams took a lot longer, we never ran out of time either (I'm not entirely sure how that happened but I think it may be that because they had more time to discuss without feeling rushed, I had to do less explaining afterwards).
All-class discussion: When there are one or two teams that select a different answer than everyone else, I will ask someone from that team to explain their answer, and then ask someone from a different team to explain why they made a different choice. I try to learn all the students' names so I can call on individuals - if I don't, there is a tendency for the same people to always answer for a given team. I'm still playing around with this; sometimes I will call on teams randomly by using a deck of cards, or a random student-picker app. If there is a lot of agreement about one particular answer, I will sometimes extend discussion by asking students to identify why someone might choose one of the other options, or I ask someone who chose a different option with PollEverywhere originally to explain their thinking.
In my next post, I'll discuss the final step of the TBL process, team evaluations.
Thank you so much for this detailed and incredibly helpful description!
ReplyDelete