Richland, Kornell, Kao - 2009 - The Pretesting Effect Do Unsuccessful Retrieval Attempts Enhance Lea

The Pretesting Effect: Do Unsuccessful Retrieval Attempts Enhance Learning? Lindsey E. Richland University of California, Irvine Nate Kornell Williams College Liche Sean Kao University of California, Irvine Testing previously studied information enhances long-term memory, particularly when the information is successfully retrieved from memory. The authors examined the effect of unsuccessful retrieval attempts on learning. Participants in 5 experiments read an essay about vision. In the test condition, they were asked about embedded concepts before reading the passage; in the extended study condition, they were given a longer time to read the passage. To distinguish the effects of testing from attention direction, the authors emphasized the tested concepts in both conditions, using italics or bolded keywords or, in Experiment 5, by presenting the questions but not asking participants to answer them before reading the passage. Posttest performance was better in the test condition than in the extended study condition in all experiments—a pretesting effect—even though only items that were not successfully retrieved on the pretest were analyzed. The testing effect appears to be attributable, in part, to the role unsuccessful tests play in enhancing future learning. Keywords: testing, learning, memory, retrieval Testing has become a central issue in the current U.S. political debate concerning education. To ensure equal access to a high- quality education, operationalized as proficiency on state academic assessments (No Child Left Behind Act, 2001), educational re- forms have replaced instruction—sometimes several weeks’ worth each year—with standardized testing in an effort to monitor stu- dents’ knowledge. These tests reduce time spent on curricula, but serve as diagnostic tools and accountability instruments, alerting teachers and administrators to low-performing student populations in need of additional services or reform. The diagnostic function of testing has merit, but there is a second benefit of testing that is often overlooked: Testing enhances memory for the tested mate- rial. Taking advantage of the memorial benefits of tests, and integrating testing into the curriculum rather than as an event that follows instruction and learning, has the potential to increase the efficiency and utility of school testing practices if this finding were better understood. A survey of naive undergraduates supports the claim that tests are viewed principally as assessments in the United States. Kornell and Bjork (2007) asked undergraduates whether they tested them- selves when they were studying, and if so, why. Whereas most students did report testing themselves (91%), most stated that they did so to “to figure out how well I have learned the information I’m studying.” Only 18% described their testing as a learning event (Kornell & Bjork, p. 222). Tests as Learning Events Research suggests that testing information that has already been studied not only provides a measure of learners’ knowledge, tests also become learning events in their own right. Indeed, testing has often been shown to be more effective than further study in encouraging retention of tested information (e.g., Bjork, 1988; Carrier & Pashler, 1992; Gates, 1917; Glover, 1989; Hogan & Kintsch, 1971; Izawa, 1970; McDaniel, Roediger, & McDermott, 2007; Roediger & Karpicke, 2006a, 2006b; Rothkopf, 1966; Tulv- ing, 1967; Whitten & Bjork, 1977; for a review, see Richland, Bjork, & Linn, 2007). Researchers studying the cognitive under- pinnings of testing have argued that testing should be considered a strategy for knowledge acquisition above and beyond its utility as a measure of current knowledge. Testing as an instrument serving larger instructional goals has traditionally been seen to have a limitation, however: The benefits of testing are most pronounced for test items that were answered correctly (Butler & Roediger, 2007; Karpicke & Roediger, 2007; Lindsey E. Richland and Liche Sean Kao, Department of Education, University of California, Irvine; Nate Kornell, Department of Psychology, Williams College, Los Angeles. The Office of Naval Research Grant N000140810186 partially sup- ported the experiments reported herein. This material is also based on work supported by the National Science Foundation under Grant 0757646. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We thank H. L. Roediger for insightful comments on the article, and Keara Osborne for invaluable assistance in running participants. Experiments 1, 2, and 4 were previously published in the proceedings of the Cognitive Science Society (Richland, Kao, & Kornell, 2008). Correspondence concerning this article should be addressed to Lindsey E. Richland, Department of Education, University of California, 2001 Berkeley Place, Irvine, CA 92697. E-mail: l.e.richland@uci.edu Journal of Experimental Psychology: Applied © 2009 American Psychological Association 2009, Vol. 15, No. 3, 243–257 1076-898X/09/$12.00 DOI: 10.1037/a0016496 243

Leeming, 2002; Roediger & Karpicke, 2006a, 2006b). Generally, items not retrieved correctly when tested see minimal, if any, benefit of testing when compared with being allowed additional study time (for exceptions, see Izawa, 1970; Kane & Anderson, 1978; Kornell, Hays, & Bjork, in press). Unsuccessful tests may even have negative consequences. Proponents of errorless learning (e.g., Guthrie, 1952; Skinner, 1958; Terrace, 1963) suggest that failing to answer a question or answering incorrectly makes future errors more likely. Furthermore, being measured alters knowledge representations, and sometimes questioning can lead to memory distortions (see Davis & Loftus, 2007; Roediger & Marsh, 2005). Thus, testing has the potential to distort knowledge, particularly for any items not recalled correctly. Providing detailed feedback after a test can ameliorate some of these challenges (e.g., Butler, Karpicke, & Roediger, 2007; Kang, McDermott, & Roediger, 2007; Metcalfe & Kornell, 2007; Pa- shler, Cepeda, Wixted, & Rohrer, 2005), but this type of feedback is burdensome and often not feasible. This is particularly true in standardized testing, when feedback is rarely individualized by question and is often available to students and teachers only after a substantial delay. Thus, for the lowest performing students, who are No Child Left Behind’s foremost priority, testing—in partic- ular, failed tests—may have little value (or worse). Can Failed Tests Improve Future learning? The current research posits that the benefits of testing may extend to items that are not answered correctly on the test, and that failure to answer test questions should not be equated with a failure to learn. Rather, five experiments were conducted to evaluate the impact of restructuring the testing environment to actually incur more failed tests. Specifically, we evaluated the benefits of testing novel science instructional content before learning. Thus, the like- lihood of failed tests was high, but we were able to extend our theory of testing to better understand whether trying and failing on test questions actually improved learners’ longer term retention of subsequently presented information. Pretests are regularly used as assessments in pre–posttest design studies with the expectation that they do not affect learning. There were some reasons, however, to expect that pretesting could en- hance learning. Many studies have demonstrated benefits of pre- training activities such as advanced organizers (see Huntley & Davies, 1976; Mayer, 1979), outlines (e.g., Snapp & Glover, 1990), and statements to activate learners’ prior knowledge sche- mas (e.g., Bransford & Johnson, 1972). Test questions have also been studied as pretraining activities, beginning with early exper- iments on the effects of integrating adjunct questions into text passages (e.g., Anderson & Biddle, 1975; Huntley & Davies, 1976; Pressley, Tanenbaum, McDaniel, & Wood, 1990; Rothkopf, 1966; Sagerman & Mayer, 1987). Adjunct questions interwoven into texts, both before and after the target information had been pro- vided, showed improved retention of information asked about in the question and, less reliably, information not asked about (see Anderson & Biddle, 1975; Mayer, 2008; Rickards, 1976). This basic pattern has held for both direct questions with basic text materials and more complex learning environments with higher level questions. For example, using “deep-level-reasoning questions” to introduce and frame interactions with an automated tutoring system, Autotutor, can greatly affect learning (e.g., Craig, Gholson, Ventura, Graesser, & the Tutoring Research Group, 2000; Craig, Sullins, Witherspoon, & Gholson, 2006; Gholson & Craig, 2006). In some circumstances, integrating these questions into instructional content can make noninteractive instruction as effective as an interactive tutor (VanLehn et al., 2007). Related research demonstrates that training “self-questioning” improves critical thinking and learners’ ability to construct knowledge from forthcoming instruction (see King, 1992, 1994). The early studies on adjunct questions, and the more recent studies with more inferential, higher level questions, did not at- tempt to contrast failures at the time of testing with successes. Rather, the most common interpretation of the questions’ effects on later retention rested on their impact on readers’ intentional learning behaviors. Rothkopf (1965, 1966, 1982) coined the term mathemagenic behaviors to explain the intentional learning behav- iors of readers that are alterable by the instructional activities they encounter. For example, Rothkopf and Bisbicos (1967) found that asking participants questions in which the answers were numbers led to better retention of all numerical information in the text, possibly because participants were able to direct their attention to the type of information that was important to learn given the test they would take. Direct tests of attention, based on measures of reading time and reaction time to a secondary task, demonstrate that people pay greater attention to reading a text when adjunct questions are interwoven (Reynolds & Anderson, 1982; Reynolds, Standiford, & Anderson, 1979). A practice guide published by the Institute of Education Sciences (an institute within the U.S. Department of Education) reviewed recent research with a similar conclusion, making the instructional recommendation: “We recommend . . . using ‘prequestions’ to activate prior knowledge and focus stu- dents’ attention on the material that will be presented in class” (Pashler et al., 2007, p. 30). In addition to affecting learners’ attention and intentional learn- ing behaviors, pretesting may provide a direct impact on memory. The cognitive benefits of testing after studying are well established to persist even when there is no opportunity to restudy information (e.g., Hogan & Kintsch, 1971; Roediger & Karpicke, 2006b), which rules out the possibility that those benefits are explainable by attention during text processing. We thus investigated whether there was a similar cognitive benefit for pretesting above and beyond the effect of drawing learners’ attention to testable infor- mation. Unlike most previous studies, Pressley et al. (1990) did distin- guish the effects of attention direction from the effects of testing itself using pretest questions. Their participants recalled more when they had been asked about the passage before reading it than when they had been presented with the same pretest questions, but had not been asked to try to answer them, before reading the passage (instead, participants were asked whether or not the ques- tions were well written). Because the questions had dichotomous answers, however, participants were frequently able to answer correctly during the pretest. The current experiments followed a similar study premise, but sought to test more directly whether unsuccessful retrieval at- tempts enhance retention of tested content beyond directing atten- tion during study. Therefore, in the current study, the prequestions required participants to produce nouns or descriptive statements that they were unlikely to be able to answer on the basis of prior 244 RICHLAND, KORNELL, AND KAO

knowledge (e.g., “What is total colorblindness caused by brain damage called?”). This allowed us to isolate and examine the effects of unsuccessful retrieval attempts. The questions tested knowledge for exact information presented in the text, rather than knowledge that would require inferential or higher level thinking. Such questions are effective (e.g., Marsh, Roediger, Bjork, & Bjork, 2007; Rickards, 1976; Rickards & Hatcher, 1977–1978; Watts & Anderson, 1971; Yost, Avila, & Vexler, 1977) but could not be used in this study. They would have prevented us from adequately controlling for the fact that test questions draw learners’ attention to testable content. Rather, we wanted to be able to create a no-pretest control condition in which we could draw participants’ attention to the same information asked about in the test questions. Typographical cuing (e.g., underlining or bolding; for reviews, see Glynn, Britton, & Tillman, 1985; Waller, 1991) is effective at drawing attention to cued items, sometimes to the exclusion of uncued items (Glynn & DiVesta, 1979). By typographically cuing participants to the aspects of the passages that would be tested, we expected to draw their attention to the key content that needed to be learned. This would allow us to distinguish between the effects of attention direction and any additional benefits of unsuccessful retrieval attempts. The Present Experiments We report four experiments that examined the learning effects of pretesting, beyond directing attention to testable information, when the questions were answered incorrectly. Theoretically, we sought to analyze the effects of attempting (but failing) to retrieve or generate test answers from memory, as distinct from partici- pants’ use of more directed search strategies while reading the text. In a fifth experiment, we further distinguished between attempting to retrieve answers to test questions and other deep processing of the pretest questions. In all experiments, participants were asked to read a scientific text about vision in an unstructured reading situation, akin to how a learner might study a textbook. In the first experiment, partici- pants were either tested prior to learning or they were given additional time to study. In Experiments 2 through 5, variations on the same procedure were used to isolate the effect of attempting to derive an answer to a question from the more mundane effect of directing attention by preexposing questions. In Experiment 2, all tested sentences were italicized in the studied text; in Experiment 3, the keyword from each tested sentence was bolded. Experiment 4 used bolded text and assessed the impact of testing versus extended study after a 1-week delay. Experiment 5 sought to differentiate between reading potential test questions and attempt- ing to answer test questions before studying. Similar to Pressley et al. (1990), we manipulated whether participants memorized the pretest questions versus produced an answer to the same questions. Experiment 1 We predicted that testing before study would enhance future recall, in spite of learners’ failure to provide successful answers to the test questions. Method Participants Participants in this study were 63 undergraduates who were given extra course credit for participating. Materials Study materials were selected from Sacks (1995). A two-page text was developed on the basis of an essay about a patient with cerebral achromatopsia (colorblindness caused by brain damage). This text was selected because of its rich scientific content and engaging narrative. The reading level was deemed appropriate for undergraduates, and Sacks’s book is assigned in undergraduate coursework. To protect against the possibility that participants had read the passage in coursework, participants were asked whether they had read the passage previously, in which case they would have been excluded. None were excluded for this reason. The length of the story was designed to ensure that participants were not under time pressure and had time to return to sections if they desired to do so. Some of the text described Sacks’s patient suffering from cere- bral achromatopsia, as in the following sample: I am a rather successful artist just past 65 years of age. On January 2nd of this year I was driving my car and was hit by a small truck on the passenger side of my vehicle. When visiting the emergency room of a local hospital, I was told I had a concussion . . . . I have visited ophthalmologists who know nothing of this color-blind business. I have visited neurologists, to no avail. Under hypnosis I still can’t distinguish colors. I have been involved in all kinds of tests. You name it. My brown dog is dark gray. Tomato juice is black. Color TV is a hodgepodge. Other parts of the text were selected from the more scientific treatment of the disorder, as in the following sample: Colorblindness, as ordinarily understood, is something one is born with—a difficulty distinguishing red and green, or other colors, or (extremely rarely) an inability to see any colors at all, due to defects in color responding cells, the cones of the retina. Total colorblindness caused by brain damage, so-called cerebral achromatopsia, though described more than three centuries ago, remains a rare and important condition. It has intrigued neurologists because, like all neural disso- lutions and destructions, it can reveal to us the mechanisms of neural construction, specifically, here, how the brain “sees” (or makes) color. (Sacks, 1995, pp. 3–4) Within the reading packet, 10 sentences were identified as testable items. Test questions were constructed on the basis of these 10 sentences. Two counterbalanced pretests were constructed such that each contained questions about 5 of the selected sen- tences. Questions were written as fill-in-the-blank or short free- response items (e.g., “What is total color blindness caused by brain damage called?” and “How does Mr. I distinguish red and green traffic lights?”). They addressed facts presented in the text, either general scientific facts or information about the specific patient. See Appendix A for all questions. A final test included all 10 of the testable items in randomized order. Thus, for participants in the test condition, 5 of the final test questions had been pretested during Time 1 (tested) and 5 had not 245 PRETESTING EFFECT

been tested previously (untested). Questions from the two pretest versions were always interspersed on the final test. All questions were new for participants in the extended study condition. Procedure The experiment was conducted in a group setting. Participants were randomly assigned to an extended study condition ( n 27) or a test and study condition ( n 36). We conducted this exper- iment in a lecture class setting, and assigned participants on the basis of seating. The lecture space had separate seating areas and participants were assigned on the basis of those. This was done to ensure that each section followed the appropriate timing, but we did not have tight control over cell size. Learning phase. Participants in the test and study condition were given one of the two counterbalanced pretests and allowed 2 min to answer the questions. They were instructed to provide an answer to all five questions, regardless of whether they knew the answer. At the end of 2 min, the pretests were collected, and participants given the text passage and told to study it for 8 min. They were instructed to read the passage through in its entirety at least once. Participants in the extended study condition were given 10 min to study the passage—the same total time that participants in the test and study condition spent in testing and study of the material. They were given the same reading instructions. Final test. The text passages were collected after the timed study periods were completed. Participants were then immediately administered the Time 2 test, which consisted of 10 questions. The test was untimed to ensure that time pressure did not affect performance. Results In the test and study condition, on the initial test that preceded the presentation of the passage, participants answered 5% of the questions correctly. Any items answered correctly on the Time 1 pretest were removed from the following analyses of Time 2 test scores on a participant-by-participant basis. Most participants gave an answer for all questions, often providing answers that were incorrect yet appropriate (e.g., writing the name of a scientist in a question referring to Isaac Newton). An independent samples t test examined the effects of testing by comparing mean posttest percentage correct for tested items in the test and study condition with the overall mean score in the ex- tended study condition. As shown in Figure 1, testing resulted in better posttest performance ( M 75%, SE 3.2) than did the provision of extra time to study the same material ( M 56%, SE 2.7), t (61) 4.26, p .0001, d 1.1. Examining performance within the test and study condition only, tested items ( M 75%, SE 3.2) were recalled on the final test significantly more often than untested items ( M 50%, SE 3.4), t (35) 5.03, p .0001, d 1.7, in spite of the fact that the analyses excluded any items that participants recalled correctly on the pretest. The benefit of testing did not spread to untested items, but neither did it hurt. There was not a significant difference between accuracy on the untested items in the test and study condition and in the extended study condition, t (61) 1.3, p .20. Discussion Experiment 1 revealed that failed tests can enhance learning for educational content. Although participants largely failed on the initial test (answering 95% of the questions incorrectly), the effect of those failures was to increase retention of studied content when compared with an extended opportunity to study the materials without being pretested. The explanation for the benefit of unsuccessful tests is not yet clear. One possibility is that the test directed learners’ attention to the key, testable points in the passage. Alternatively, attempting to retrieve an answer to the test problem may have provided an additional benefit above and beyond the impact of attention direc- tion. Experiment 2 used the same procedure as Experiment 1, but all testable sentences were italicized to equalize participants’ at- tention to key concepts in the text. We reasoned that under such conditions, allocation of attention would not differ meaningfully between conditions; therefore, differences in learning would be attributable to the impact of retrieval attempts during the pretest. Experiment 2 We predicted that pretesting followed by study would enhance future recall more than the provision of extended time to study an instructional text, even when differences in attention direction were minimized by italicizing key sentences in the text in both conditions. Method Participants The participants were 61 undergraduates (mean age 21 years, 44 women and 17 men) who were given extra course credit for participating. Participants were sampled from an upper division 0 10 20 30 40 50 60 70 80 90 100 Extended Study Test and Study Posttest Accuracy (%) Untested Items Tested Items Figure 1. Experiment 1: Performance on a final test across conditions when studying an unmarked text. 246 RICHLAND, KORNELL, AND KAO

psychology course on human stress. Data from 2 participants were excluded from analyses because of a failure to respond to final test questions. Materials The study materials were the same text and testable sentences as used in Experiment 1. The key difference was that within the reading packets, the 10 testable sentences were italicized. Italiciz- ing was considered a way to ensure that all participants were equally alerted to what was deemed to be important information in the same way that many textbooks emphasize key elements of a chapter. Participants in both conditions read the same italicized text. For example, see the following text paragraph: The history of our knowledge about the brain’s ability to represent color has followed a complex and zigzag course. Newton, in his famous prism experiment in 1666, showed that white light was com- posite—could be decomposed into, and recomposed by, all the colors of the spectrum. The rays that were bent most (“the most refrangible”) were seen as violet, the least refrangible as red, with the rest of the spectrum in between. (Sacks, 1995, p. 18) Procedure The procedure was exactly the same as the procedure in Exper- iment 1. Participants were tested in a group setting and were randomly assigned to the extended study condition ( n 26) or to the test and study condition ( n 33). Participants were not given any specific instruction regarding the text italics. Results In the test and study condition, participants answered 22% of questions on the initial pretest correctly. Correct answers were distributed across test problems. The population of participants in this experiment seems to have had a higher level of relevant background knowledge on pretest items than in Experiment 1, perhaps because they were sampled from a higher level psychol- ogy course, but as in Experiment 1, any items answered correctly at Time 1 were removed from the following analyses on a participant-by-participant basis. If anything, this led to inflation in participants’ scores in the untested conditions, counter to our hypothesis. The data revealed benefits for testing over the provision of extra time for studying the same material. As Figure 2 shows, recall of tested and italicized items in the test and study condition ( M 71%, SE 5.6) was significantly greater than recall of italicized- only items in the extended study condition ( M 54%, SE 3.7), t (57) 2.3, p .022, d 0.61. Examining performance within the test and study condition, tested items were recalled on the final test ( M 71%, SE 5.6) significantly more often than untested, italicized-only items ( M 53%, SE 4.3), t (32) 3.27, p .003, d 0.63, in spite of the fact that the analyses excluded items that participants recalled correctly on the pretest. Testing did not appear to negatively affect the untested items; there was not a significant difference between accuracy on the italicized-only items in the test and study condi- tion and the italicized-only items in the extended study condition, t (57) 0.15, p .88. Discussion The results of Experiment 2 replicated the results from Exper- iment 1, and again suggest that the testing effect can and should be extended to failed tests. Testing items created more potent learning opportunities than extended study of the same items, even when the key information in both conditions was italicized, equalizing attention direction. Thus, testing appears to provide a unique benefit above and beyond directing learners’ attention to content that has a high probability of being tested later. In textbooks, italicized sentences are less common than bolded keywords, which are ubiquitous. It remains possible that partici- pants in Experiment 2 were unfamiliar with the meaning of italics within text, and thus differences in attention were not minimized. To rule out that possibility, Experiment 3 used the same procedure as Experiment 2, but bolded keywords were used instead of ital- icized sentences because we expected that bolding might act as a stronger (and more realistic) attention prompt. Experiment 3 thus examined the impact of testing when compared with extended opportunities to study text in which the key test items were bolded. Experiment 3 We predicted, similar to Experiment 2, that testing before read- ing would enhance future recall above and beyond the impact of extended study time. Instead of presenting key sentences in italics, keywords were presented in bold. Method Participants Participants in this study were 64 undergraduates (44 women, 17 men, 3 unstated) who were given extra credit in their courses for participating. Participants’ average age was 22 years. 0 10 20 30 40 50 60 70 80 90 100 Extended Study Posttest Accuracy (%) Untested/Italicized Items Tested/Italicized Items Test and Study Figure 2. Experiment 2: Performance on a final test across conditions when studying text with italicized key sentences. 247 PRETESTING EFFECT