CC BY-ND
35
https://doi.org/10.17323/jle.2022.13380
Accuracy Gains from Unfocused Feedback: Dynamic Written Corrective Feedback as Meaningful Pedagogy
Kendon Kurzer ®
University of California, Davis
ABSTRACT
Background: A primary question among L2 writing instructors is how to best deliver written corrective feedback (WCF) to support student learning. One promising WCF method is Dynamic Written Corrective Feedback, in which instructors provide unfocused/comprehensive feedback using a coding system coupled with regular rounds of editing on short, in-class student-written Purpose: While research generally indicates that unfocused WCF may not be the most effective method of supporting linguistic mastery, unfocused WCF that is delivered using a coding system and in manageable portions may result in meaningful uptake of target linguistic forms; however, further research on best practices to deliver WCF is needed. In this study, I explored the impact on student accuracy of unfocused DWCF on brief student-produced texts in intermediate and advanced developmental ESL writing classes.
Methods: Utilizing a quasi-experimental research design using t-test analyses, I coded, tallied, and contrasted the errors in term-final paragraphs of 130 students who participated in classes that used DWCF with 79 students in control sections that did not include DWCF. Results: I found statistically significant improvements in the treatment sections at both levels for nearly all error types (including but not limited to verb form/tense, sentence structure, word order, word choice, determiner, noun form, and punctuation errors; the only error type that did not return significance differences was prepositions at the intermediate level). Conclusion. These results suggest that unfocused written corrective feedback may be effectively used in multilingual writing classrooms, at least given certain parameters to help ensure that feedback is manageable and specific, per the DWCF process.
KEYWORDS
dynamic written corrective feedback, developmental writing instruction, comprehensive grammar feedback, unfocused feedback, L2 writing
INTRODUCTION
Citation: Kurzer K. (2022). Accuracy Gains from Unfocused Feedback: Dynamic Written Corrective Feedback as Meaningful Pedagogy. Journal of Language and Education, 8(4), 102-116. https://doi.org/10.17323/jle.2022.13380
Correspondence:
Kendon Kurzer, [email protected]
Received: November 19, 2021 Accepted: March 31, 2022 Published: December 26, 2022
An extensive corpus of research on written corrective feedback (WCF) spanning decades has provided extensive support indicating that WCF can be used to foster increased accuracy, at least in certain contexts and with certain students (Ferris & Kurzer, 2019), despite some concerns about its efficacy and appropriateness (e.g. Bruton, 2009; Truscott, 1996). At this point, a primary question about WCF has become not one of if instructors should provide WCF, but rather how to most effectively deliver WCF to support meaningful student learning (Ferris & Kurzer, 2019). One specific WCF method that matches many of the best practices
identified by WCF research and that has a growing body of supportive research is Dynamic Written Corrective Feedback (DWCF) (Evans et al., 2010).
Teachers can use DWCF to "help L2 learners improve the accuracy of writing by ensuring that instruction, practice, and feedback are manageable, meaningful, timely, and constant" (Hartshorn & Evans, 2012, p. 30) for all involved, by keeping feedback approaches manageable for the instructors and accessible/comprehensible for the students, per this process:
1. An instructor requires students to write short paragraphs (for roughly
10 minutes) during each class period (or at established times throughout the term, roughly equally spaced).
2. The instructor then codes the errors found in the paragraphs using a series of codes that is explicitly explained in class (Appendix A contains the coding system used in this study) and returns the coded first drafts to the students during the next class meeting.
3. The instructor has the students edit these first drafts in class and submit the second drafts to their teacher for further coding of any remaining or new errors. As originally devised, the entire editing process is repeated until the draft is completely error free (resulting in perhaps four or five drafts written for a single round of DWCF on occasion).
4. The instructor also has students record a tally of all present types of errors in a log (Appendix B); this error log allows the students and the teacher to track students' individual error patterns, which may promote increased autonomy (Ferris, 2006; Lalande, 1982).
DWCF as originally developed is accordingly focused exclusively on matters of linguistic/grammatical accuracy, rather than other concerns such as idea development or organization.
DWCF and Second Language Acquisition Theories
Briefly, DWCF may be grounded in various established second language acquisition theories, as is WCF generally. For instance, language learners first develop declarative knowledge (what they actually know) prior to procedural knowledge (application of that declarative knowledge in real-world contexts) (DeKeyser, 2001, 2007). Via DWCF, students can obtain this procedural knowledge from the initial coding stage and in-class instruction and then develop procedural knowledge by automatizing target grammatical features in their L2 via practice from the extensive editing process.
DWCF can also help instructors connect with their students' Zone of Proximal Development and promote internalization (Vygotsky, 1978) and transferability of grammatical concepts. DWCF can be an effective method of scaffolding grammar feedback (Wood et al., 1976) while maintaining comprehensible input—or /'+1 (Krashen, 1985). Corrective feedback like DWCF has been explicitly promoted as an extension of Krashen's Input Hypothesis as it may facilitate language acquisition for some linguistic features (Long, 1996).
DWCF and Established Best Practices of Written Corrective Feedback
As the primary aim of DWCF is to provide feedback that is "manageable, meaningful, timely, and constant" (Hartshorn & Evans, 2012, p. 30), DWCF aligns with the established WCF research literature on WCF in a number of manners.
DWCF is a type of /nd/rect WCF since the existing errors are coded but not corrected (when a correction is provided, it is d/rect WCF); indirect WCF may promote more meaningful long-term acquisition of linguistic features such as grammar mastery (Ferris, 2006) as it likely results in increased internalization (Kurzer, 2018a; Lalande, 1982). Similarly, a coding system like DWCF that utilizes expl/c/t codes may trigger previous grammar knowledge of students when compared to unlabeled WCF (Bitchener, 2008; Bitchener & Knoch, 2010; Ferris, 2006; Sheen, 2007). DWCF codes may remind multilingual students of prior instruction and connect that declarative knowledge (DeKeyser, 2001) to their produced writing in the target language. Explicit WCF also tends to be appreciated by many multilingual students when compared to unlabeled WCF (Lee, 2005). Additionally, DWCF may result in improvements among grammatical concepts that feature idiosyncratic rules (Hartshorn & Evans, 2012)—untreatable errors that are difficult to teach. Some of these untreatable features, like "word order, sentence boundaries, phrase construction, word choice, or collocations" may "obscure meaning" (Ferris, 2010, p. 193) despite being challenging to teach. DWCF may be an effective intervention for those kinds of features in particular, perhaps due to increased focused practice using the target language.
Most salient for the purposes of this special issue, WCF research tends to advocate for focused WCF that prioritizes only a single or a narrow range of error types compared to unfocused WCF on all types of grammatical errors, as the research indicates increased levels of accuracy when WCF is focused on a single or small number of error types (Bitchener, 2008; Ellis et al., 2008; Sheen, 2007). When we consider that many studies investigated only a single grammatical feature like articles/determiners to identify increases in accuracy (e.g. Bitchener & Knoch, 2010; Ellis et al., 2008), focused WCF seems pedagogically sound. However, this ignores the ecological reality of many classrooms in which instructors do not likely limit grammatical feedback to only a single feature (Ferris & Kurzer, 2019). While unfocused WCF may seem overwhelming to students, the explicit codes on short pieces of student writing and rapid editing approach of DWCF may meaningfully scaffold student learning (Hartshorn & Evans, 2012; Kurzer, 2018a).
Empirical Research on Dynamic Written Corrective Feedback
While DWCF has been featured in a number of prominent publications on WCF in recent years, only 11 articles have presented the results of empirical research specifically on the intervention as of 2021. Table 1 contains an overview of these articles.
The first studies conducted on DWCF identified improvements regarding general linguistic accuracy attributable to the DWCF treatment in an Intensive English Program (IEP) connected to a research university in the Western United
Table 1
Summary of Previous Research on DWCF
Study Control Large N (>30) Context Longitudinal
Evans et al., 2010 No No IEP No
Hartshorn et al., 2010 Yes No IEP No
Evans et al., 2011 Yes No Undergrad No
Hartshorn & Evans, 2012 Yes No IEP No
Marzban & Arabahmadi, 2013 Yes No ?* No
Hartshorn & Evans, 2015 Yes No IEP Yes
Kurzer, 2018a Yes Yes Undergrad No
Kurzer, 2018b Yes Yes Undergrad No
Kurzer, 2019 No No CC No
Eckstein et al., 2020 No No Grad No
Messenger et al., 2020 No No IEP Instructor No
*The context of this study was unclear to me, with the authors indicating only that the study was of "two intact intermediate classes at the private
institute in Iran" (p. 1001).
Table 1 is also seen in (Kurzer, in print).
States (Evans et al., 2010; Hartshorn & Evans, 2015; Hartshorn et al., 2010) and in elective credit-bearing language support classes for matriculated multilingual students at that same university (Evans et al., 2011). Another study in this IEP context found that DWCF resulted in improvements on these linguistic/grammatical features: lexical, verb, semantic, and mechanical accuracy, determiners, numeric agreement, and sentence structures (Hartshorn & Evans, 2012). One longitudinal study conducted thus far about DWCF has tracked students across two semesters, finding meaningful gains in accuracy (Hartshorn & Evans, 2015). These initial studies all reported medium to large effect sizes attributable to DWCF, although they featured small sample sizes (fewer than 30 student participants).
In a larger study with 325 students from three levels of credit-bearing pre-first-year composition developmental courses, I explored improvements in accuracy more broadly, looking at error categories of global, local (per Bates et al., 1993), and mechanical, finding statistically significant gains in the treatment sections which used DWCF for each category and at each level (Kurzer, 2018a). Students in the treatment sections also demonstrated statistically significant improvements in self-editing skills compared to students in treatment sections; this provides evidence that DWCF can indeed help L2 students develop stronger autonomy (Ferris, 2006; Lalande, 1982).
A final article—the only research study investigating graduate students specifically—reported on a study that contrasted the timing of DWCF (spread throughout the term or
lumped together at the end of the term) with 22 multilingual graduate students, finding gains in fluency and complexity in the group that received regular feedback, but no statistically significant improvements regarding grammatical accuracy in either group (Eckstein et al., 2020). These results may identify a ceiling level upon which DWCF may no longer be effective at supporting increased grammatical accuracy in student writing, although it may still be helpful at influencing student writing in other manners.
These studies suggest that DWCF can help many students—in IEP, developmental, and first year composition contexts—produce more accurate writing, at least in short student-produced paragraphs. However, an improvement in accuracy is simply a single aspect of a WCF-based pedagogical intervention to determine if it is meaningful for students' language learning; student and instructor opinions about the target pedagogies should also be considered. In an expansion of the study I explained previously, I investigated the impact of DWCF on students' efficacy regarding writing, finding that treatment students who used DWCF rated the value of peer feedback, quality of grammar feedback, and quality of general class instruction statistically more strongly than the control section students did, although the differences did not include their perceptions of their own grammar abilities (Kurzer, 2018b). Students who used DWCF also ranked it highly in terms of classroom interventions they valued, a finding replicated in a small action research study I conducted in an intermediate L2 writing class I taught at a community college (Kurzer, 2019). Students in that study reported that they found DWCF to be better at matching
their current levels of language master than the course grammar textbook, although they also valued the textbook as a resource, reinforcing the idea that students may best respond to DWCF as a pedagogy that augments traditional grammar instruction, rather than replacing it completely. In addition to students responding positively to the DWCF treatment, one study has investigated five experienced ESL teachers' perspectives via interviews, who reported it to be a promising pedagogy to promote meaningful uptake of grammar although they also offered some suggestions to keep it manageable (Messenger et al., 2020).
Taken collectively, the growing body of research on DWCF paints a promising picture of a pedagogical intervention that is well-received by students and instructors and which has resulted in measurable increases in accuracy, at least in certain circumstances. The research on DWCF also adds nuance to the research on unfocused feedback, indicating that more sound classroom practices like providing unfocused/comprehensive feedback still may prove to be valid. However, while some of the initial studies conducted about DWCF explored the impact on specific error types in an IEP, we have yet to see those results replicated in other contexts or with larger sample sizes. In order to begin to fill this gap, I conducted this study in which I hypothesized that matriculated multilingual students in developmental writing classes that included DWCF to augment traditional composition and grammar instruction would show an increased level of grammatical accuracy across some or most error types on final paragraphs than students who did not use DWCF in their classes.
METHODS
Data collected as a part of this IRB-approved study have contributed to research reports published elsewhere exploring possible impacts of DWCF on self-editing skills and accuracy improvements in global, local, and mechanical error categories broadly defined (Kurzer 2018a) and student perceptions of DWCF as a classroom intervention (Kurzer 2018b). Accordingly, the methodologies of these various reports are quite similar in nature. Also of note, these data were collected from in-person classes that occurred prior to the global coronavirus pandemic.
Research Design and Context
In this study, I explored the impact of unfocused DWCF in credit-bearing developmental writing classes for multilingual domestic and international students using a quasi-experimental study design. All students enrolling as freshmen at this institution take a brief timed writing exam; based on their needs as revealed from their score on this test, many students are then placed into up to three-levels of credit-bearing developmental writing classes with a focus on academic composition processes via several out-of-class pa-
pers commonly emphasized in the US, particularly on using sources. These matriculated students must pass through the developmental in a timely manner prior to enrolling in first year composition, typically while also taking foundational courses in their selected majors.
For this quasi-experimental study, I collected and contrasted student data from existing sections of several intermediate and advanced developmental writing. While the courses primarily focused on composition, teachers of control sections secondarily employed traditional grammar instruction (a grammar book with exercises and lesson units on specific grammatical features) and teachers of treatment sections also employed an adaptation of DWCF (as explained in the introduction of this article) alongside the traditional grammar instruction. As with other small assignments designed to scaffold writing/language instruction, students earned points for engaging with grammar activities/DWCF.
To keep the study ecologically valid to this real-world classroom context, instructors taught grammar per their own best practices, while adhering to departmental guidelines, with corrective feedback beyond DWCF being limited to correction on grammar exercises and any feedback on out-of-class papers the instructors deemed appropriate to provide as typically done in these developmental writing classes. In the developmental writing program in my institution, the primary focus is on out-of-class essays with a secondary linguistic/grammar focus. This is in contrast to the original studies on DWCF conducted in an IEP in which DWCF replaced all instruction in grammar classes.
When instructors were teaching two or more sections of the same class, I had them teach one of those sections using DWCF and one without to help reduce variability in teaching practices. Teachers who taught one or more treatment sections over the course of the study (which was conducted over two consecutive 10-week terms) participated in a professional development session on how to use DWCF effectively in which I explained the DWCF process (as I outlined earlier).
DWCF was originally designed to replace grammar instruction, so instructors had students write new paragraphs or revisions daily in class, something instructors in our program felt to be prohibitive given the primary composition focus of these classes. Accordingly, we adjusted the rounds of DWCF required, based on level (ten at the intermediate level and five at the advanced level, meaning that students had to write ten/five initial paragraphs and all required edits for each, spread out roughly equally throughout the term). To reduce variability in instruction approaches, I provided a list of ten prompts designed to solicit specific grammatical features for the instructors of intermediate classes to use (Appendix A), although instructors could determine the order in which to assign the prompts as long as they covered all ten. Instructors of advanced classes were required
to include at least five rounds of DWCF, although the specific topics were left to them to determine, to best adapt their instruction to their more proficient students' needs. Instructors of control sections did not include DWCF in their classes.
Participants
Table 2 contains an overview of the numbers of class sections, teachers, and student participants by intermediate/ advanced class level and treatment/control group.
About 80% of the student participants were international (studying in the US on a student visa) while the remaining 20% of the student participants were Generation 1.5 students (late- or early-arrival immigrants)(Ferris, 2009). The international students primarily were Chinese (roughly 90%), with the remaining being from South Korea, Saudi Arabia, Mexico, and Japan, while Generation 1.5 students were less homogenous, although roughly 85% were from Spanish- or Chinese-speaking backgrounds. I collected data from about one third of all intermediate and advanced developmental writing classes offered at the time which resulted in some students moving between control and treatment groups as they progressed through the program and/or enrolled in courses that were not ultimately included in the study.
Following the guidelines of DWCF, I required students from the treatment and control groups to write paragraphs (the diagnostics in Appendix C) at the start of the two terms of the study for pre-test data allowing a comparison of the control and treatment groups. 7"-tests comparing the means of the two groups revealed no significant differences for all measured variables (numbers of errors of all error types per 100 words outlined in the next section and word counts) at both the intermediate and advanced levels. Because of this lack of statistically significant difference, the control and treatment groups at both levels could be properly compared for this research study.
had been teaching ESL classes for several years. Beyond two instructors who were new to our program and started out teaching control sections, I randomly assigned the other instructors to be either treatment or control sections and had instructors who were teaching two sections of the same class teach one using DWCF and one without (to reduce variability in teaching approaches/instructor bias as much as possible); sections were accordingly quite comparable in instruction beyond the absence or inclusion of DWCF. Prior to the start of the study, I provided a professional development session on how to use DWCF to all the instructors of treatment sections to ensure that the approaches, coding systems, and numbers of DWCF rounds were standardized. Student paragraphs written across the terms and collected and analyzed in Kurzer (2018a) reveal that instructors successfully adhered to the parameters of DWCF and assigned all required rounds of DWCF, with paragraphs written, coded, and edited regularly throughout the terms as expected.
Materials and Coding System
The coding system used in these classes was one I adapted from the original DWCF coding system (Evans et al., 2010) based on feedback from instructors in our program. While the original system had 20 codes, I combined some to reduce redundancy based on the needs of our more advanced student population compared to the students in the IEP of the original study, resulting in 16 codes that still captured the range of student errors frequently present in student writing in our program (Appendix A). In order to stress to the instructors and students in this program which types of errors should be emphasized as being essential for meaningful communication, I grouped these by error type: global errors that are more likely to impede easy comprehension, local errors that may be distracting but do not impede comprehension (Bates et al., 1993; Lane & Lange, 2012), and mechanical (punctuation, spelling issues, or missing/extra words), as follows in Table 3:
I recruited instructors from the group of lecturers scheduled to teach in the ESL developmental writing program during the academic year I conducted the study. Instructors in the program held a terminal degree (either a master's or PhD), typically in TESOL or a composition-related field, and
Table 2
Study Participants
While the coding system was unfocused and captured all main error types seen in L2 writing, grouping the codes helped prioritize the importance of mastery of grammatical features that contribute to communication first (Bates et al., 1993; Bitchener & Ferris, 2012), rather than local errors that
Intermediate
Advanced
Treatment
Control
Treatment
Control
# of Sections
# of Teachers
# of Students
# of DWCF Rounds
4 4
66 10
2 2
31
4
3
64
5
3 2
48
are frequently prioritized by many studies on written corrective feedback but typically do not actively impede comprehension (Ferris, 2010).
Table 3
L/st of Error Codes Global Errors
Verb Form Verb Tense/Time Sentence Structure Word Order Word Choice Local Errors
Prepositions Determiners Noun Forms Word Forms Mechanical Errors Spelling Punctuation Capitalization Insert Omit
Unclear Meaning
_Awkward Wording_
Instructors also required their students to record error type and frequency in an error log (Appendix B), as done in the original DWCF approach. However, as instructors felt that requiring the students to edit their drafts until no errors at all remained would be unrealistic and demotivating for the students, we elected to adjust the acceptability threshold at which students would no longer need to edit. Ultimately, we decided that students would no longer need to edit if they had three or fewer global errors remaining (without requirements for local or mechanical errors) as the emphasis on global errors would better serve our student population by prioritizing comprehensibility rather than native-speaker-like accuracy.
Data Collection and Statistical Analysis Procedures
To explore my hypothesis regarding any possible improvements regarding specific error types from DWCF, at the end of the 10-week term teachers of both the control and treatment sections instructed their students to write a 10-minute paragraph that possibly could elicit all target grammatical features formally instructed on in these developmental writing classes (Appendix C includes this post-test prompt). This approach mitigated the concerns about research on WCF which has largely relied on student-edited language production, rather than entirely new, original student language (e.g. Truscott, 2007). I collected and anonymized these par-
agraphs and coded all errors per the coding system (Appendix A) as a collective whole, before sorting them into treatment/control groups to reduce the chance of personal bias. I then tallied the errors and calculated error numbers per 100 words as a measure of standardization. The errors per 100 words numbers for each error type were then compared across treatment and control groups using t-tests. The level of significance of these analyses is presented at both a standard p<.05 and a more conservative p <.003 (calculated per a Bonferroni correction, which is used to protect against Type 1 Error when conducting multiple analyses on a single dependent variable as in this case).
Other measures to evaluate writing accuracy such as error-free 7-units (Wolfe-Quintero et al., 1998) or error-free clause ratios (Wigglesworth, 2008) have been used in DWCF studies, although they may conflate fluency with accuracy, may be overly simplistic (Larson-Freeman, 2009), or may not be more valid than other more straight-forward measures, such as errors per total words (Polio & Shea, 2014). Given the lack of tangible benefit of these other measures, I elected to use the errors/100 words metric commonly employed in other WCF research (e.g. Chandler, 2003; Truscott & Hsu, 2008). An additional benefit of this metric is that I was able to tally all errors independently of each other in cases of multiple errors in a single 7"-unit or clause. As a result, in any instances when errors overlapped (i.e. when a preposition error was identified within a sentence structure error), I coded and tallied them as distinct errors.
I coded and tallied the errors in the post-test student-produced paragraphs myself; to help ensure reliability in the coded data, a subset of the paragraphs was also coded by a different teacher from the program who had been teaching using DWCF herself for several years but was not participating in the study in either a control or treatment capacity. Pearson's r inter-rater reliability agreement estimates were reasonably high for these data (at .82 for all error types), especially given that many grammatical errors could have multiple possible conflating codes/corrections.
RESULTS
To determine the impact of DWCF on student writing at the intermediate and advanced levels, I contrasted the number of errors of each type in paragraphs treatment and control students wrote at the end of the term using t-tests. For the complete intermediate level t-test results and effect sizes identified from each variable, see Table 4.
As seen in Table 4, the students in the treatment sections of intermediate developmental writing courses produced statistically more accurate term-end paragraphs than their peers in the control sections for all global, local, and mechanical error types except prepositions at p<.05. Even utilizing a much more conservative significance threshold of p
Table 4
T-test Results for All Error Types per 100 Words at the Intermediate Level
Variable Treatment 1 Control2 t P Cohen's d
M SD M SD
Verb Form .48 .43 1.11 .153 -3.17 <.001 .56
Verb Tense .38 .39 1.06 .99 -3.96 <.001 .904
Sentence Structure .54 .44 1.26 .85 -4.32 <.001 1.06
Word Order .07 .05 .22 .29 -1.85 .03 .72
Word Choice .37 .3 1.37 1.68 -5.23 <.001 .829
Preposition .69 2.11 .81 .76 -.407 .34 .076
Determiner .43 .49 1.88 2.93 -5.74 <.001 .69
Noun Form .46 .62 2.26 4.03 -6.15 <.001 .624
Word Form .23 .21 .78 .79 -3.9 <.001 .951
Spelling .18 .2 .62 1.25 -2.71 .004 .492
Punctuation .21 .18 .56 .37 -3.16 .001 1.2
Capitalization .01 .49 .11 .16 -1.84 .035 .274
Insert .43 .37 .99 .78 -3.42 <.001 .917
Omit .48 .41 .94 .63 -2.88 .002 .865
Unclear .25 .23 .61 .66 -2.57 .006 .728
Awkward .32 .26 .69 .46 -2.79 .003 .99
1 Treatment n=66
2 Control n=31
<.003 from the Bonferroni correction, the majority of the error type differences were statistically significant (except for word order, spelling, capitalization, and unclear). Most error types also featured effect sizes that were moderate or large (greater than .5), apart from prepositions and capitalization.
I repeated this analysis with the final paragraphs written by students in treatment and control sections of the advanced developmental writing class, conducting t-tests on each error type (Table 5).
As seen in Table 5, the students in the treatment sections of advanced developmental writing courses produced statistically more accurate term-end paragraphs than their peers in the control sections for all global, local, and mechanical error types, including prepositions at p<.05. At the more conservative p-value threshold calculated per a Bonferroni correction (p<.003), differences of all error types except capitalization remain statistically significant. All error types also featured effect sizes that were moderate or large (greater than .5). While I cannot unequivocally connect the DWCF treatment to the gains in accuracy seen in the treatment students' final paragraphs, due to the statistically significant differences noted in the t-tests, DWCF seems to contribute
to improvements in grammar usage among this student population.
DISCUSSION
The results of this study included statistically significant differences in most global, local, and mechanical error types in final paragraphs of intermediate and advanced treatment and control groups, even when performing a Bonferroni correction to determine a more conservative threshold of significance. The moderate to large effect sizes I identified support unfocused feedback via the DWCF process as a meaningful pedagogical intervention to improve accuracy, at least in short, timed writing contexts.
The findings of this study strengthen previous research (Hartshorn & Evans, 2012) supporting DWCF's possible role in facilitating untreatable grammatical errors as I found statistically significant improvements among all untreatable grammatical concepts except for prepositions at the intermediate level. Hartshorn and Evans (2012) identified statistically significant improvements in lexical, verb, mechanical, and semantic accuracy, sentence structure use, numeric
Table 5
7-test Results for All Error Types per 100 Words at the Advanced Level
Variable Treatment1 Control2 t P Cohen's d
M SD M SD
Verb Form .16 .12 1.21 2.61 -5.02 <.001 .568
Verb Tense .09 .06 1.55 2.99 -6.66 <.001 .69
Sentence Structure .08 .07 1.61 2.06 -8.32 <.001 1.05
Word Order .01 .01 .2 .23 -3.06 .001 1.17
Word Choice .06 .04 1.65 2.83 -7.52 <.001 .794
Preposition .22 .15 .85 .9 -4.79 <.001 .976
Determiner .26 .21 2.02 3.82 -6.97 <.001 .651
Noun Form .5 .33 1.88 2.67 -6.25 <.001 .725
Word Form .17 .16 .85 1.19 -4.61 <.001 .801
Spelling .19 .2 .72 1.26 -3.47 <.001 .588
Punctuation .08 .07 .68 1.58 -3.69 <.001 .537
Capitalization 0 0 .19 .47 -2.16 .016 .572
Insert .16 .09 1.24 1.9 -6.08 <.001 .803
Omit .23 .27 .91 1.11 -4.43 <.001 .842
Unclear .03 .02 .7 1 -5.22 <.001 .947
Awkward .15 .12 .71 .7 -4.87 <.001 1.12
1 Treatment n=64
2 Control n=48
agreement, and determiners. It is interesting to note that prepositions in particular may be difficult for students to master in DWCF settings, as significant improvements in prepositions were absent in the IEP (Hartshorn & Evans, 2012) and intermediate groups of this study, although the advanced students did have statistically significant gains in prepositions. Beyond prepositions, the medium to large effect sizes in the remaining 15 error types measured (all 16 for the advanced students) further suggest that DWCF can be effective at helping students produce more accurate writing, at least in short in-class paragraphs.
The practice of self-editing writing utilizing scaffolding via coded errors as in DWCF may result in stronger self-awareness of untreatable grammatical concepts. This suggests that, rather than exclusively relying on instruction using grammar books and worksheets to give students exposure to and practice using untreatable grammatical concepts, practice using those concepts in their own writing - and then self-editing scaffolded coded errors - may prove to be more effective.
When connected to the studies that have shown that some students value DWCF (Kurzer, 2018b, 2019) and other stud-
ies that have shown that many multilingual students value and expect grammatical support via WCF (Bates, Lane, & Lange, 1993; Ferris et al., 2013; Han & Hyland, 2015), DWCF may align with students' expectations for language classes. Some instructors and researchers are rightly concerned that a strong emphasis on form and corrective feedback in the target language may overly stress dominant narratives/ideologies and trivialize students' home languages, creating an environment toxic to socially progressive pedagogies (Loza, in print); I argue that, provided that DWCF is implemented to augment other pedagogies and that communication (rather than an arbitrary notion of correctness or "native-like fluency") is emphasized when introducing DWCF to students, there is ample room within a socially progressive classroom for DWCF. Instructors should stress that accuracy is but one facet of language acquisition and take steps to avoid instilling/reinforcing unrealistic expectations of fluency or biased ideologies (Kurzer, 2021).
While early studies on DWCF in an IEP used DWCF to completely replace grammar instruction (Evans et al., 2010; Hartshorn & Evans, 2015; Hartshorn et al., 2010), my own research explored its impact in developmental writing courses as a supplementary pedagogy to augment composition in-
struction. A previous analysis of the data currently presented broadly identified improvements at all levels in global, local, and mechanical error categories and stressed communication over strict overall accuracy (Kurzer, 2018a) but did not include a fine-grained analysis like the current study did. This narrower analysis of specific error types reinforced DWCF's appropriate role to augment—but not completely replace—language/grammar support in writing classes, reinforced by previous studies that included students' positive perceptions of DWCF in their classes (Kurzer, 2018b; Kurzer, 2019). This current study also found statistically significant improvements in accuracy in the two levels despite differences in frequency of rounds of DWCF. DWCF appears to be effective with far fewer rounds of DWCF in classes than originally implemented, depending on students' language proficiency. I also identified statistically significant improvements with students only editing their work two or three times to a threshold of no more than three global errors, rather than completely eliminating all errors. This further indicates that the DWCF process can be adapted to varied student needs.
Finally, we have yet to see adequate published literature identifying a ceiling effect at which DWCF is no longer effective. While designing the study, I questioned whether I would see improvements at the advanced level since I required fewer rounds of DWCF and therefore students did not self-edit as frequently as in other contexts. However, the statistically significant differences revealed improvements even in that context.
More broadly, some researchers have highlighted concerns with WCF research that have identified improvements in narrow grammatical features but neglected to consider possible negative impacts of focusing on only a single or narrow subset of grammatical error types on accuracy elsewhere (Xu, 2009). As I identified statistically significant improvements in accuracy across nearly all grammatical types, this study helps counter that valid concern. Similarly, the gains in accuracy I noted were in new student writing, not simply edited drafts, a limitation of other WCF research as noted by Truscott (2007). These results also support the common assumption that coded/explicit corrective feedback is a meaningful pedagogical practice (Bitchener, 2008; Bitchener & Knoch, 2010; Ferris, 2006).
Additionally, studies investigating unfocused/comprehensive feedback in non-DWCF contexts have largely presented contradictory results thus far (Bitchener, 2019), with only two comparing unfocused and focused specifically (Ellis et al., 2008; Sheen et al., 2009); even these studies had methodological limitations that prevented adequate comparison of the two to produce "unequivocal finding[s]" (Bitchener, 2019, p. 97). While my study did not compare focused and unfocused/DWCF feedback, it does provide one more point of evidence supporting that unfocused feedback, providing that it is given in manageable manners that are accessible
for the students, may be effective. While unfocused WCF addressing all possible grammatical concepts—when added to feedback on other concerns such as idea development, rhetoric mastery, and organization as students are likely to see in composition classes—on larger papers may be quite overwhelming to students and likely does not adhere to /'+1 precepts, if grammatical WCF is conducted only on shorter paragraphs as done in DWCF, unfocused WCF still seems to be comprehensible for students, at least in certain contexts. A separation between grammatical WCF assignments and larger scale feedback on other assignments (likely out-of-class process papers) accordingly seems prudent to keep workloads reasonable.
CONCLUSIONS, LIMITATIONS, AND RECOMMENDATIONS FOR ADDITIONAL RESEARCH
The results of this study complicate the literature and assumptions regarding unfocused corrective feedback and provides evidence that the DWCF process (using a coding system to code errors in manageable student-produced paragraphs with deliberate editing and error recording stages) can be used to promote increased grammatical accuracy alongside a composition-focused curriculum in developmental writing classes for multilingual students. By including data from a large number of participants (130 treatment students and 79 control students) across two levels, this study also burgeons the limited research base on DWCF specifically.
While the metric of using newly written authentic student-produced writing samples at the end of the terms of the study and the relatively large sample size invested in an ecologically valid context helped counter some of the limitations seen in other studies of DWCF and WCF, this study still has some limitations. First, while the primary point of data collection—the term-end paragraphs—were authentically produced by the students, they still were stripped of context because the topics were not related to their other writing assignments for their classes, an adaptation of DWCF I have since encouraged. Stronger connection between DWCF topics and course themes (i.e. using DWCF drafts for brainstorming, reading reflections, or dialogue journaling) may be pedagogically sound. The paragraphs were doubtless an improvement compared to grammar book work with limited application to authentic writing or grammar cloze exercises, but still lacked the importance of the main papers students were expected to write for these composition classes. In this study, I did not explore the possible impact of DWCF on students' process writing/papers written outside of the classroom, which is arguably a more important context to consider, and is a limitation of the larger body of research on DWCF and WCF in general.
Second, as the instructors of the courses used in this study volunteered to participate, the results could have been potentially biased. That said, the treatment and control groups shifted over the course of the study as some teachers moved from treatment to control and vice versa and, when instructors taught more than one section, I had them teach one control and one treatment section as possible. Accordingly, this potential bias was likely reduced. However, one teacher—who was hired at the time the study started -taught control sections for the two consecutive terms of the study; her students submitted term—final paragraphs that were slightly shorter than seen in the other groups. As the word count differences were not statistically significant, this instructor's potential effect on the outcomes of this study was likely minimal.
Additionally, I collected data at the different levels of developmental writing during consecutive terms using only approximately a third of the total offered sections in the program; thus some students enrolled in sections that may or may not have included DWCF but did not feature data collection prior to or after enrolling in treatment/control sections. Students also switched between control and treatment sections as I could not dictate enrollment. Additional possible lurking variables beyond the scope of the study include the following: students may have gained increased practice at editing their own writing, developed stronger language skills stemming from lengthier exposure to English-dominant contexts, and/or simply developed stronger writing skills as a result of taking several intensive writing courses in consecutive terms.
Further research into unfocused feedback and DWCF specifically addressing longitudinal impact (beyond Hartshorn & Evans, 2015) via delayed post-tests in subsequent terms throughout students' undergraduate experiences would provide a much stronger sense of how effective DWCF may be. Additional research looking at transferability of language accuracy to out-of-class process papers would contribute meaningfully to the literature base on DWCF. We also need
more studies on DWCF in different contexts (other IEPs, first year composition, graduate writing support courses, writing courses in specific disciplines, etc.) and using different approaches (varying the frequency of DWCF rounds/coding systems, length of student texts, using DWCF alongside anti-racist pedagogies, etc.) to develop a more robust research base upon which to ascertain DWCF best practices for different contexts and student populations. Researching and identifying valid adaptations of DWCF that adhere to the ecologies of different classroom contexts and student populations would provide L2 writing/grammar instructors with a stronger set of pedagogies to help promote meaningful but manageable grammar acquisition for their students.
Along with this study, the growing body of research on DWCF indicates that it is a method of providing corrective feedback using a comprehensive/unfocused coding system to target all common student errors. As DWCF emphasizes indirect feedback that places the responsibility for practicing and acquiring the target features on the multilingual students themselves while still providing meaningful scaffolding, DWCF's use prioritizes individual student requirements and may promote increased automatization of grammatically accurate language production. This automatization may then result in stronger self-editing abilities to provide multilingual students with increased self-sufficiency that will prove valuable to them throughout their language learning experiences.
ACKNOWLEDGMENTS
Acknowledgements, funding information and declaration of competing interests have been removed for blind review.
DECLARATION OF COMPETING INTEREST
None declared.
REFERENCES
Bates, L., Lane, J., & Lange, E. (1993). Writing clearly: Responding to ESL compositions. Heinle & Heinle.
Bitchener, J. (2008). Evidence in support of written corrective feedback. Journal of Second Language Writing, 17(2), 102-118. http://dx.doi.org/10.1016/jjslw.2007.11.004
Bitchener, J. (2019). The intersection between SLA and feedback research. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues, (2nd ed., pp. 85-105). Cambridge University Press. http://dx.doi. org/10.1017/9781108635547.007
Bitchener, J. & Ferris, D. R. (2012). Written corrective feedback in second language acquisition and writing. Routledge. http://dx.doi. org/10.4324/9780203832400
Bitchener, J. & Knoch, U. (2010). The contribution of written corrective feedback to language development: A ten month investigation. Applied Linguistics, 31(2), 193-214. http://dx.doi.org/10.1093/applin/amp016
Bruton, A. (2009). Improving accuracy is not the only reason for writing, and even if it were... . System, 37(4), 600-613. https:// doi.org/10.1016/j.system.2009.09.005
Chandler, J. (2003). The efficacy of various kinds of error feedback for improvement in the accuracy and fluency of L2 student writing. Journal of Second Language Writing, 12, 267-96. http://dx.doi.org/10.1016/S1060-3743(03)00038-9
DeKeyser, R. (2001). Automaticity and automatization. In P. Robinson (Ed.), Cognition and second language instruction (pp. 97113). Lawrence Erlbaum. http://dx.doi.org/10.1017/CB09781139524780.007
DeKeyser, R. (Ed.). (2007). Practice in a second language: Perspectives from applied linguistics and cognitive psychology. Cambridge University Press. http://dx.doi.org/10.1017/CB09780511667275
Eckstein, G., Sims, M., & Rohm, L. (2020). Dynamic Written Corrective Feedback among graduate students: The effects of feedback timing. TESL Canada Journal, 37(2), 78-102. http://dx.doi.org/10.18806/tesl.v37i2.1339
Ellis, R., Y. Sheen, M. Murakami & H. Takashima (2008). The effects of focused and unfocused written corrective feedback in an English as a foreign language context. System, 36(3), 353-371. http://dx.doi.org/10.1016/j.system.2008.02.001
Evans, N. W., Hartshorn, K. J., Strong-Krause, D. (2011) The efficacy of dynamic written corrective feedback for university-matriculated ESL learners. System, 39, 229-239. http://dx.doi.org/10.1016Zj.system.2011.04.012
Evans, N. W., Hartshorn, K. J. , McCollum, R., Wolfersberger, M. (2010). Contextualizing corrective feedback in L2 writing pedagogy. Language Teaching Research, 14, 445-463.
Ferris, D. (2006). Does error feedback help student writers? New evidence on the short- and long-term effects of written error correction. In K. Hyland & F. Hyland (Eds.), Feedback in second language writing: Contexts and issues (pp. 81-104). Cambridge University Press. http://dx.doi.org/10.1017/CB09781139524742.007
Ferris, D. R. (2010). Second language writing research and written corrective feedback in SLA. Studies in Second Language Acquisition, 32(2), 181-201. http://dx.doi.org/10.1017/S0272263109990490
Ferris, D. R. & Kurzer, K. (2019). Does error feedback help L2 writers? Latest evidence on the efficacy of written corrective feedback. In K. Hyland and F. Hyland (Eds.) Feedback in second language writing: Contexts and issues (2nd Ed., pp. 106-124). Cambridge University Press.
Ferris D. R., Liu, H., Sinha, A., & Senna, M. (2013). Written corrective feedback for individual L2 writers. Journal of Second Language Writing, 22, 307-329. http://dx.doi.org/10.1016/jjslw.2012.09.009
Foin, A. T., & Lange, E. J. (2007). Generation 1.5 writers' success in correcting errors marked on an our-of-class paper. The CATESOL Journal, 19(1), 146-163.
Han Y. & Hyland, F. (2015). Exploring learner engagement with written corrective feedback in a Chinese tertiary ESL classroom. Journal of Second Language Writing, 30, 31-44. https://doi.org/10.1016/jjslw.2015.08.002
Hartshorn, K. J., & Evans, N. W. (2012). The differential effects of comprehensive corrective feedback on L2 writing accuracy. Journal of Linguistics and Language Teaching, 3, 16-46.
Hartshorn, K. J., & Evans, N. W. (2015). The effects of dynamic written corrective feedback: A 30 week study. Journal of Response to Writing, 1(2), 6-34.
Hartshorn, K. J., Evans, N. W., Merrill, P. F., Sudweeks, R. R., Strong-Krause, D., Anderson, N. J. (2010). Effects of dynamic corrective feedback on ESL writing accuracy. TESOL Quarterly, 44(1), 84-108. http://dx.doi.org/10.5054/tq.2010.213781
Hyland, K. & F. Hyland (2006). Feedback on second language students' writing. Language Teaching, 39(2), 83-101. http://dx.doi. org/10.1017/S0261444806003399
Krashen, S. D. (1985). The input hypothesis: Issues and implications. Pergamon.
Kurzer, K. (in print). Dynamic written corrective feedback. In H. Mohebbi (Ed.) Insights into teaching and learning writing: A practical guide for early-career teachers. Castledown Publishers.
Kurzer, K. (2021). More than just grammar deficiencies: Moving faculty in the disciplines toward a difference-as-resource pedagogical paradigm. In E. Lee, N. Fahim, J. Johnson, and B. R. Schreiber (Eds.), Building a more linguistically just campus: Theory, pedagogy, and advocacy (pp. 198-213). Multilingual Matters.
Kurzer, K. (2019). Dynamic written corrective feedback in a community college ESL writing class setting. In S. M. Anwaruddin (Ed.) Knowledge mobilization in TESOL: Connecting research and practice (pp. 59-75). Brill.
Kurzer, K. (2018b). Student perceptions of dynamic written corrective feedback in developmental multilingual writing classes. Journal of Response to Writing, 4(2), 34-68.
Kurzer, K. (2018a). Dynamic written corrective feedback in developmental ESL writing classes. TESOL Quarterly, 52(1), 5-33. doi: 10.1002/tesq.366
Lalande, J.F. II (1982). Reducing composition errors: An experiment. Modern Language Journal 66.2, 140-149. http://dx.doi. org/10.1111/j.1540-4781.1982.tb06973.x
Lane, J., & Lange, E. (2012). Writing clearly: Grammar for editing (3rd ed.). Heinle Cengage Learning.
Larsen-Freeman, D. (2009). Adjusting expectations: The study of complexity, accuracy, and fluency in second language acquisition. Appl/ed L/ngu/st/cs, 30(4), 579-589. http://dx.doi.org/10.1093/applin/amp043
Lee I. (2005). Error correction in the L2 classroom: what do students think? 7ESL Canada Journal 22, 1-16. http://dx/doi. org/10.18806/tesl.v22i2.84
Long, M. H. (1996). The role of linguistic environment in second language acquisition. In W. Ritchie & T. Bhatia (Eds.), Handbook of second language acqu/s/t/on. Academic Press. http://dx.doi.org/10.1016/B978-012589042-7/50015-3
Loza, S. (in print). Oral corrective feedback in the Spanish heritage language context. In S. Loza & S. M. Beaudrie (Eds.), Her/tage language teach/ng: Cr/t/cal language awareness perspect/ves for research and pedagogy. Routledge.
Marzban, A., & Arabahmadi, S. (2013). The effect of written corrective feedback on Iranian EFL students' writing. Proced/a-Soc/al and Behav/oral Sc/ences, 83, 1000-1005. http://dx.doi.org/10.1016/j.sbspro.2013.06.186
Messenger, R. A., Evans, N. W., & Hartshorn, K. J. (2020). Managing Dynamic Written Corrective Feedback. Journal of Response to Wr/t/ng, 6(1), 5.
Polio, C., & Shea, M. C. (2014). An investigation into current measures of linguistic accuracy in second language writing research. Journal of Second Language Wr/t/ng, 26, 10-27. http://dx.doi.org/10.1016/j.jslw.2014.09.003
Sheen, Y. (2007). The effect of focused written corrective feedback and language aptitude on ESL learners' acquisition of articles. 7ESOL Quarterly, 41(2), 255-283. http://dx.doi.org/10.1002/j.1545-7249.2007.tb00059.x
Sheen, Y., Wright, D., & Moldawa, A. (2009). Differential effects of focused and unfocused written correction on the accurate use of grammatical forms by adult ESL learners. System, 37(4), 556-569. http://dx.doi.org/10.1016/j.system.2009.09.002
Truscott, J. (1996). The case against grammar correction in L2 writing classes. Language Learn/ng, 46(2), 327-369. http://dx.doi.org/10.1111/j.1467-1770.1996.tb01238.x
Truscott, J. (2007). The effect of error correction on learners' ability to write accurately. Journal of Second Language Wr/t/ng, 16(4), 255-272. http://dx.doi.org/10.1016/jjslw.2007.06.003
Truscott, J., & Hsu, A. Y. (2008). Error correction, revision, and learning. Journal of Second Language Wr/t/ng, 17, 292-305. http:// dx.doi.org/10.1016/j.jslw.2008.05.003
Vygotsky, L. (1978). Interaction between learning and development. Read/ngs on the Development of Ch/ldren, 23(3), 34-41. http://dx.doi.org/10.2307/jxtvjf9vz4.11
Wigglesworth, G. (2008). Measuring accuracy in second language performance. Paper presented at the Annual Conference of Teachers of English to Speakers of Other Languages (TESOL), New York.
Wolfe-Quintero, K., Inagaki, S., & Kim, H. (1998). Second language development in writing: Measures of fluency, accuracy, and complex/ty. University of Hawaii at Maona.
Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Ch/ld Psych/atry and Psychology, 17, 89-100. http://dx.doi.org/10.1111/j.1469-7610.1976.tb00381 .x
Xu, C. (2009). Overgeneralization from a narrow focus: A response to Ellis et al. (2008) and Bitchener (2008). Journal of Second Language Wr/t/ng, 18, 270-275.
APPENDIX A
DWCF Writing Correction Marks
Code Error Type Example
VF Verb Form It was happened yesterday. Psychology expose you to behavior.
(Л VT Verb Time It happen yesterday.
CD LU "та -Q о SS Sentence Structure (incl. Run-on and incomplete) They brought the man who them him found. Because they thought it was good. Recaiise friendship takes effort, so it is time-consuming.
и iwloj Word Order Especially, I miss home.
WC Word Choice (that impacts comprehension) Candy makes children feel a sweet taste.
PP Prepositions I was responsible of everything.
(Л О LU D Determiner (articles) The trip to United States was enjoyable.
"та и о NF Noun Form All family member are supposed to get along. She limited the amount of candies I could eat.
WF Word Form Money brings themselves more opportunities.
SPG Spelling I never worried about my teech getting bad.
P Punctuation When I was visiting; one morning scared me.
та и Ъ C Capital letter Students love to party. they also love to eat pizza.
и Use with SS
s t/1 л Insert something A good major helps you earn a lot money.
CD LU <U -C ^ Omit something I chose this major is because it is interesting.
О ? Meaning is not clear He borrowed some smoke.
AWK Awkward wording (that is still comprehensible) She says that raising a pet needs responsibility.
APPENDIX B
Error Log
1 2 3 4 5 6 7 8 9 10 Total
Paragraph Score:
VF
CD VT
LU -Q SS
O (w\oi
WC
Local Errors PP
D
NF
WF
Other Errors (Mechanical) SPG
P
C
A
?
AWK
APPENDIX C
DWCF Paragraph Prompts
Diagnostic/Pre-test (used at both intermediate and advanced levels): Discuss what you want to accomplish this quarter. What do you need to do in order to accomplish these goals? (Remember that these paragraphs shouldn't be returned to the students for editing until the end of the quarter, for our study.)
Study prompts (used at the intermediate levels):
1. Describe your week so far. What have you accomplished? What do you still want to do? (Verb tense)
2. Write about your most recent vacation. What did you do? Where did you go? (Verb time)
3. What is a regret you have? What should you have done and why? (Modals)
4. What is the best gift you have ever received and why? (Passive voice)
5. Think of a prominent historical figure. What are his/her qualities? (Subject/verb agreement)
6. What is your definition of success? What makes a successful person? (Word order)
7. If you were given the chance to change your life at this moment, what would you do and why? (Conditional sentences)
8. Describe an embarrassing moment you've experienced. (Clauses)
Post-test prompt (used at both intermediate and advanced levels): If you were given the chance to change your life right now, what would you do and why?
