Replication research in AL: What needs to be replicated,

why, and how to do it.

Summary:

Marsden’s AAAL 2019 plenary highlighted the Open Science Initiative and the
importance of the sharing and accessibility of data, underscoring increasing
concerns about low publication rates and quality of replication research. This
colloquium takes up the challenge of meeting these concerns from a practical
standpoint. Researchers who work in the fields of form-focused instruction, L2
writing, task-based instruction, L2 classroom interaction, and vocabulary
acquisition will defend the need for replication of key studies in their field and
provide practical suggestions on how to go about this.

Organizer’s abstract

Marsden’s AAAL 2019 plenary highlighted the Open Science Initiative and the
importance of the sharing and accessibility of data, underscoring increasing
concerns about low publication rates and quality of replication research.
Replicating a study is predicated on the idea that research cannot include or
safely control for the many variables that can affect an 0utcome. Despite built-in
safeguards and the precision with which we prepare and execute our study, what
we do is inevitably subject to potentially significant limitations, bias, and error.
Critically, something will always remain that merits further clarification,
tweaking, or investigation. Many studies stand to benefit from renewed attention
if they can have their findings more precisely validated, their reliability assessed,
and their generalization tested or delimited. While recent publications have
embraced the pressing need for more replication, and more journals have offered
space for their publication,progress is hampered by initial drawbacks: many budding
replication researchers remain unaware of what needs to be replicated and how
to go about doing this.
This colloquium takes up the challenge of meeting these concerns from a
practical standpoint and focuses on studies that can be logistically replicated.
Participants will select significant studies, in terms of content and impact on the
field, discuss the need for replication, and suggest ideal methodology and
analysis to make the replication worthwhile. Thus, concomitantly, attendees will
appreciate how a replication submission to a journal is best structured. The
colloquium starts with a 5-minute contextualization of replication studies in
applied linguistics. Five 15-minute presentations follow, with 3 minutes for brief
clarifications after each one. The final section of the colloquium consists of a 20-
minute open discussion among all participants and attendees which will both
critically address the nature of the 5 presentations and connect them to future
directions in the area of replication studies.

Individual paper presentations

Investigating the role of recasts versus prompts: Replications of three

interactional feedback studies

This paper proposes approximate replications of three interactional feedback

studies that examined two major feedback types: recasts and prompts. Three
well-known studies are Lyster (2004), Ammar and Spada (2006), and Author
(2009). Lyster (2004) examined the effects of recasts and prompts on learning
French grammatical gender in French immersion classrooms. Ammar and Spada
(2006) compared their impacts on learning English third person possessive
determiners in intensive ESL classrooms. Both studies reported an advantage of
prompts over recasts with Ammar and Spada finding mediating effects for
language proficiency. However, these results should be treated cautiously: both
used feedback plus form-focused instruction and compared one recast type with
multiple prompt types. Thus, it is unclear whether the effects shown were
because of the feedback, form-focused instruction, or multiplicity of prompt
types. By conducting replication studies that focus on one recast type versus one
prompt type without instruction, the differential effects of these two feedback
types can be better evaluated. Author (2009) examined extensive feedback using
a research design that pretested the target forms that arose incidentally during
interaction. The results showed an advantage of recasts over prompts on the
immediate posttest. No other studies have yet examined incidental feedback in
pretest-posttest studies. Therefore, replication in needed to assess the
generalizability of its findings. Approximate replications of the suggested studies
will contribute importantly to our understanding of the role of recasts versus
prompts in L2 teaching and learning.

Author (2009).
Ammar, A., & Spada, N. (2006). One size fits all?: Recasts, prompts, and L2
learning. SSLA, 28, 543-574.
Lyster, R. (2004). Differential effects of prompts and recasts in form-focused
instruction. SSLA, 26, 399-432.
2

Using conceptual replications in studies of prompt effects on L2 writing

Studies examining writing prompts as an independent variable affecting learners’
linguistic production are ultimately concerned with fair assessment (He & Shi,
2012), how writing prompts might best promote language learning (Ruiz-Funes,
2015), or with testing competing theories of SLA (Kuiken & Vedder, 2008).
Different theoretical framings and a range of dependent variables makes
comparisons of the many studies challenging. Three studies are discussed with a
focus on clearly operationalizing the independent (i.e., prompt) variable and
standardizing the dependent variables. All are feasible for novice researchers to
conduct within a one-semester graduate course.
Kuiken & Vedder’s widely-cited study of task complexity in writing aimed to
compare Skehan’s and Robinson’s competing hypotheses. Despite a large n(91),
the number of independent variables made the resulting groups small.
Operationalization and the linguistic variables may also have produced
inconclusive results. Ruiz-Funes had a small sample size and a task that
conflated genre and complexity. The limited range of linguistic outcome
measures made comparison with other written task complexity studies difficult.
He & Shi’s study of topic familiarity had few participants at each proficiency level.
Limited outcome measures made the study difficult to compare to those situated
within the task complexity literature.
I will compare the studies regarding prompt variables and outcome measures and
suggest replications that strengthen each to make them more rigorous,
conclusive, and comparable to other studies.

Ruiz-Funes, M. (2015). Exploring the potential of second/foreign language writing

for language learning: The effects of task factors and learner variables. JSLW, 28,
1-19.
He, L. & Shi, L. (2012). Topical knowledge and ESL writing. Language Testing, 29,
443-464.
Kuiken, F. & Vedder, I. (2008). Cognitive task complexity and written output in
Italian and French as a foreign language. JSLW, 17, 48-60.

3
Replication in task-based language teaching (TBLT) research

This presentation calls for the replication of Kim (2012) and Shintani (2012),
highly-cited studies investigating the effectiveness of TBLT in promoting L2
development. Kim studied the impact of manipulating task variables on
interaction-driven L2 learning opportunities and development; Shintani studied
the potential of input-based tasks to promote interaction and L2 development.
Unlike most existing TBLT research, besides performance, both studies
investigated development and were conducted in classroom rather than
laboratory contexts. In doing so, Kim made a significant contribution to the large
body of research examining output-based tasks, whereas Shintani moved the field
forward by investigating input-based tasks, an area which is less explored.
Additionally, both are well designed pieces of research, with accessible research
instruments and clear descriptions of the data collection and analytical
procedures, making replication feasible.

Attendees will be encouraged to conduct approximate and conceptual

replications of the studies. The internal validity of Kim could be enhanced by
adding an independent measure of task complexity, manipulating the treatment
tasks along a single rather than multiple task dimensions, or assigning more than
one intact class to the experimental conditions. Replication with a larger sample
and teachers other than the researcher would strengthen the methodology.
Alternative replications might usefully change the populations, contexts, task
types, or modality to test generalizability. Conceptual replications of both studies
employing verbal protocols would also be valuable to examine the cognitive
processes in which participants engaged during the treatment. This would allow
researchers to arrive at less tentative explanations regarding task effects on
development.

Kim, Y. (2012). Task complexity, learning opportunities and Korean EFL learners’
question development. SSLA 34.4, 627–658.

Shintani, N. (2012). Input-based tasks and the acquisition of vocabulary and

grammar: A process–product study. Language Teaching Research 16.2, 253–279

Replicating key interactionist studies

The findings of these interaction and corrective feedback studies have had
substantial impact on subsequent interactionist research. The prominence of
these studies is evidenced by the fact that each has roughly 800 citations
according to Google Scholar (1088, 981, and 799 respectively). Every study has
methodological strengths and weaknesses, and this presentation will explore how
a close replication might rectify some of these studies’ weaknesses. For example,
a small sample size (i.e., a range of between only 7 and 12 participants per group)
is a considerable threat to the internal validity of each of these studies.
Additionally, each study was conducted within a specific context and examined a
limited number of variables. Partial replication that increases the sample size and
extends the research contexts and research variables (as suggested below) would
provide important information about the robustness and generalizability of each
studies’ results. For instance, partial replications should investigate different:
 types of research contexts (other than ESL in private schools, or Italian as
a foreign language in a university),
 types of learners (e.g., proficiency, age, first language),
 target languages (other than English and Italian),
 target language features (other than English question forms and regular
past tense).
 measures of second language development
By altering key components of these studies, we can have a better understanding
of the effects of interaction and corrective feedback.

Ellis, R., S. Loewen & R. Erlam (2006). Implicit and explicit corrective
feedback and the acquisition of L2 grammar. SSLA 28.3, 339–368.

Mackey, A., Gass, S. & McDonough, K. (2000). How do learners perceive

interactional feedback? SSLA, 22, 471-497.

Mackey, A. (1999). Input, interaction, and second language development:

An empirical study of question formation in ESL. SSLA 21.4, 557–587.

Replication in vocabulary learning and teaching research

Research on the effectiveness of different learning conditions on the acquisition
of vocabulary abounds; however, very few attempts have been made to explore
how the different components of vocabulary knowledge develop over time.
Moreover, despite the current interest in the acquisition of formulaic sequences,
few studies have empirically compared the learning of single words and formulaic
sequences. This presentation calls for the replications of Schmitt (1998) and
Laufer & Girsai (2008). Unlike most studies on vocabulary learning and teaching,
Schmitt followed a longitudinal approach to examine the development of several
lexical components over a year. However, this study was an exploratory
investigation with only three learners. A close replication with a larger sample
size would help confirm the original results and assess the generalizability of its
findings. The original study used a challenging test of receptive knowledge.
Conducting a replication with an easier receptive test would provide a more
detailed picture of receptive-productive knowledge. The addition of a second
rater in this approximate replication would also provide reliability to the original
findings. Laufer & Girsai investigated the effectiveness of three focus-on form
approaches to the acquisition of both single words and collocations. However,
the statistical analyses did not compare the learning of these two types of lexical
items. Thus, a close replication including item type as an independent variable
would allow us to examine potential differences in the acquisition of single words
and collocations. These replications would constitute an important contribution
to the field of vocabulary learning.

Laufer, B., & Girsai, N. (2008). Form-focused instruction in second language

vocabulary learning: A case for contrastive analysis and translation. Applied
Linguistics, 29(4), 694–716.

Schmitt, N. (1998). Tracking the incremental acquisition of second language

vocabulary: Longitudinal study. Language Learning 48.2, 281–317.

Open discussion and future directions for applied linguistics replication

studies

The colloquium will be closed with a 20-minute discussion and question-and-

answer session which will seek to both encourage critical address of the
replications suggested and open out into an interactive debate concerning the
future path for research replication in applied linguistics.