Research Notes

Issue 65 February 2017

Impact of Cambridge English exams and English
language learning programmes in a variety of contexts

A quarterly publication reporting on teaching, learning and assessment

ISSN 1756-509X
Research Notes
Issue 65/February 2017
A quarterly publication reporting on learning, teaching and assessment

Senior Editor and Editor

Dr Hanan Khalifa, Head of International Education Strategy, Cambridge English Language
Assessment
Dr Fiona Barker, Principal Research Manager, Cambridge English Language Assessment
Editorial Board
Roger Johnson, Chief Operating Officer, Cambridge English Language Assessment
Christine Nuttall, Director of Partnerships, Projects and Policy, Cambridge English Language Assessment
Production Team
Matthew Heyes, Marketing Project Co-ordinator, Cambridge English Language Assessment
John Savage, Publications Assistant, Cambridge English Language Assessment

Printed in the United Kingdom by Canon Business Services

CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|1

Research Notes
Issue 65 February 2017

Contents
Editorial2

Understanding language learning in Malta 3

Nahal Khabbazbashi, Hanan Khalifa, Martin Robinson, Sarah Ellis and Charles Mifsud

An investigation of Portuguese students attitudes to assessment and Cambridge English exams 24

Jane Lloyd, Elaine Blaus and Helder Sousa

Assessing the English language progress of students in a trilingual education framework in the
Basque Country 39
Belinda Cerd, Andrew Blackhurst and Christine Walker

Which factors affect English language attainment? A study of school students in Chile 51
Agnieszka Walczak, Graeme Harrison, Mercedes Muratorio, Carolina Flores, Sofia Brunner and
CoreenDocherty

The impact of the Antioquia English in the Park initiative on language teaching and practice 65
Daniel Brooker, Jane Lloyd, Martin Robinson and Sergi Casals

Improving student learning through upskilling teachers: The case of Lebanon75

Coreen Docherty, Tania Barakat, Elaine Kniveton, Lama Mikati and Hanan Khalifa

UCLES 2017 CONTENTS

2|ISSUE 65 / FEBRUARY 2017
Editorial
In this issue of Research Notes we share six educational
reform projects that Cambridge English has undertaken
in collaboration with ministries of education and other
institutions in a variety of international contexts. From 2013
to the present, Cambridge English has supported educational
initiatives in Malta, Portugal, the Basque Country, Chile,
Colombia and Lebanon; we have used our English language
exams and qualifications for teachers to build a profile of the
attitudes and aptitude of learners, teachers and parents, with
the intention that our findings and recommendations help to
support the country or regions education strategy.
The opening article by Khabbazbashi, Khalifa, Robinson, Ellis,
and Mifsud outlines the efforts by the Maltese Ministry for
Education and Employment to benchmark the English language
level of Maltese school students and the training of Maltese
teachers at an institutional level. Cambridge English Language
Assessment provided the language tests and then compared
the results according to background factors. Surveys were
also conducted to discover the attitudes of learners, teachers
and parents towards learning and assessment. The CEFR level
according to students gender and school sector was ascertained
for each skill, and responses regarding attitudinal factors such as
views on assessment and factors influencing performance such
as use of technology, were triangulated. The article presents and
reflects on these findings, and concludes with recommendations
on how to narrow the achievement gap amongst learners,
improve learner motivation and autonomy, and provide greater
scope for improvements in teaching practice.
In the following article Lloyd, Blaus and Sousa detail a multi-
phase study where the Portuguese Ministry of Education
and Science used a Cambridge English exam to measure and
monitor Grade 9 pupils language learning and to inform how
teacher development could be refined; and additionally to offer
an internationally recognised qualification to pupils of diverse
socioeconomic backgrounds. Cambridge English: Preliminary was
introduced as the external exam for the 2014/15 academic year.
Quantitative data such as responses to attitudinal questions
and candidate test scores were gathered and analysed,
including descriptive and factor analyses. The findings led to
the conclusion that use of English outside the classroom and
increased pair work had a positive effect on language learning,
and the nature of the feedback given to pupils needs to be
reviewed to increase its positive influence on test scores.
The third article, by Cerd, Blackhurst and Walker, is a third
European study which provides insights into the performance
of the 2014 primary (4th and 6th year) and secondary (1st
and 3rd year) cohort of the Marco de Educacin Trilinge
(Trilingual Framework project), to whom Cambridge English
and the Basque Institute for Research and Evaluation in
Education simultaneously administered language tests of
English and Spanish. This project aimed to ascertain the CEFR
levels and compare the performances of two control groups
and an experimental group, with one control group receiving
the minimum legal requirement of English teaching and the
second control group receiving additional teaching. The second
control group was found to contain the strongest performers,
but all three groups displayed strengths and weaknesses
which has provided a strong argument to support further
improvements in foreign language education and proficiency in
the Basque Country.
Moving to South America, the next article focuses on Chile,
in which Walczak, Harrison, Muratorio, Flores, Brunner
and Docherty investigate the background factors that
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
influence English language attainment for school students.
The article analyses the data gathered from the 2012 and
2014 administrations of the Simce Ingls exam provided by
Cambridge English. After establishing the educational context
in Chile, the article addresses the questions of what CEFR
level the 2014 cohort achieved, how performances changed
between 2012-14 , as well as the influence of background
factors. Questionnaire responses from students, parents
and teachers provide insights into their socioeconomic
background, education level, use of English within and outside
the classroom, and motivation for learning or teaching English.
Based on extensive analysis, the authors conclude that
students increased exposure to English, greater scaffolding
for teachers professional development and self-reflection, and
increased monitoring of school resources and policy to ensure
equality of opportunity for all students are the steps needed to
close proficiency gaps in Chile.
We remain in South America for the next article, in which
Brooker, Lloyd, Robinson and Casals discuss the Colombian
province of Antioquias English in the Park Initiative. In
collaboration with the Education Secretariat of Antioquia,
Cambridge English developed self-access web portals,
workshops and provided language placement tests to improve
motivation, professional development and equality in language
teaching and learning in the province. This article investigates
the impact of this initiative through the analysis of attitudinal
and demographic data on pupil and teacher motivation and
practice. The findings showed that the initiative led to a
positive response to language learning and teaching, and that
the introduction of Cambridge English teaching qualifications
and gathering data for specific year groups taking the
placement test will provide a precise foundation to build on the
findings obtained so far.
We go to the Middle East in the closing article, in which
Docherty, Barakat, Kniveton, Mikati and Khalifa describe
the Cambridge English evaluation of the Developing
Rehabilitation Assistance to Schools and Teachers
Improvement (D-RASATI 2) programme designed to
support the Lebanese Ministry of Higher Education in
improving the public school system, involving both teachers
of English and teachers teaching through English. A key
indicator in this project was for 245 teachers to sit the
Cambridge English: First exam, as its B2 level is the target
proficiency for English-medium teachers, and investigations
of key stakeholders attitudes and perceptions of D-RASATI
2. By analysing both qualitative and quantitative data, it
was found that the initiative gained the positive result
of increased teacher self-reflection and communication,
and established that some current misconceptions and
misunderstand of the purpose of language courses could
be combatted through the use of increased promotion and
training of teaching methodology.
The educational reform projects presented in this issue
demonstrate how ministries are moving towards international
benchmarking of their English language education systems
through collaboration with one of the world-renowned
assessment boards. All of the projects described in this issue
have succeeded in securing the engagement of stakeholders
and have shown improvements in test scores based on
evidence, but all acknowledge that now pupils and teachers
better understand the benefits of participating in these
programmes, future programmes must focus on closing
achievement gaps and expanding professional development
possibilities for teachers and teacher trainers.
EDITORIAL
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|3

Understanding language learning in Malta

NAHAL KHABBAZBASHI CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT

HANAN KHALIFA CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT

MARTIN ROBINSON CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT

SARAH ELLIS CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT, ITALY

CHARLES MIFSUD FACULTY OF EDUCATION, UNIVERSITY OF MALTA

Introduction English language standards. The two factors of school

sector and gender were also identified as key variables in
The Ministry for Education and Employment (MEDE) in the project. Their impact on performance was investigated
Malta has a strategic objective to maintain and strengthen and will be discussed in more detail.
standards of English language proficiency within the school
Results of surveys designed to elicit attitudes of the main
education system so that Malta remains a high-performing
stakeholders (i.e. learners, parents and teachers) towards
Commonwealth and European Union country. This should
English language learning, teaching and assessment will
perpetuate the existing, successful provision of bilingual
also be presented. The findings and recommendations from
education (Mifsud and Vella forthcoming 2017, Ministry
this phase of the project will serve as an evidence-based
of Education 2016) which prepares individuals from early
framework of current and target proficiency levels and can
on in life, who are equally fluent in Maltese and English, for
be used to inform further areas for research.
the global employment market. To this end, the Ministry
and Cambridge English Language Assessment entered into
an agreement for an integrated solution which included
the benchmarking of student English language levels in
Conceptual framework
the school sector (Year 1) and the institutional capacity- The construct of communicative language competence,
building of Maltese teachers (Year 2). which has become widely accepted as the goal of language
education and as central to good classroom practice
Cambridge English Language Assessment, in collaboration
(Bachman and Palmer 1982, Canale and Swain 1980)
with the Ministry, conducted the Year 1 benchmarking
was central to the design of this project. Communicative
project, which aimed at presenting a snapshot of English
language competence comprises linguistic competence,
language proficiency in two key grades at Primary and
as well as the ability to functionally use that competence
Secondary education in relation to international standards,
in language activities which involve oral and/or written
namely the Common European Framework of Reference
reception, production and interaction in different domains.
(CEFR, Council of Europe 2001). The six CEFR reference
The communicative view of language is also very much in
levels are now widely accepted as the international
line with the principles and learning outcomes identified
standard for grading an individuals language proficiency.
in the National Curriculum Framework (NCF) for All
In addition to benchmarking learner proficiency, a (Ministry for Education and Employment 2012) for
comprehensive profile of learner, teacher and parent languagecompetence.
attitudes towards English language education in
All instruments used in the benchmarking project
Malta was investigated. This aspect of the project was
(tests and surveys) were therefore designed with the
designed to provide a comprehensive profile of the
communicative view of language competence in mind.
Maltese educational context by bringing together views
The assessment instruments are also linked to the CEFR,
from the main stakeholders. Results also feed into the
which provides the gold standard for measuring language
Ministrys desire to deliver institutional capacity-building
competence and which allows the findings of the project
in assessment and teaching methodology, a strategic
to be considered against a broader international context
plan for continuous professional development of the
where the CEFR is used.
teaching cadre, and potential international certification for
learnersandteachers. Finally, an important approach to investigating English
language proficiency in this project is the triangulation
of data from multiple sources; while the current project
Project aims is predominantly quantitative in nature, efforts were
taken to (a) collect qualitative data in the form of open
The aim of the project was to provide MEDE with a clear
comments in surveys and (b) ensure that any resulting
picture of how a representative sample of learners at two
data can systematically inform future projects by linking
key school stages Primary Year 5 and Secondary Form 4
the survey items and open comments to areas of interest
is currently performing against internationally recognised
by the Ministry (e.g. professional development of

UCLES 2017 UNDERSTANDING LANGUAGE LEARNING IN MALTA

4|ISSUE 65 / FEBRUARY 2017
teachers, language policies across sectors, sharing of good
practices). Such an approach ensures that results of current
and future research can be brought together coherently
in developing an in-depth view of the Maltese English
educational context and in highlighting areas that warrant
furtherattention.
Research questions
The project was guided by the following key research
questions (RQs):
RQ1. How do learners at two key school stages Primary
Year 5 and Secondary Form 4 in Malta perform on
a set of Cambridge English language tests on the four
skills ofReading, Writing, Listening and Speaking against
theCEFR?
RQ2. How do learner performances at Primary and
Secondary stages compare according to school sector
(State/Church/Independent) and gender (female/male)?
RQ3. What are the attitudes of learners, parents and
teachers towards the learning, teaching and assessment of
English language in Maltese schools?
RQ4. What are some key trends in English language
teachers pedagogical and assessment practices in
Malteseschools?
Methodology
The benchmarking project aims at (a) building a profile of
English language ability levels in Malta, (b) establishing
stakeholder attitudes to English language learning,
teaching and assessment, (c) identifying the professional
development needs of English language teachers and
(d)highlighting areas that require further attention in
creating a truly bilingual educational system. The scope
of the project therefore necessitates a research design
which allows for the collection of different types of data
from a variety of sources and using a range of instruments.
A premise recognised in educational reform is that a key
characteristic of the educational process is that student
learning is influenced by many small factors rather than a
few large ones (Chapman, Weidman, Cohen and Mercer
2005:526); therefore, any recommendations made in this
project needed to be based on an in-depth understanding of
all aspects of the educational system in order to ensure that
they are achievable and reduce the chances of any negative
unintended consequences. As a result, the project focuses
not only on measuring English language levels of learners,
but also on investigating the context of learning both inside
and outside of school, the availability of resources, the
extent of parental support, and stakeholder perceptions. A
mixed methods approach therefore formed the basis of the
study and an explanatory sequential design (Creswell and
Plano Clark 2011) was selected in addressing the studys
research questions.
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
Research design
Given the Ministrys emphasis on continuous research
and multi-phase projects, a mixed methods explanatory
sequential design (Creswell and Plano Clark 2011), where
a quantitative stage is followed by a qualitative stage,
was considered to be the most appropriate design for a
multi-phase project. Using this design, the results of the
Year 1 data can systematically inform the focus of future
work; for example, the benchmarking data can be used
to identify high-and low-performing schools and classes
and to subsequently focus observations on differences
in pedagogies and practices in these schools (which will
also have been stratified by school sector). Questionnaire
data from Year 1 can also be used to design observation
instruments and inform areas to be addressed in focus
groups in future stages, and together provide a more
systematic and comprehensive approach to addressing the
research questions in more depth.
It is important to note that while the Year 1 benchmarking
project is predominantly quantitative, care was taken to
also collect a small sample of qualitative data (through
open comments in the questionnaires administered to the
different stakeholders) to complement the quantitative
data and provide a more in-depth understanding of the
educational context. These two strands of data were
brought together and integrated in drawing conclusions and
making recommendations using Creswell and Plano Clarks
(2011) convergent parallel mixed methods (see Figure 1).
Figure 1: Research design for Year 1 benchmarking project
(based on Creswell and Plano Clark 2011)
Quantitative data Quantitative data
collection analysis
Benchmarking tests
(learners)
Mapping of test results
onto CEFR levels
Multiple-choice surveys Descriptive statistics
(learners, parents, of survey responses
teachers) Multi-level modelling Integration and
interpretation
Qualitative data Qualitative data
collection analysis
Open comments
in surveys (learners,
Thematic analysis of
survey open comments
parents, teachers)
Sampling framework
A stratified sampling design was used as the sampling
framework for the project and ensured the inclusion of
a representative sample of learners across the two key
variables of school sector and gender. Schools were
sampled from a comprehensive national list of schools in
Malta. Sampling was undertaken separately for Primary
Year 5 and Secondary Form 4. The agreed total number
of learners to be sampled was 1,255 (Primary N=624;
Secondary N=631) for the administration of the Reading,
Listening and Writing exams. Given practical constraints
and the resource-heavy nature of examining Speaking
face-to-face, it was agreed for the Speaking tests to
be administered to a smaller proportion of the sample.
Approximately 50% of schools from the original sample
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES
were chosen for the administration of the Speaking test
using a purposeful sampling approach. This smaller
sample for Speaking means that the precision criteria of
results are different from the ones from Reading, Listening
and Writing tests. Therefore, more caution is needed
in interpreting the Speaking results. Note that in the
purposeful sampling approach, important variables such as
school sectorand gender segregation of learners in Church
and Independent schools were taken into account for
school selection. This means that the quality of purposeful
sampling is high, but we still cannot make the same
statistical inferences as with the full sample. Nevertheless,
given the importance of assessing Speaking face-to-face
and in interaction, the fact that Speaking was included is
an important strength of the project. A similar sampling
framework was also recommended for learner surveys.
Parent and teacher participation in the study was on a
voluntary basis.
It should be noted that while the original sampling
framework specified approximately 625 learners per year
(for Reading, Listening and Writing), MEDE administered
the Reading, Listening and Writing exams to all Primary
and Secondary classes within selected schools for equity
purposes. This resulted in a larger number of learners
than that outlined in the sampling framework. Cambridge
English marked all available Reading and Listening exams
(N=3,073) and the agreed number of Writing (N=1,375)
and Speaking (N=714) exams. A total of 1,250 learners,
793 parents and 132 teachers completed the surveys.
Please note that not all survey respondents completed all
sections of the surveys (the total number indicates all who
attempted the surveys).
Data collection instruments
A range of instruments was used to collect both the
quantitative and qualitative data. More details are
presented below. Ethical guidelines from the University of
Cambridge, the British Association for Applied Linguistics
and the British Educational Research Association were
followed during all data collection and data analysis phases
of this project.
Benchmarking English language tests for learners
The Cambridge English benchmarking tests aimed to
provide information on learners language proficiency, in
terms of the four skills of Reading, Listening, Writing and
Speaking as measured against the CEFR.
Attitudinal and background surveys
A series of learner, teacher and parent surveys were used
in the project in order to gather stakeholder perceptions
of and attitudes towards English language learning,
teaching and assessment in the Maltese context. Validated
statements were selected from the Cambridge English
Questionnaire Item Bank and additional questions, specific
The translation of the parent surveys from English to Maltese was arranged by MEDE.
1
UCLES 2017
ISSUE 65 / FEBRUARY 2017|5
to the Maltese context and the identified areas of focus
inthe project, were developed in collaboration with MEDE.
The statements include a variety of response options with
the most common being Likert scale items consisting of a
4-point scale as well as Not Sure or Not Applicable options
where relevant. The most typical response options are:
Strongly Agree, Agree, Disagree and Strongly Disagree.
A key feature of the surveys is the inclusion of parallel
statements, which allows for a comparison of responses
from learners, parents and teachers where relevant and is
used for triangulation purposes and for providing a more
comprehensive picture. Paper-based versions of the surveys
were distributed to learners. Online and paper-based
versions of the surveys were made available to parents in
both Maltese1 and English so that they could select the
language in which they were most comfortable responding.
Teachers were invited to complete the surveys online and in
English using SurveyMonkey.
Data analysis
The data analyses comprised the following for the
quantitative strand:
CEFR level mapping: Rasch analysis and ability
estimates.
Descriptive statistics in the quantitative strand: aimed
to provide an overall picture of CEFR language level,
stakeholder perceptions, as well as the amount of
variability within each group. The analysis focused on
the cohort as a whole (e.g. all Form 4 learners) and on
specific variables within the cohort (e.g. Form 4 boys
and girls; Form 4 Church, Independent, State schools).
Chi-square test of independence: aimed to investigate
whether the different variables of interest (school sector,
gender, etc.) were related to questionnaire responses.
These variables were used to define groups within the
data and compare responses. Standardised residuals
were also computed to identify which responses were
contributing to the test of significance.
Multi-level modelling: aimed to explore and confirm
whether any background or attitudinal variables
(school sector, gender, language used at home, etc.)
played a significant role in predicting the language level
oflearners.
For the small qualitative strand, a thematic analysis of
open comments in surveys was carried out with the aim
of identifying key themes that indicated (any) important
issues brought up by the different stakeholders participating
in the project. A final stage involved the integration of
different sources of data in providing a coherent narrative
on the main findings from the project.
UNDERSTANDING LANGUAGE LEARNING IN MALTA
6|ISSUE 65 / FEBRUARY 2017
English language proficiency
In addressing the studys first three research questions,
this section provides a snapshot of the established learner
proficiency levels for Primary and Secondary learners and
highlights the most salient and meaningful differences in
proficiency levels that could be attributed to key variables
of interest, such as school sector and gender.
A greater number of learners took the Reading, Listening
and Writing tests (paper-based) compared to Speaking
(face-to-face). The Reading, Listening and Writing results
can therefore be generalised to the Maltese Year 5 and
Form 4 populations. The smaller sample for Speaking
means that the precision criteria of results are different
from the ones from Reading, Listening and Writing
tests. Therefore, caution is needed in interpreting the
Speakingresults.
Overall profile: Primary (Year 5)
Primary school learners in Malta achieved a mode of A2 in
the skills of Listening, Reading and Writing whereas a mode
of B2 was observed for the skill of Speaking.
When looking at the proportion of candidates at the
B1/B2 levels, results are very positive and show that a
large proportion of learners are at the B levels and can
be considered independent users of English (65.8% for
Speaking, 45.3% for Writing, 41.3% for Listening and 20.1%
for Reading). Speaking is the learners strongest skill, with
18% of candidates achieving Levels C1 and above. The high
levels of performance for Speaking at this age group can be
explained by the bilingual context of Malta, the reported
levels of English language use at home and outside of
school, and the use of English in a variety of domains in
Malta (Council of Europe 2015).
Compared to the other skills, Reading has the
comparatively highest proportion of learners at A1level
(32.8%) which highlights Reading as the learners
weakest skill. A possible explanation for the comparatively
lower performance on Reading can be that of literacy.
According to the Progress in International Reading Literacy
Study (2011) surveys of learner competence in Reading,
Maths, Science and Literacy, Maltese 10-year-olds were
performing significantly lower than the international
average on Reading. The PISA (Organisation for Economic
Co-operation and Development 2009) surveys also
suggested that the proportion of 15-year-olds in Malta who
were classified as low achievers in reading literacy was
significantly higher than the EU average (Organisation for
Economic Co-operation and Development 2009:4142). It
is recognised in the second language acquisition literature
that first language literacy can affect the development of
second language reading and writing (Bernhardt 2005,
Koda 2007). In response, the National Literacy Strategy
(Ministry for Education and Employment 2014) is now
education.gov.mt/en/Documents/Literacy/ENGLISH.pdf
2
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
committed to improving biliteracy in English and Maltese
through a number of initiatives.2
The distribution of CEFR levels, which cover a range from
pre-A1 to C levels, reflects Maltas educational context and
linguistic milieu in which language levels vary quite widely,
with Maltese as the dominant language for some learners
and English for others.
The score results were cross-checked against Primary
learners self-reports of their weaknesses in English as
well as parents and teachers perceptions. The results
in Figure2 show that learners, parents and teachers
all unanimously believe Writing to be the learners
weakest skill. Speaking is the second most selected
skill by parents and teachers even though the data
suggested Speaking to be the strongest skill for the
sub-sample. These findings show a mismatch between
score data and stakeholder perceptions of weaknesses
in English and suggest that Reading should be given
more emphasis for Primary learners, although a review
of both parents and teachers open comments suggests
that more focusonReadingisalready considered an
importantpriority:
If students would take the time to read every day they
wouldimprove considerably. (Secondary teacher, Church
school)
My daughters level of English would certainly improve if she
reads more. (Father of State Secondary learner)
Another noteworthy observation in Figure 2 is that
approximately 50% of learners had selected the not sure
option when asked to identify their weaknesses in English,
which might indicate the absence of self-assessment skills.
While this is expected for younger learners, an awareness
of their strengths and weaknesses will allow learners to
develop self-regulatory learning strategies which can in turn
enhance learner autonomy (in line with key areas of focus
within the NCF).
Figure 2: Weaknesses in English: stakeholder perceptions (%)
Primary
60
Students
Parents
50
Teachers
40
30
20
10
0
Listening Reading Writing Speaking Not Sure
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|7

Overall profile: Secondary (Form 4) Figure 3: Weaknesses in English: stakeholder perceptions (%)
Secondary
The most frequently occurring level in secondary school
learners was B2 in all skills except for Listening, where B1 50
Students
was the observed level. When focusing on the proportion 45 Parents
of candidates at the B1/B2 levels, results show that, on 40 Teachers
average, about 60% of candidates are at the B1/B2 levels 35
and that a large proportion of candidates (approximately
30
28%) achieve C levels for the skills of Speaking and
25
Listening. These results reflect the bilingual context of
Malta where the strong use of Speaking and Listening 20
in everyday contexts and a variety of domains (Council 15
of Europe 2015) is observed in the score results. On 10
the other hand, findings also show that a proportion 5
of learners are still at A1/A2 levels (17% for Listening,
0
24.7% for Reading, 26.8% for Writing and 9.7% for Listening Reading Writing Speaking Not Sure
Speaking) and therefore below average compared to the
target attainment levels of the English Language Syllabus
for Secondary Schools (attainment levels 45 aligned
School sector profiles
to B1/B2ontheCEFR)(English Language Resource Figures 4 and 5 display the distribution of CEFR levels for
Centre2007). each of the four skills by school sector for Primary and
Secondary learners respectively. It is worth reiterating
Compared to the results of the European Survey on
that a representative sample of learners from the different
Language Competences (ESLC) (European Commission
school sectors was included in the project. The figures
2012), in which 60% of Secondary learners were reported
show some differences amongst the school sectors in
to achieve B2 level on the CEFR using the global average
terms of learner performance; for example, for the skill of
of the three skills of Reading, Listening and Writing
Listening at Primary level, approximately 67% of learners in
(Jones 2013), the findings from the benchmarking project
State schools are at the A levels on the CEFR whereas these
show that approximately 50% of learners are at Levels
proportions are comparatively lower for Church (48%)
B2 and above, with 33% of Form 4 Secondary learners
and Independent (23%) schools. However, in the skill
achieving Level B2 and a further 15% achieving C levels
of Writing, Primary learners in State schools outperform
when the average of the three skills is used. Focusing on
learners in both Church and Independent schools, with a
the skill of Speaking, results show that approximately
comparatively higher proportion of State learners at the
62% of Secondary learners have achieved CEFR Level B2
B1/B2 levels (53%) compared to Independent (51%) and
andabove.
Church (37%) schools.
The Secondary score results were cross-checked against
For the skill of Listening at Secondary Form 4, there
learners self-reports of their weaknesses in English as
is a higher proportion of learners at C1/C2 levels in
well as parents and teachers perceptions. Similarly to the
Independent schools (49%) compared to Church (33%)
Primary results, Figure 3 shows that learners, parents and
and State (24%) schools. However, at the CEFR B levels,
teachers all unanimously believe Writing to be the learners
these proportions are much more similar across school
weakest skill but unlike Primary results, this perception
sectors, with 45%, 55% and 58% of learners achieving
matches the score data where, compared to all other skills,
B1/B2 levels at Independent, State and Church schools
Writing had the highest proportion of learners at pre-A1
respectively. For the skill of Speaking, a similar proportion of
and A1/A2 levels (26.8%).
learners at State (22%) and Church (27%) schools achieve
Figure 3 also shows that similar to the Primary learners, C1/C2 levels on the CEFR. This proportion was highest for
a large percentage of Secondary learners (40%) selected Independent schools, with 64%.
the not sure option when asked to identify their
It is important to draw attention to the observed
weaknesses in English. This is somewhat concerning;
variations within each school sector where a distribution
Secondary school learners should have more meta-
of performances is observed from A levels to C levels for
cognitive awareness about their language ability and as
State, Church and Independent schools. While there is
a result more capacity to self-assess. Engaging learners
a trend of higher performance at Independent schools
in awareness-raising activities in their lessons can help
compared to Church and State schools, no individual school
learnersidentifytheirstrengths and weaknesses and
sector is associated with a clustering of only high-or only
in turn, allow them to set appropriate language goals
low-performing learners.
forthemselves.

UCLES 2017 UNDERSTANDING LANGUAGE LEARNING IN MALTA

 8|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Figure 4: Distribution of CEFR levels by school sector (%) Writing at Secondary level (Table 5) where 49% of girls are
Primary at B2 level compared to 31% for boys and approximately
11% of girls are at the C levels compared to 4% for boys.
Independent
Speaking

Church
One tentative explanation for the observed differences in
State
performance of boys and girls relates to the differential
Independent rate of literacy development in the two groups, which
Writing

Church has been supported in previous research (Maccoby and

State Jacklin 1974). Another possible explanation put forward by
Independent Chavez (2000) relates to the differences in the approach
Reading

Church to learning that boys and girls have been found to have.
State There is a tendency for boys to focus more on competition
Independent and achieving mastery goals, resulting in less willingness
Listening

Church to take risks in language classes (e.g. experimenting with

State the language) whereas girls emphasise co-operation and
0 10 20 30 40 50 60 70 80 90 100
collaboration in their language learning, which is associated
with better learning outcomes. While differences in
Pre-A1 A1 A2 B1 B2 C1+
performance of boys and girls are generally found to be
small and not a cause for concern, it is nevertheless an
area that can be monitored by MEDE to ensure that all
Figure 5: Distribution of CEFR levels by school sector (%) learners receive the kind of support that they need in their
Secondary languagedevelopment.

Independent
Speaking

Church
State
Primary versus Secondary
Independent performance
Writing

Church
The shift in language levels can be seen in the notable
State
decrease in the proportion of learners at CEFR A levels in
Independent
Primary and a higher proportion of learners achieving the
Reading

Church
higher B and C levels in Secondary stage.
State
Independent The CEFR levels can roughly be seen as representing three
Listening

Church important thresholds for success in a bilingual context:

State Level B1 represents low intermediate learners who are
0 10 20 30 40 50 60 70 80 90 100 independent users of the language, but in a limited range
of familiar contexts; Level B2 is widely recognised as the
Pre-A1 A1 A2 B1 B2 C1 C2
threshold for high intermediate, independent users of a
language who can function in a second language in a range
Gender profiles of familiar and unfamiliar contexts; Level C1 is seen as the
level at which advanced language proficiency is displayed.
Performances by boys and girls were also compared (see
Tables 1 and 2 for Primary and Secondary levels). Results The high proportions of Secondary learners at B2 and
show that girls systematically outperform boys on all skills, C1+ levels generally suggest readiness for independent
with a higher proportion of boys at the lower CEFR levels functioning in a variety of contexts for a large percentage
compared to girls and an associated higher proportion of of Secondary learners. However, there is still a considerable
girls at the higher CEFR levels compared to boys. These proportion of Secondary learners who are at the A
differences are generally small, except for the skill of levels (Listening=17%, Reading=24.7%, Writing=26.8%

Table 1: Distribution of CEFR levels by gender (%) Primary

Skill Gender Pre-A1 A1 A2 B1 B2 C1+

Listening Male 1.8 8.3 44.3 25.2 14.2 6.2

Female 1.5 8.6 35.7 28.4 15.1 10.8
Reading Male 2.4 35.4 43.9 15.2 2.7 0.4
Female 1.5 29.7 45.8 17.4 5.2 0.4
Writing Male 1.6 6.3 51.6 39.7 0.9
Female 3.5 46.0 48.8 1.7
Speaking Male 5.3 11.8 30.5 35.3 17.1
Female 3.6 11.4 31.4 34.3 19.3

UCLES 2017 UNDERSTANDING LANGUAGE LEARNING IN MALTA

CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|9

Table 2: Distribution of CEFR levels by gender (%) Secondary

Skill Gender Pre-A1 A1 A2 B1 B2 C1 C2

Listening Male 0.2 2.6 17.3 35.3 19.7 20.0 4.9

Female 0.3 1.8 12.8 33.4 21.3 24.9 5.7
Reading Male 0.9 4.4 25.1 28.4 33.8 5.7 1.6
Female 0.8 1.8 17.8 27.3 41.0 9.9 1.4
Writing Male 3.8 9.3 27.5 23.6 31.0 4.4 0.3
Female 1.0 2.0 11.4 25.5 49.0 11.1
Speaking Male 0.6 11.5 31.0 35.6 8.6 12.6
Female 2.8 4.7 25.8 31.5 16.0 19.2

and Speaking=9.7%).3 While it is expected for a small Respondent profiles

proportion of the population to not go beyond the A levels,
An overview of the distribution of learner, parent and
about one quarter of the learners have yet to become
teacher respondents by school sector and gender is
independent users of English, which may require further
provided in Table 3 and shows that the project has
investigation. The NCF emphasises the need for a high level
been successful in including a representative sample of
of proficiency in the bilingual context of Malta and for a
survey respondents across the different school sectors.
variety of social, educational and employment purposes:
While there is a gender balance in the learner data, the
in Malta it is considered paramount that young people are
distribution for parent respondents (over 82% female)
enabled to develop a high level of proficiency in English
suggests that mothers are primarily responsible for
for active participation in society, success in education,
overseeing their childrens education. The teacher data
employment and personal life, and the expression of
also shows that the majority of teachers (84%) are female.
identity (Ministry for Education and Employment 2015:6)
This might suggest the absence of male language role
as well as in meeting the competitive demands of a
models for boys, which could also explain why boys were
globalised economic environment (Ministry for Education
outperformed by girls.
and Employment 2014:41). These requirements roughly
translate into a minimum English language level of B2
Table 3: Distribution of survey respondents by school sector and
across the four skills for Secondary school leavers. It is gender
recommended for MEDE to take into consideration the
language levels established in the benchmarking project Parti- Grade School sector Gender
cipants
and the proportion of Secondary learners who have not State Church Inde- Male Female
yet progressed beyond the A1/A2 levels on the CEFR pendent

in reviewing and/or setting of language proficiency and Learners P 40% 45% 15% 56% 44%
attainment levels. S 51% 37% 12% 50% 50%

Parents P 36% 47% 17% 15% 85%

Attitudinal and background factors S 42% 50% 8% 18% 82%

Teachers P 57% 31% 12% 15% 85%

In order to better understand the context of learning and S 71% 22% 7% 16% 84%
teaching in Malta, learners, parents and teachers were
P=Primary, S=Secondary
asked to complete surveys which focused on key variables
that might affect performance and levels of achievement.
Exposure to English
These included learners level of exposure to English at
It is a widely accepted premise both in the theoretical and
school and outside of school, learner attitudes towards
practical language learning domains that exposure to a
learning, and parental support and encouragement
foreign language within the learning environment and/or
amongst others. The results from the different surveys that
the home environment plays a positive role in learning. The
touch on similar themes are pulled together and integrated.
ESLC (European Commission 2012) found that greater use
As far as possible, results for Primary and Secondary levels
of English, by both learners and teachers, in the classroom
are presented together for ease of comparison. The key
was positively related to language ability (Jones 2013).
findings which emerged as significant4 and meaningful are
The ESLC also found that parents knowledge of the foreign
summarised in the next sections.
language being studied and learners exposure to it in

3
The results of candidates at pre-A1 and A1 levels were cross-checked against the special circumstances information provided by MEDE; approximately 15% of these candidates
were listed as special circumstances candidates for the different skills.
4
For the chi-square analysis, the critical value to determine statistical significance was rr <r.005. Standardised residuals were also computed to identify which responses were
contributing to the test of significance. Values larger than +2.0 or -2.0 were used to determine whether any particular response was showing a meaningful difference between
groups.

UCLES 2017 UNDERSTANDING LANGUAGE LEARNING IN MALTA

10|ISSUE 65 / FEBRUARY 2017
the home or community was positively related to learner
outcomes (Jones 2013). Therefore, in the surveys, we
investigated the extent to which learners are exposed to
English either in their home environment or at school.
Language spoken at home
When learners were asked to select their first language and
language spoken at home, the distribution of responses was
very similar. Therefore, only the results of language spoken
at home are presented here. For Primary learners, there was
a split between Maltese (approximately 50%) and English
or bilingual (47%) with a small percentage (3%) selecting
other. The distribution was more skewed for Secondary
learners; 71% of respondents selected Maltese as their first
language, 27% selected English or bilingual and a small
percentage (2%) selected other5 which suggests the
presence of linguistic minorities in the Maltese educational
context. The parent reports of language spoken at home
match the learner response data.
These findings suggest very different levels of exposure
to English in the home environment, which can influence
performance in schools; a finding that is illustrated in the
following teacher comments:
Parents have limited knowledge in English language usage.
This is reflected in their children. (Primary teacher, State
school)
Due to the fact that most children have a Maltese speaking
background most of them find difficulties to communicate in
English all the time. (Primary teacher, State school)
English language levels of parents/guardians
Learners were also asked to indicate how well their
parents/guardians spoke English on a Likert scale from
cannot speak English to can speak English very well,
with a not sure option. The results indicated that a strong
majority of parents can speak English either very well or
moderately well. However, as pointed out by the teachers
in the previous section, a small minority of parents cannot
speak English and 723% cannot speak English very well.
When cross-checked with parents self-reports of language
proficiency, similar results were found. These findings
once again show the disparity in learners English language
experiences in the home environment.
English language use outside of school
The strong majority of Primary and Secondary learners
report using English very often or sometimes when they
watch TV or films, use the internet, talk to tourists and read
books. However, the distribution of responses for activities
such as talking in English with family and friends is more
varied, with a much larger proportion of learners reporting
never speaking English with family, friends, or other people
in their home towns. Note that in order to triangulate
the data, these responses were checked against parents
reports of how often their children engage in different
activities in English.
5
The listed first languages are Chinese, Filipino, French, German, Greek, Italian, Libyan, Nigerian, Serbian and Spanish.
6
These comments refer to both Primary and Secondary students.
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
It is also interesting to note that when looking at pattern of
responses for watching TV or using the internet, there were
no statistically significant differences between responses
across school sectors. However, in terms of English
language use with friends and family, those in Independent
schools were more likely to select speaking English very
often compared to Church and State schools. This may
reflect a different milieu, with learners in Independent
schools having family and friends with whom they can
speak English whereas learners at State schools may
not have the same access to English speakers in their
immediate social circle. This differential exposure to English
outside of school is likely to have an impact on performance
at school level and can contribute to widening the
achievement gap between learners. Approximately 20%
and 10% of Primary and Secondary learners respectively
reported taking private English language lessons. When
parents were asked to provide reasons for why their child
takes private lessons, comments generally fell in the
following categories6:
improving Reading
improving English (amongst other subjects such as
Maths and Maltese)
learning issues (e.g. dyslexia) and need for more support
for the purpose of revision and exam preparation.
English language use in the classroom
The extent of English language use in the classroom was
elicited on a 4-point Likert scale from never to very
often from both learners and teachers. Findings show a
clear trend of high English language use by the teachers,
with decreasing levels of use by learners as they talk
to the teacher and as they talk to each other, and also
demonstrate a high level of consistency between what is
independently reported by teachers and learners, which
strengthens the findings from these surveys.
When looking at pattern of responses across school sectors,
no statistically significant differences were found for English
language use by teachers (as reported by learners). However,
both Primary and Secondary learners in Independent schools
were more likely to select speaking English to their teacher
and other learners very often compared to Church and State
schools. This may reflect a different approach to English
language instruction in the different school sectors and/or
differences in the characteristics/profile of the learners who
attend these schools.
Teachers report switching to Maltese only occasionally
during English classes and their open comments suggest
that code-switching practices take place mainly due
to lower English ability levels of some learners or for
classroom management purposes.
90% of Primary and Secondary learners reported studying
other school subjects in English, with similar results for
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES
English language use as above. The trend of lower English
language use by learners can also be explained by an open
comment by one of the teachers, which sheds light on the
complex socio-cultural roles of English and Maltese which
can manifest themselves in the classroom:
English is still regarded as an artefact [sic] in which only snobs
take pride. Most of the time students address me in Maltese
I either switch to French threatening them that I cant follow
because were not speaking Maltese during class. Sometimes
I am successful but most of the time I am bugged by their
insistence to switch to their native tongue typical answer
but Im Maltese so why do I need to learn English? They are
not concerned by any answer which I provide and this may be
possibly a backlash from our Maltese colonial attitude. Will
have to consult Edward Said about this.
Another possible explanation could be that English
lessons are too teacher dominated and thus limiting
learners opportunities to use English and/or to speak
English to other learners. Earlier findings suggested that
paired and group activities do not occur as frequently as
other activities in the classroom. Given that no classroom
observations were carried out, this explanation is only
tentative and would need to be further explored in the
second phase of the project.
Opportunities to use English outside of the
classroom (in schools)
Teachers were asked to indicate the extent to which their
schools offer opportunities for learners to use English
outside of the classroom (English clubs, English books,
films, etc.). Results show that schools generally offer such
opportunities although there is still a 10% disagreement
rate. Note also that statistically significant differences were
observed for teachers from Independent schools selecting
strongly agree for this statement compared to State and
Church schools. This suggests that Independent schools
may offer more English language support compared to
other school sectors. These out-of-class activities create
not only opportunities to use and practise English, but
can also lead to the increased popularity of English and
enhance learner motivation and positive attitudes towards
English. This finding also links in with the previous section
where it was shown that learners in Independent schools
were more willing to speak in English in the classroom.
These results may reflect differences in the school sectors
such as background characteristics of learners enrolled
in Independent schools. It is also likely that Independent
schools emphasise these features such as additional
English language support to differentiate their schools
from the other sectors, in order to attract parents to their
schools.
Attitudes towards learning English
The academic literature on second/foreign language
acquisition has indicated that learners attitudes towards
learning a language and the extent to which they perceive
the language to be useful can influence learner behaviour,
both in terms of the amount of effort exerted on language
learning and the extent to which they persist with learning
UCLES 2017
ISSUE 65 / FEBRUARY 2017|11
it when it becomes difficult (Csizr and Drnyei 2005,
Gardner 1985, Oxford and Ehrman 1993). Learners beliefs
about their own capacity to learn often referred to as self-
efficacy have also been found to be positively associated
with academic outcomes (Mills, Pajares and Herron 2006,
Multon, Brown and Lent 1991). These constructs were
elicited in the surveys.
Several statements in the surveys were designed to shed
further light on the motivational factors that may be
influencing learners language learning behaviour, such as
instrumentality or milieu (Gardner 1985). Instrumental
motivation refers to the utilitarian benefit or incentives
associated with learning a language, such as getting a
job or travelling, whereas the motivational dimension of
milieu refers to the influence of learners immediate social
environment (i.e. parents, family and friends), excluding
teachers, in shaping their attitudes to learning. Learners
who perceive family support for language learning are more
likely to persist with it and more willing to work harder at
it (Colletta, Clment and Edwards 1983, Gardner 1985).
Although these two dimensions of motivation are not
necessarily causally linked to learner outcomes because
other variables such as instructional quality, learning
opportunities and learner ability all play a crucial role, the
results nevertheless provide an indication of how much
effort learners are willing to place on learning.
Parental attitude towards education and learning, as well as
the extent of support and encouragement, have also been
identified as having an influence on their childs level of
attainment (see Bartram 2006, Gu and Saville 2012) and
were therefore included in the surveys.
Teacher attitudes towards a language can influence learner
behaviour in terms of effort expended on learning and
learner success (Csizr and Drnyei 2005, Drnyei 2003,
Horwitz 2001). Teacher beliefs in learners capacity to learn
can also influence or reinforce learner attitudes towards a
language (Bandura 1977, Mills, Pajares and Herron 2006,
Multon et al 1991). Therefore, the value placed on learning
English by the teachers and attitudes towards learning
English were considered important constructs to evaluate.
Key survey findings on these different constructs are
summarised below.
Learner attitudes and self-efficacy
Approximately 90% of Primary and Secondary learners
agreed or strongly agreed that (a) they like learning English,
(b) learning English is important to them, (c) they believe
they can learn English and (d) they know how to improve
their English, all of which demonstrate positive attitudes
towards English and high levels of self-efficacy in learning
within the Maltese educational context.
These results are confirmed in both the teacher surveys,
where 94% and 86% of Primary and Secondary teachers
agree or strongly agree that their learners like learning
English, and in the parent surveys, where 95% of Primary
parents and 90% of Secondary parents agree or strongly
agree that their child (a) enjoys learning English, (b) is
UNDERSTANDING LANGUAGE LEARNING IN MALTA
12|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

motivated to learn English and (c) believes he/she can learn Note that a statistically significant difference was observed
English. Over 90% of parents also strongly agreed with the for the pattern of responses for the statement I enjoy
statement It is important to me that my child learns English, English lessons at my school across school sectors, where
which further demonstrates the value placed on learning strongly agree was selected more than expected in
English in the home environment. responses of Independent school learners compared to
Church and State schools, with the widest gap observed
When comparing the pattern of responses for the
between Independent and State schools. This was the case
above statements for Primary and Secondary learners,
for both Primary and Secondary data.
a statistically significant difference was found for the
statements pertaining to enjoyment of learning English,
Learner motivation
importance of learning English and motivation to learn, with
The results in Table 4 show that learners recognise the
a smaller proportion of learners and parents opting for the
functional role and importance of learning English e.g. for
strongly agree option for Secondary learners compared
employment and university entrance purposes, with the
to Primary learners. This might suggest a shift in learners
majority of both Primary and Secondary learners opting for
attitudes and/or enthusiasm towards language learning
the very important option when evaluating these different
(or learning in general) as they grow older. This is in line
reasons for learning English. Some of the open comments
with previous research (e.g. Lepper, Corpus and Iyengar
further illustrate the value placed on English:
2005:192) that positive academic beliefs and behaviours
gradually erode as children progress through the school I am learning English because I wish to be an Accountant and
system. Research from other educational contexts points to it requires English. (Secondary learner, State school)
the more fun-oriented nature of lessons in Primary school,
For an amazing future. (Secondary learner, Independent
where teaching can be done through games, whereas in
school)
secondary school, learners are dealing with more cognitively
demanding tasks and there is the added dimension that Everything. English is just very very important. Easy to
learners at this age are developing their sense of self, communicate with people who can speak it. (Secondary
which can affect their attitude and motivation (Docherty, learner, State school)
Gratacs Casacuberta, Rodrguez Pazos and Canosa
High levels of importance were also placed on other
2014:8). Secondary learners may also start to think more
reasons, such as understanding English films, TV series and
seriously about the exams and their future, which could
programmes. Open comments included understanding
potentially detract from the joys of learning. The majority of
English music, talking to English people and using English
Primary (92%) and Secondary (81%) learners nevertheless
for social media, which indicate cultural interest in English.
reported enjoying English language lessons at their schools,
which suggests that the schools are successful in creating Learning English to please parents/guardians was not
a positive environment for English language learning. The perceived to be as important as the other reasons, although
following comment by a Primary learner captures this: over 60% of Primary learners still regarded this as very
important. Not surprisingly, Secondary learners attributed
1. My teacher love English. 2. I love English. 3. My friends
far less importance to pleasing parents, suggesting that
love English. 4. Every one love English. (Primary learner,
Stateschool)
parental influence is becoming less important as learners
become older.

Table 4: Reasons for learning English and perceived importance(%)

Learning English will ... Grade Very Important Important Not Very Important Not Important At All Not Sure

... help me get into a good university. P 81% 13% 3% 1% 2%

S 77% 20% 2% 1% 1%
... help me get a good job. P 83% 14% 1% 0% 1%
S 81% 17% 1% 1% 0%
... make it easier for me to travel to P 68% 21% 6% 2% 3%
other countries.
S 74% 23% 2% 0% 1%
... make it easier for me to talk to P 70% 20% 6% 1% 3%
people who dont speak my first
language (either online or in person). S 77% 20% 1% 0% 1%

... help me use the internet to get P 69% 22% 5% 2% 3%

information.
S 70% 26% 3% 1% 0%
... help me understand English films/TV P 72% 19% 5% 2% 2%
series/programmes.
S 69% 25% 5% 1% 1%
... please my parents/guardians. P 65% 22% 6% 2% 5%
S 26% 33% 29% 8% 4%

P=Primary, S=Secondary

UCLES 2017 UNDERSTANDING LANGUAGE LEARNING IN MALTA

CAMBRIDGE ENGLISH : RESEARCH NOTES
The strong majority of Primary and Secondary teachers
(over 95%) agreed or strongly agreed that learning English
is important to their learners. Parents of Primary and
Secondary learners also place great importance on learning
English for a variety of reasons but particularly for university
entrance and employment purposes, with 98% selecting
important or very important in their evaluation. Parents programme at their school.
open comments generally referred to the instrumental
role of English for education and jobs, the ability to talk to
relatives and family members and English as a universal
language. Some illustrative examples are presented below:
It is imperative to learn English even for daily use e.g. to
understand what is going around you Euro news and CNN.
Ingredients on products are not in Maltese. Core subjects like
physics, accounting, economics, sciences etc. are to be studied
in English so how can you understand a subject if you do not
hold a strong understanding and communication in English?
(Mother of Primary learner, State school)
It is a universal language and our second official language.
(Mother of Primary learner, State school)
I work in a bank and all communication through emails is
always in English though we are Maltese. (Mother of Primary
learner, State school)
Generally speaking, the results show great importance
given to English from the perspective of all major
stakeholders learners, parents and teachers in the
Maltese context with evidence of high motivation for
learning English for a variety of reasons.
Parental support and encouragement
In terms of the motivational dimension of milieu, the degree
of parental support was elicited in the learner surveys.
While 75% of Primary learners agreed with the statement
My family encourages me to learn English, the disagreement
rate (compared to other statements) was relatively high
at 16%. This disagreement rate was much lower for
Secondary learners at 5%; a tentative reason might be that
Secondary learners are at a more crucial stage in terms of
taking English exams, applying to university or entering
the employment market where English plays a more
important role, and as such learning English is more actively
encouraged by families.
The extent of parental support in improving English
outside of school (e.g. by helping with homework, or
providing opportunities to use English) was elicited in
the parent surveys. 90% of Primary parents reported
providing this support sometimes or very often, with 2%
reporting never providing such support. The proportion of
Secondary parents providing this support was lower, with
75% selecting sometimes or very often and 10% never
providing such support.
The extent to which parents actively participate in their
childrens English education (participating in school
activities, talking to teachers about progress, etc.) was
also elicited from teachers. 35% of Secondary teachers
reported that parents did not actively participate in their
childrens education compared to 12% for Primary learners.
UCLES 2017
ISSUE 65 / FEBRUARY 2017|13
This is also confirmed in the parents survey results for the
statement I know a lot about my childs English programme
at school, which received relatively high disagreement
rates of 17% and 25% for Primary and Secondary learners
respectively. It may therefore be necessary for schools to
allocate additional time to inform parents about the English
Teachers open comments suggested that the degree of
parental encouragement can vary widely for a number
of reasons, including parents educational background.
Teachers also touched on the complex linguistic milieu
in Malta where the choice of language has strong social
connotations which can influence the extent of family
encouragement in learning English:
Some parents tend to help and encourage their children, others
just dont care or they are not able to do so due to social and
educational issues. (Secondary teacher, State school)
There is a rift in the country between the Maltese speaking
and the so called tal Pepe [the English-speaking Maltese],
some learners want to distance themselves from their Maltese
heritage in an effort to appear more forward thinking, others
insist on their heritage and resist English influence on their
lives. This all depends on the parents outlook and how they
treat English in the homestead. Its a complicated linguistic
milieu. (Secondary teacher, Independent school)
Schools can also ensure that they have established a clear
language policy which could be used to guide and support
parents in the most effective ways to help their children
learn English. The absence of strong parental engagement
with learning as children grow older was touched on in
the teachers open comments, where it was noted that
parents can still play an important role in their childrens
education and that they should continue their support in
Secondaryschool:
I believe that we need to educate parents to involve
themselves more in what their children are doing as at times
they believe as soon as they start attending Secondary
school they no longer feel the need to follow on them doing
homework and at times some students tend to feel lost or fall
behind. (Secondary teacher, State school)
Teacher attitudes, pedagogic
practices and professional
development needs
In order to better understand the context of learning and
teaching in Malta, and also, identify teachers professional
development needs, teachers at the two key school
stages Primary Year 5 and Secondary Form 4 were
invited to participate in the benchmarking project on a
voluntary basis. The Senior Management Teams (SMT) at
different schools, the Malta Union of Teachers and MEDE
informed schools and teachers about the project and
encouraged participation. A total of 132 teachers completed
the teacher surveys. The following sections present the
findings from the surveys. The data for all Primary and
Secondary teachers is generally presented together, except
UNDERSTANDING LANGUAGE LEARNING IN MALTA
14|ISSUE 65 / FEBRUARY 2017
for statements for which patterns of responses were
significantly different for the two school stages. Where
relevant, data from parent and learner surveys which relate
to classroom activities and practices and their attitudes is
integrated to provide a more comprehensive picture of the
learning environment.
Attitudes to teaching English
The survey results suggest that teachers have a positive
attitude towards teaching English, with all but 1.4% of
teachers reporting that they enjoy teaching English and are
confident in their ability to plan appropriate English lessons
for their learners, all of which can positively influence
learning outcomes in the classroom.
The results of both learner and parent surveys suggest
that teachers are generally successful in creating a positive
environment for English language learning, as the majority
of learners report enjoying their English lessons at school.
The following learner comment illustrates this:
I believe I have a good English teacher. She takes pride in her
work and explains well. She gives us hints and ideas to do
better. (Secondary learner, State school)
These results are confirmed in parent surveys, with over
90% of parents agreeing or strongly agreeing that their
child (a) enjoys his/her English lessons at school, (b)is
making good progress in English and (c) likes his/her
English teacher.
Key trends in pedagogical practices
The surveys included a section on teachers pedagogical
practices, and teachers were asked to report on the
frequency of the listed activities which represent
examples of good practice in the classroom throughout
the school year.
Overall, the listed activities are reported to occur in some
or most lessons, which is a positive finding. The four
most frequently occurring practices are those related to
assigning homework, linking the homework to classroom
learning, reviewing learner homework as well as checking
understanding of topics. Only 35% of teachers report
explicitly stating the learning goals and objectives of the
classroom in almost every lesson and a smaller percentage
(29%) report providing a short summary of the previous
lesson. These activities can direct learners to what they
need to focus attention on in the classroom and to also
help them recognise the link between different lessons,
and should ideally occur in every lesson. Learners will not
be able to develop autonomy if they are not clear on the
objectives of their lessons. Results also show that only
about 10% of teachers use paired/small group work in
almost every lesson. It is therefore not surprising that more
highly learner-centred activities such as asking learners to
suggest topics for the classroom or to help plan classroom
activities were reported to occur far less frequently in
the lessons. The above findings suggest that classes may
benefit from more activities that promote self-regulated
learning. This is also in line with areas of prioritisation in the
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
NCF, which puts emphasis on promoting learner autonomy:
given that one of the greatest effects on learning is when
learners become their own educators (Hattie 2012), a
pedagogy that cultivates learner autonomy should be
cultivated (Ministry for Education and Employment
2015:30). On the other hand, as noted by one of the State
Secondary teachers, time restrictions and the demands of
the curricula may be a limiting factor:
Although we have 6 lessons per week we have to prepare
students for 2 O Levels (Lang & Literature) and we are
restricted with time so there isnt enough time for students to
suggest topics theyd like to discuss. By the time one sets up
the laptop and does the class correction there isnt much time
left for creativity etc. (Secondary teacher, State school)
The impression of time restrictions may be due to the
shortening of lesson times, which may also be compounded
by the recent introduction of co-ed in all State schools
which may have led to new challenges for some teachers.
The statement Students work in groups based upon their
abilities, where only 22% of teachers report incorporating
this practice in almost every lesson or in most lessons,
suggests that teachers may not be recognising the
importance of group work not only as an opportunity
to practise speaking but also to talk about learning.
Language is the tool learners use to think about a subject
and talk about their thinking, which leads to cognitive
development. When the response to this statement is
combined with the high disagreement rate of 30% for the
statement I find it easy to adapt lessons to cater for different
student abilities in the same class, the findings suggest that
teachers could benefit from more support in the area of
differentiated instruction. This is particularly important in
the Maltese educational context where the wide range of
ability in the observed score data as well as the striking
differences in levels of exposure to English would most
likely translate into mixed-ability classes, and a key
aspect of differentiated instruction is how to strategically
group learners for different purposes to improve
learningopportunities.
The above findings are restricted to teacher self-reports,
and while classroom observations would have greatly
enhanced the robustness of the findings, this was not
possible in the first phase of the project. Nevertheless, in
order to address this limitation, some of these teaching
practices were repeated in the learner surveys as a different
approach for triangulating the data.
Table 5 shows the learner statements paired with the
associated teacher statements and the distributions of
responses for the frequency of occurrence of different
activities. Results generally show comparability across the
two data sets, thus increasing confidence in the findings.
One of the most noticeable differences in the table is the
extent to which learners work on their own in completing
classroom tasks, with learners reporting it to happen more
frequently than teachers. While learners can certainly
benefit from focused individual work, increasing well-
planned, paired and group activities can enhance learner
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|15

Table 5: Triangulation of teaching practices

Frequency of English use in classroom Source In Almost Every In Most In Some Never or N/A
Lesson Lessons Lessons Hardly Ever

The homework is useful for learning English. Learners 65% 21% 9% 4% 1%

I link the English homework to what Teachers 76% 20% 2% 0% 2%
students are learning in class.
Our teacher gives us English homework. Learners 65% 26% 9% 0% 0%
I give my students English homework. Teachers 55% 39% 4% 2% 1%
Our teacher tells us what the learning goals Learners 56% 23% 17% 4% 2%
of the lesson are.
I explicitly state the learning goals/ Teachers 35% 40% 22% 2% 0%
objectives/outcomes to the students.
At the beginning of the lesson, we revise Learners 28% 30% 28% 11% 3%
the previous lesson.
At the beginning of the lesson I present a Teachers 29% 33% 37% 2% 0%
short summary of the previous lesson.
The teacher asks us to think about how well Learners 42% 27% 18% 10% 3%
we are learning and how we can improve.
Students evaluate and reflect upon their Teachers 27% 34% 31% 6% 2%
own work.
I work on my own in completing classroom Learners 44% 31% 20% 3% 2%
tasks.
Students work on their own in completing Teachers 15% 37% 41% 6% 1%
classroom tasks.
The teacher walks around the classroom Learners 38% 28% 24% 9% 2%
and helps me while Im working.
I work with individual students. Teachers 14% 29% 45% 10% 2%
I work with other students (in pairs or Learners 23% 25% 39% 10% 2%
groups) in completing classroom tasks.
Students work in pairs or small groups in Teachers 10% 29% 60% 2% 0%
completing classroom tasks.
The teacher gives us a test or quiz to check Learners 22% 22% 38% 16% 3%
our learning progress.
I administer a test or quiz to assess student Teachers 10% 31% 53% 5% 2%
learning.

interaction and opportunities to use English, and also help
promote learner autonomy.
Use of textbooks and technology in the classroom
90% of teachers report using English textbooks in their
lessons, the majority of whom find them useful (84%).
Nevertheless, textbooks are not the main focus of most
lessons according to teachers, and over 98% of teachers
use additional material/resources to supplement the
textbooks and also express confidence in their ability
to create these materials. Teachers open comments on
textbooks suggested that some textbooks may be too old,
not suitable for all ability levels, not necessarily related to
end-of-year exams and lacking practice materials, which
explains why teachers frequently need to supplement the
textbook with additional materials something which they
do not always find easy due to huge teaching loads and
limited time. These findings suggest that teachers may
benefit from further training in textbook adaptation.
Learner and teacher perceptions of technology and its
usefulness were also elicited in the surveys. Technology
is reported to be used in most or almost every lesson by
the majority of teachers. There is also a strong consensus
(98%) amongst teachers in the usefulness of technology
for learning English and high levels of confidence in their
ability to use technology for teaching English. Teachers
also unanimously agree that their learners like using
technology in the classroom, all of which highlight the
important role that technology is increasingly playing in
educationalcontexts.
Technology is generally viewed positively and sometimes in
contrast to textbooks:
Technology makes lessons more attractive and students
participate more, cooperate with each other, and learn through
exploration. Its a resource which [makes] students interact
more than when using textbooks. (Primary teacher, State
school)
Other comments referred to the complementary use of
different resources: I tend to be very eclectic in my choice
of material, usually moving between basic textual resources
and more digital ones. Another comment referred to the
usefulness of audio books and exposing learners to poems
read out by their actual creators like Ian McEwan. One teacher
believed blogs to be very useful for learning, subject to
teaching loads.
However, the issue of lack of technological access or few
opportunities for use were listed as problematic, and a

UCLES 2017 UNDERSTANDING LANGUAGE LEARNING IN MALTA

16|ISSUE 65 / FEBRUARY 2017
perceptive comment by a teacher also questioned the value
of (current uses of) technology in enhancing learning:
Technology is yet to be introduced into the classroom in a way
that actually aids the acquisition of language. Interactive white
boards only serve to reiterate outdated concepts of teacher
centred practice. Learners need the technology, not us. iPads
are also meaningless in terms of providing linguistic acquisition
for all, their interface is simple, but it lacks the input options
for learners to input language in ways deemed necessary by
the NCF. (Secondary teacher, Independentschool)
The observation that education plus technology does not
necessarily equate to better learning is echoed in a recent
Organisation for Economic Co-operation and Development
(OECD) (2015) report on Students, Computers and Learning,
where analysis of the PISA data suggested that where
technologies are used in the classroom, the impact on
performance is mixed, at best (2015:15). An interpretation
offered by OECD was that building deep, conceptual
understanding and higher-order thinking requires intensive
teacherstudent interactions, and technology sometimes
distracts from this valuable human engagement. While
the report is based on student achievement data in
reading, mathematics or science, the findings certainly
hold relevance for language learning. These are important
issues to consider in terms of integrating technology in
the classroom, and efforts should be made to incorporate
pedagogies that make the most of technology as adding
21st century technologies to 20th century teaching practices
will just dilute the effectiveness of teaching (Organisation
for Economic Co-operation and Development 2015).
It is therefore useful to consider how technology can best
be incorporated in Maltese educational settings in ways
that can increase opportunities to communicate in English
and improve the provision of feedback. It is also important
to offer professional development support to teachers in
using these technologies in a linked-up fashion and in a
manner that best supports the provision of individualised
learning. Technologies can also have great potential in
assisting teachers with differentiated instruction.
English literature in the classroom
While the above sections focused on general pedagogical
practices in the English classroom, this section focuses
specifically on the use of English literature in the classroom.
This is in light of the emphasis placed on English literature
in the Maltese Learning Outcomes Framework (LOF)
(Ministry for Education and Employment 2015:29) in which
the importance of regularly exposing learners to a variety of
literary texts is highlighted in order to enhance [learners]
sensitivity to language and fire their imagination.
Approximately 90% of teachers believed that using
literature can help learners learn English. While teachers
generally believed that their learners were good at
understanding English literature, there was a 17%
disagreement rate. Teachers positive sentiments towards
teaching English through literature are captured in the
following quotes:
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
If language is not experienced (and enjoyed) through
literature, what would be the point of learning grammar?
(Secondary teacher, State school)
Literature allows students to delve into the literary not factual
world and allows them to widen their imagination. (Secondary
teacher, Church school)
Students understand English literature but the challenging part
is to get them to become critical learners and to analyse and
discuss important themes. (Secondary teacher, State school)
My students really enjoy the literature lessons and it allows
them to explore different aspects of the language. (Secondary
teacher, State school)
The majority of Secondary learners (79%) agree or
strongly agree that using English literature in the
classroom helps them learn English. However, there was
a 15% disagreement rate for the statement I believe I
am good at understanding English literature, which is in
line with the teacher evaluations. A close examination
of learner open comments illustrates learners mixed
feelings towardsEnglish literature, with some enjoying
theselessons:
It is my favourite part of the lesson and it is quite interesting
to ... and understand the thought of ... it. (Secondary learner,
State school)
I really love English literature and I think its amazing how
writers make use of the language and make it sound so lovely.
(Secondary learner, State school)
Other learners, however, find the lessons more challenging
or do not see their value:
I think it is boring + difficult. I dont know why we have to do it
since Im not going to study further on English. I wish it was an
option. (Secondary learner, State school)
I dont think English Literature really helps us in life.
(Secondary learner, Church school)
I personally think its useless, because I only need grammar for
my writing. Its not like Im going to University and write an
Essay on English Literature. (Secondary learner, Church school)
It therefore seems that, unlike learning English, not all
learners attribute as much importance to studying English
literature. These findings may also reflect a recent change in
the teaching and assessment of literature which requires a
more critical response from learners. These perceptions can
nevertheless be changed by teachers through awareness-
raising activities that outline the benefits of using English
literature, e.g. in enhancing English language competence:
Slater and Collie (1987:3) list valuable authentic material,
cultural enrichment, language enrichment and personal
involvement as reasons for using literary texts. Other
reasons include universality, non-triviality, personal
relevance, variety, interest, economy and suggestive power
and ambiguity (Himanolu 2005:54).
Text selection is also a key consideration. Some of the
teacher open comments referred to the need for texts
which are both accessible in terms of language levels but
also appropriate for different age groups.
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|17

Our school definitely is in need of a much wider choice of assessment, whether formative or summative in nature,
literature texts which cater for all abilities. This is especially is that it should improve learning (Carless 2009). LOA
useful and important when teaching literature to the lower involves the collection and interpretation of evidence
tracks. (Secondary teacher, State school) about performance so that judgements can be made about
Finding the right texts which are accessible but not babyish is further language development (Purpura 2004:236). This
a difficulty as my students are 15 years old but their English is approach to assessment requires learners to be involved
below elementary level. (Secondary teacher, State school) in assessment through self/peer assessment as well as by
using the feedback they receive from different sources (e.g.
Using graded readers linked to the relevant CEFR level of
teachers, peers, tests) to decide on what they need to do
the learners is recommended in order to control for the
next. LOA practices have the potential to increase learner
difficulty levels of texts. Selecting literary texts which are
autonomy, motivation and engagement, and as such the
relevant for young adults is also key in motivating learners.
use of LOA practices was investigated in the questionnaire.
Another recommendation from a parent (in an open
The next sections focus on stakeholder attitudes to
comment) was to make literature more accessible through
assessment by bringing together the results of teacher,
arts e.g. theatre and drama.
learner and parent surveys as well as the extent to which
LOA practices are observed in the Maltese educational
Attitudes to assessment and assessment
contexts based on teacher and learner reports.
practices
Assessment has multiple functions in education, including
Learner views on assessment
the measurement of achievement, public accountability
Results in Table 6 suggest a strong link between summative
and providing feedback to learners, and it also tells learners
forms of assessment and learning from the perspective
what we value or what they should pay attention to (Boud
of learners, with the strong majority of both Primary and
2000). Learning-oriented assessment (LOA) represents
Secondary learners attributing an important role to tests
an approach that recognises that the main function of

Table 6: Learner attitudes towards assessment

Statements Grade Strongly Strongly Dont

Agree & Disagree & Know
Agree Disagree

Tests are important because they motivate me to study. P 95% 4% 2%

S 80% 18% 2%
Tests are important because they help me focus on what I need to learn. P 95% 5% 1%
S 87% 11% 2%
I really have to understand the lessons to get good marks in English. P 95% 4% 1%
S 82% 15% 2%
I work harder in class when preparing for a test. P 92% 6% 2%
S 82% 15% 4%
It is clear to me what I am supposed to learn. P 95% 3% 2%
S 84% 10% 5%
It is clear to me what I will be tested on. P 88% 8% 4%
S 82% 12% 6%
It is clear to me how what I am supposed to learn fits in with what I will be tested on. P 88% 6% 6%
S 81% 11% 8%
The feedback the teacher gives me during class helps me perform better at the end-of-year P 92% 5% 3%
exams.
S 87% 11% 3%
The annual exams are related to the work we do in the classroom throughout the year. P 90% 5% 5%
S 89% 6% 4%
Taking English tests helps me understand my strengths and weaknesses. P 88% 9% 4%
S 89% 9% 3%
Judging my own work and/or the work of other students in class is useful for learning English. P 77% 16% 7%
S 72% 21% 7%
I am able to judge the quality of my work and the work of other students in my class. P 75% 15% 11%
S 75% 18% 7%
I worry about taking English tests. P 53% 43% 4%
S 54% 41% 4%

P=Primary, S=Secondary

UCLES 2017 UNDERSTANDING LANGUAGE LEARNING IN MALTA

18|ISSUE 65 / FEBRUARY 2017
in motivating them to study harder, focusing them on
what they need to learn and making them work harder
in class, despite half of the respondents reporting feeling
worried about taking English tests. Note that the proportion
of Primary learners who agree with the statements is
consistently higher than Secondary learners, which might
suggest a shift in attitude towards exams at the different
school stages, which possibly reflects the higher-stakes
nature of tests at higher levels. Although assessment within
Malta has a formative component in providing feedback to
students and parents (e.g. half-yearly exams), some exams
are used for determining the banding/setting of the Primary
and Secondary English classes which may explain why
learners indicate some anxiety about examinations.
Learners also notice the relationship between annual exams
and work that is done in the classroom, and attribute a
strong positive role to teacher feedback in influencing their
performance. There is also evidence of learner awareness of
the diagnostic value of tests, with the majority of learners
agreeing with the statement Taking English tests helps me
understand my strengths and weaknesses. In light of these
positive attitudes, it is important to build a continuous
assessment programme that takes advantage of these
positive attitudes by encouraging student reflection based
on results and helping learners identify next steps in their
learning programme.
The highest proportion of disagreement is observed for
statements relating to the usefulness of self and peer
assessment for learning English as well as learners ability
to carry out this evaluation. The percentage of Primary
(11%) and Secondary (7%) learners selecting the dont
know option in terms of their ability to self/peer assess
confirms earlier findings regarding weaknesses in self-
regulating learning.
Parent views on assessment
Parental attitudes towards different forms of assessment
were also elicited in the surveys. No statistically significant
differences were found between patterns of responses
of parents of Primary and Secondary school learners. It
is interesting to note that the statement It is important
that my child is able to assess his/her own progress elicited
the highest agreement rate from parents, indicating the
value placed on self-assessment and increasing childrens
autonomy from the parents perspective. Similar to learners,
parents also view the diagnostic role of tests in helping
children understand their strengths and weaknesses.
Results suggest a positive attitude towards summative
assessment in increasing motivation and focusing learning,
and the low agreement rate for the statement In general,
preparing for tests will not help my childs English improve
shows that parents also see a link between summative
forms of assessment and learning.
Parents open comments on assessment touched on the
need for more feedback, past papers, model answers
and holistic assessment of English (not limited to testing
Writing). It should be pointed out that summative
assessment of all four skills takes place regularly in both
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
Primary and Secondary stages. English literature is also
assessed at Secondary school and feedback is provided
on all components. Nevertheless, parents emphasised the
need for assessment at more regular intervals (rather than
annually or bi-annually) as an efficient approach in reducing
cramming and test-related anxiety, and encouraging regular
work and studying. There was also evidence of mixed
attitudes towards tests:
In my opinion its important for my child to have tests and
assessments in English to see where she stands. (Father of
Secondary learner, Church school)
I do not believe that exams reflect a childs true capability
or potential in any subject. (Father of Primary learner,
Churchschool)
Teacher views on assessment and LOA practices
Results show that approximately 90% of teachers attribute
a positive and important role to tests in motivating learners
to study, helping them focus on what they need to learn and
improving learning in general. Teachers in general have a
positive attitude towards summative forms of assessment
and annual exams in supporting learning. While the
majority of teachers (83%) believe the annual exams to be
at an appropriate level for their learners, further analyses
of responses by school sector suggested that teachers in
State schools were less likely to strongly agree with this
statement compared to Church and Independent schools.
This might reflect different assessment policies across school
sectors; the State annual exams are set centrally and include
an evaluation process whereby teachers are invited to send in
their feedback, whereas IndependentandChurchschools set
their own school exams.
It is interesting to note that while half of the teachers
believe that learners see summative assessment as jumping
through hoops, the learner results suggested otherwise,
with the strong majority of both Primary and Secondary
learners reporting feeling motivated by tests. One of the
teacher comments refers to the double-edged sword of
summative assessment in not only motivating learners but
also creating test-related anxiety:
Exams do motivate students to study harder but it is also true
that they bring a lot of stress on students as well as parents.
(Secondary teacher, State school)
The majority of teachers (95%) strongly agree or agree
that learners should assess their own work. However,
this proportion is markedly lower for peer evaluation
(58%); this result should be further examined, given the
importance placed on self/peer assessment in the LOA
framework in promoting learner autonomy.
Teachers also report engaging in LOA practices such as
informing learners about the criteria on which they will be
evaluated and providing feedback on learner strengths and
weaknesses following tests. These results were confirmed
in the learners reports of classroom practices and therefore
triangulate the teacher self-reports. A teachers open
comment also referred to alternative ways of assessment in
the classroom with a focus on learning:
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES
I believe that testing and exams should not be the centre of
teaching. Children should be made more aware of the need
to learn English. I can still assess my pupils by giving them
short task to perform such as writing a story or engage in a
discussion. (Primary teacher, State school)
Lastly, the majority of teachers (86%) report developing
their own classroom tests and also express their confidence
in their ability to design appropriate classroom tests. Given
the common practice of test development by teachers, an
assessment literacy course could greatly benefit teachers in
designing high-quality tests that best support learning.
Improvements to English language learning and
teaching
Lastly, learners, parents and teachers were asked to
describe the one thing that they believed would improve
the learning and teaching of English. The most recurrent
themes are summarised below:
providing more resources (textbooks, books, films) in
schools and more opportunities to use English outside of
class (e.g. film clubs, school trips)
implementing full English immersion in classes, with a
consistent language policy across school sectors
teacher training that focuses on bilingual education and
differentiated instruction
motivating learners through extra-curricular activities,
use of educational technologies, integrating literature
with English learning and introducing a wider variety of
activities (e.g. games, videos and quizzes)
increased (and better) use of educational technologies
revising textbooks, making them more interesting and
relevant to the Maltese context
reducing class sizes to allow more teacherlearner
interaction
increase in paired and group work
increased feedback from teachers and monitoring of
learner progress
more regular revisions for exams
less emphasis on final tests/exams and more focus on
formative assessment.
Factors influencing performance
A multi-level analysis (hierarchical linear modelling
HLM) was performed on the score data in order to explore
whether performance of Primary and Secondary learners
in the Listening, Reading, Writing and Speaking7 tests
varies by a number of different individual (e.g. gender, first
language) and grouping (e.g. school sector) variables.
Given the sample size, there was a limit on the number of
7
The results of HLM analysis on Secondary Speaking scores are not reported because only 67 observations were available in the data matrix (due to missing values in the
dependent and explanatory variables).
UCLES 2017
ISSUE 65 / FEBRUARY 2017|19
variables that could be entered into the model so factors
hypothesised to have a strong impact on performance (e.g.
school sector, level of exposure to English, gender) were
included in the analysis. The following is a summary of
theresults.
Primary
School sector: in the skills of Reading, Listening and
Speaking, learners in Independent schools performed
better than learners in State schools. A different trend
was observed for the skill of Writing: learners in State
schools outperformed those in Church schools, which
was statistically significant.
Gender: in the skills of Reading, Listening and Speaking,
there was no statistically significant difference between
performance of male and female learners, although
girls performed slightly better than boys. In the skill of
Writing, girls outperformed boys with a statistically
significant difference.
Language spoken at home: results showed that those
learners who report speaking both English and Maltese
at home (bilinguals) perform significantly better on the
Listening and Speaking exams compared to those who
report only speaking Maltese at home. No statistically
significant difference was observed for performance on
Reading. Those learners who had reported other as
the language spoken at home (i.e. linguistic minority
learners) performed significantly worse than Maltese L1
learners on the Writing exam.
Speaking English with family: learners who reported
speaking English very often with their family performed
significantly better in the Listening exam than those who
reported never speaking English with their families.
Use of technology in the classroom: learners who
reported using technology in the classroom in some/
most/almost all lessons performed significantly better
than those classes where technology was never/hardly
ever used for Listening and Speaking.
Paired and group work in the classroom: learners who
reported working with other learners in most lessons
performed significantly better than learners who never
or hardly ever participated in pair/group work for
Speaking and Reading.
Secondary
School sector: no statistically significant effects were
found for school sector for the skills of Listening and
Reading, although a trend of higher performances
in Independent schools compared to Church and
State schools was observed. However, learners
in Church schools outperformed those in State
schools on the Writing exam and this difference was
statisticallysignificant.
UNDERSTANDING LANGUAGE LEARNING IN MALTA
20|ISSUE 65 / FEBRUARY 2017
Gender: girls outperformed boys in all three skills of
Listening, Reading and Writing, although this difference
only reached statistical significance for Reading
andWriting.
Language spoken at home: unlike the Primary results,
no statistically significant differences were found for the
effects of the language spoken at home, although a trend
of higher performances by those who either use both
English and Maltese or just English at home compared
to those who only use Maltese in the home environment
was observed.
Speaking English with family: learners who reported
speaking English with their family not very often
performed slightly better in the Writing exam than those
who reported never speaking English with their family.
Recommendations
Improving language levels and narrowing the
achievement gap amongst learners
Findings from the benchmarking study generally showed
high levels of English language performance by learners;
a large proportion of Primary Year 5 learners were found
to be at the B1/B2 levels, with Speaking as their strongest
skill and Reading as the weakest skill. A high proportion
of Secondary learners achieved CEFR Levels B2 and above
across the four skills, which generally suggests readiness
for independent functioning in a variety of contexts.
However, a considerable proportion of Secondary learners
were still found to be at A1/A2 levels and yet to become
independent users of English. Focusing on school sector,
some variations were observed amongst the different
sectors in terms of learner performance, with a trend of
higher performances by Independent schools compared to
Church and State schools, although there were exceptions
to this trend. Variations were also observed within each
school sector where a wide range of performances was
observed from A levels to C levels for State, Church and
Independent schools. In other words, no individual school
sector was associated with a clustering of only high-or
only low-performing learners. Lastly, the wide range of
observed CEFR levels from pre-A1 to C2 reflects a complex
educational context and linguistic milieu in which language
levels vary quite widely in the population. In light of these
findings, the following recommendations are suggested for
improving language levels and narrowing the achievement
gap between learners:
MEDE to take into consideration the language levels
established in the benchmarking project in reviewing
and/or setting of language proficiency/attainment levels.
It is important for the Ministry to investigate why a
proportion of Secondary learners have not progressed
beyond CEFR A1/A2 levels (26.8%, 24.7%, 17% and 9.7%
for the skills of Writing, Reading, Listening and Speaking
respectively) and are yet to become independent users
of English. An understanding of the reasons may allow
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
MEDE to identify a variety of measures to best support
this group of learners in their language learning journey.
The results would also need to be taken into account
in ensuring that the LOF has the necessary procedures
in place to support teachers in addressing the needs of
lower-level learners and in helping them improve their
language proficiency. Note that a number of interventions
have already been put in place by the Department of
Curriculum Management within the Directorate for
Quality and Standards in Education (DQSE) and include
the introduction of Core Curriculum Programme (CCP)
and Alternative Learning Programme (ALP).
Benchmarking results suggested Reading to be the
weakest skill at Primary Year 5. This is likely to be
associated with literacy levels. More emphasis on
quality experiences at Primary school and in Reading
(literacy) is therefore recommended. These efforts can
support the work of the National Literacy Strategy, 2014.
which is committed to improving literacy in Maltese and
English through a number of initiatives.
The wide range of observed abilities from A1 to C1
and/or above within classrooms and the striking
differences in levels of exposure to English outside
the classroom and in the home environment reflect
the bilingual context of Malta. In order to reduce the
achievement gap between learners, it is important for
schools to offer additional support and opportunities
for practising English outside of the classroom, e.g.
English clubs, libraries, school events or online activities.
Providing parents with straightforward and non-
technical information on how people learn languages,
the importance of exposure to English in the home
environment and its positive effects on learning along
with useful tips on what this means for them as parents
and how they can support their children may also
proveuseful.
The wide range of observed abilities from A1 to C1 within
classrooms means that teachers can greatly benefit
from training in differentiated instruction and techniques
for teaching mixed-ability classrooms; a professional
development need which was heavily emphasised in the
teacher surveys.
Steps should be taken to reduce variations in
performance across the school sectors. Sharing of best
practices can be one solution. The teacher surveys
suggested a strong collaborative learning community
amongst the Maltese teachers. This can be built upon by
bringing teachers from different school sectors together,
arranging peer observations between schools so that
examples of good practice could be shared. Language
policies which are considered effective at the school
level (e.g. Content and Language Integrated Learning)
could be shared in the teaching community in an effort
to reduce achievement gaps between learners and
within school sectors.
Girls systematically outperformed boys at both
school levels. While these differences in performance
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES
were generally found to be small and not a cause
for concern, it is nevertheless an area that can be
monitored by MEDE to ensure that all learners receive
the kind of support that they need in their language
development. Boys can also benefit from having more
malelanguage role models in increasing their motivation
to learnEnglish.
Improving learner motivation and attitudes
towards English
Generally speaking, the survey results showed great
importance given to English from the perspective of all
major stakeholders learners, parents and teachers in the
Maltese context, with evidence of high levels of motivation
for learning English for a variety of reasons. However, a few
issues such as the socio-cultural role of English in Malta,
the extent of parental support at different school stages
and attitudes towards English literature emerged from the
surveys. The following recommendations are made in light
of the findings:
Teachers in Malta touched on the complex linguistic
milieu in Malta where the choice of language in daily
life has strong social connotations. This can influence
English learning, with some learners positively
associating the use of English with the modern world
while others resist the English influence in their lives as
they negatively associate it with Maltas colonial past.
This can affect attitudes towards learning and their
motivation to learn English or Maltese, and care should
be taken to address (any) such stereotyping at schools
sensitively in order to reduce any negative attitudes
towards language learning.
More parental involvement in learner education
(particularly for Secondary learners) was emphasised
by teachers. It is therefore recommended for schools
to encourage more parental involvement and raise
awareness of the important role of parents in increasing
their childrens motivation. For example, parents
could be directed to useful online sources such as
Cambridge English TV (www.youtube.com/user/
cambridgeenglishtv) or the Cambridge English website,
which offers tips and advice for parents in supporting
and motivating childrens language learning (www.
cambridgeenglish.org/learning-english/parents-and-
children/information-for-parents), or other similar
online materials. Schools can arrange for and/or allocate
more time for face-to-face information sessions with
parents about the English programme and the language
policies adopted at their childrens school or engage
with parents through more regular communication and
via different media, e.g. school website, texts, emails or
tweets.
While learners (and parents) at both school stages
placed great importance on learning English, attitudes
towards learning English literature were more mixed.
These perceptions can nevertheless be changed by
teachers through awareness-raising activities that
outline the benefits of using English literature, e.g.
UCLES 2017
ISSUE 65 / FEBRUARY 2017|21
in enhancing English language competence. It is
recommended for teachers to select texts which are
both accessible in terms of language levels but also
appropriate for different age groups. Using graded
readers linked to the relevant CEFR level of the learners
is recommended in order to control for the difficulty
levels of texts. Selecting literary texts which are relevant
for young adults is also key in motivating learners.
Teachers emphasised the need for more resources
(e.g. textbooks, books, films) in schools and more
opportunities to use English outside of class (e.g. film
clubs, school trips). Out-of-class activities not only
create opportunities to use English but can also lead to
its increased popularity and enhance learner motivation
and positive attitudes towards English.
Improving pedagogic and assessment practices
and promoting learner autonomy
Survey results generally suggested that teachers have
a highly positive attitude towards teaching English and
report high levels of confidence in their ability to plan
appropriate English lessons for their learners, all of
which can positively influence learning outcomes in the
classroom. Findings showed that a number of different
pedagogic activities which represent examples of good
practice in the classroom occur in some or most lessons,
which is a positive finding. However, some more learner-
centred activities did not occur very regularly in the
classrooms. Technology was reported to be used frequently
in the classrooms and was viewed positively by learners
and teachers alike, although lack of technological access,
opportunities for use as well as the extent of technologys
effectiveness in enhancing learning were listed as
possible issues. In light of the findings, the following
recommendations are made:
Results showed that only about one t hird of teachers
explicitly state the learning goals and objectives of the
lesson while about 10% use paired/small group work in
almost every lesson. Highly learner-centred activities
such as asking learners to suggest topics for classroom
lessons or to help plan classroom activities are reported
to occur far less frequently. Learners will not be able to
develop autonomy if they are not clear on the objectives
of their lessons or if they are not regularly engaged in
peer interaction and group work as well as other learner-
centred activities. Altogether these findings suggest that
classes may benefit from more activities that encourage
self-regulated learning. Classes may therefore benefit
from more paired and group work which is monitored
and supported by the teacher in order to increase the
effectiveness of such activities. Another suggestion is to
incorporate more online resources and activities which
can be completed outside of the classroom in order to
promote self-regulated learning. Some schools in Malta
are encouraging their students to reinforce their learning
on the Virtual Learning Environment and through digital
lessons. If proven to be useful, these practices can be
more widely shared amongst schools.
UNDERSTANDING LANGUAGE LEARNING IN MALTA
22|ISSUE 65 / FEBRUARY 2017
Results suggested the absence of self and peer
assessment skills amongst learners. It is recommended
for teachers to include more awareness-raising activities
in the lessons in order to help learners identify their
strengths and weaknesses and in turn, allow them to
set appropriate language goals for themselves. As an
example, teachers can introduce the concept of learning
contracts where students, with their teachers support,
set their own language goals and outline actions which
they will take to achieve these goals. Teachers should
also be trained to support learners in effective self and
peer assessment skills as part of their teacher training.
Technology is used frequently in the classrooms and
it is viewed positively by learners and teachers alike,
although lack of technological access, opportunities
for use as well as the degree of the technologys
effectiveness in enhancing learning were listed as
possible issues. When integrating technology in the
classroom, it is important not only to ensure access to all
but also to incorporate pedagogies that make the most
of technology (Organisation for Economic Co-operation
and Development 2015). It is therefore recommended
for schools and teachers to consider how technology can
best be incorporated in Maltese educational settings in
ways that can increase opportunities to communicate in
English and improve the provision of feedback. It is also
important to offer professional development support
to teachers in using these technologies in a linked-up
fashion and in a manner that best supports the provision
of individualised learning. Technologies can also have
great potential in assisting teachers with differentiated
instruction. One of the areas of professional
development that teachers rated as having high levels
of impact was individual or collaborative research on a
topic of professional interest. Technology and how to
best incorporate it in the Maltese educational contexts
can be one such area of focus for action research
and can be promoted by the Ministry. In order to
encourage collaborative research, teachers should also
be presented with incentives such as the opportunity to
publish papers.
Classroom test development was found to be commonly
practised by teachers. An assessment literacy course
can therefore greatly benefit teachers in designing high-
quality tests that best support learning.
The scope of the current project did not allow for the
inclusion of more qualitative methods such as classroom
observations in examining teaching practices in the
classrooms. Future phases of the project could include
such methods in order to more accurately capture what
goes on in the classrooms and to illustrate a richer
picture of the teaching and learning context in Malta.
Acknowledgements
Special thanks go to Minister Evarist Bartolo for his
educational vision and for his trust in the quality of work
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
provided by Cambridge English Language Assessment.
Our gratitude goes to Mr Joseph Caruana, Permanent
Secretary;Mr Ian Mifsud, Director General of Quality and
Standards in Education; Mr Gaetano Bugeja, Director of
Department of Curriculum Management; and Ms Sandra
Ebejar, Assistant Director of Department of Curriculum
Management.
This project has always been viewed as a partnership
between two institutions. The study has benefited from the
support, insights and feedback from Ms Lina Debattista,
Ms Clarissa Padovani, Ms Marika Vella, Ms Clare Wigg
and Ms Pamela Zerafa from the Directorate of Quality
and Standards in Education. We would also like to thank
Cambridge English staff, in particular, Ms Coreen Docherty,
Ms Christine Walker, Dr Agnieszka Walczak and Mr
Anthony Deane for their kind support on this project.
Last but not least, we would like to thank the school
principals, teachers, students and parents who participated
in this study. This study would not have been possible
without their contribution.
References
Bachman, L F and Palmer, A S (1982) The construct validation of
some components of communicative proficiency, TESOL Quarterly
16 (4), 449465.
Bandura, A (1977) Self-efficacy: The Exercise of Control, New York:
W H Freeman.
Bartram, B (2006) An examination of perceptions of parental
influence on attitudes to language learning, Educational Research
48 (2), 211221.
Bernhardt, E (2005) Progress and procrastination in second
language reading, Annual Review of Applied Linguistics 25, 133150.
Boud (2000) Sustainable assessment: Rethinking
assessmentforthe learning society, Studies in Continuing Education
22 (2), 151167.
Camilleri, A (1995) Bilingualism in Education, the Maltese Experience,
Heidelberg: Julius Groos Verlag.
Canale, M and Swain, M (1980) Theoretical bases of
communicative approaches to second language teaching and
testing, Applied Linguistics 1 (1), 147.
Carless, D (2009) Trust, distrust and their impact on assessment
reform, Assessment and Evaluation in Higher Education 34 (1),
7989.
Chapman, D W, Weidman, J, Cohen, M and Mercer, M (2005) The
search for quality: A five country study of national strategies to
improve educational quality in Central Asia, International Journal of
Educational Development 25, 514530.
Chavez, M (2000) Teacher and student gender and peer group
gender composition in German foreign language classroom discourse:
An exploratory study, Journal of Pragmatics 32 (7), 1,0191,058.
Colletta, S P, Clment, R and Edwards, H P (1983) Community and
Parental Influence: Effects on Student Motivation and French Second
UNDERSTANDING LANGUAGE LEARNING IN MALTA
CAMBRIDGE ENGLISH : RESEARCH NOTES
Language Proficiency, Quebec: International Center for Research
onBilingualism.
Council of Europe (2001) Common European Framework of
Reference for Languages: Learning, Teaching, Assessment, Cambridge:
Cambridge University Press.
Council of Europe (2015) Language Education Policy Profile, Malta:
The Language Policy Unit, Council of Europe, available online:
www.coe.int/t/dg4/linguistic/Source/Profile-Malta_final2015_
EN.pdf
Creswell, J W and Plano Clark, V L (2011) Designing and
Conducting Mixed Methods Research (2nd edition), Thousand Oaks:
SagePublications.
Csizr, K and Drnyei, Z (2005) The internal structure of language
learning motivation and its relationship with language choice and
learning effort, The Modern Language Journal 89, 1936.
Docherty, C, Gratacs Casacuberta, G, Rodrguez Pazos, G
and Canosa, P (2014) Investigating the impact of assessment
inasingle-sex educational setting in Spain, Research Notes 58, 315.
Drnyei, Z (2003) Introduction, Language Learning 53 (1), 12.
English Language Resource Centre (2007) English Language
Syllabus for Secondary Schools 2007, available online: curriculum.
gov.mt/en/Curriculum/Form-3-to-5/Documents/curric_f3_f5/
syllabus_english_lang_f3_5.pdf
European Commission (2012) First European Survey on Language
Competences: Final Report, Luxembourg: Publications Office of the
European Union, available online: ec.europa.eu/languages/eslc/
index.html
Gardner, R C (1985) Social Psychology and Second Language Learning:
The Role of Attitudes and Motivation, Baltimore: Edward Arnold.
Gu, X and Saville, N (2012) Impact of Cambridge English: Key for
Schools and Preliminary for Schools parents perspectives in
China, Research Notes 50, 4855.
Hattie, J (2012) Visible Learning for Teachers: Maximizing Impact on
Learning, London: Routledge.
Himanolu, M (2005) Teaching English through literature, Journal
of Language and Linguistic Studies 1 (1), 5366.
Horwitz, E (2001) Language anxiety and achievement, Annual
Review of Applied Linguistics 21, 112126.
Jones, N (2013) The European Survey on Language Competences
and its significance for Cambridge English Language Assessment,
Research Notes 52, 26.
Koda, K (2007) Phonology and literacy, in Pennington, M (Ed)
Phonology in Context, Basingstoke: Palgrave Macmillan, 219244.
Lepper, M R, Corpus, J H and Iyengar, S S (2005) Intrinsic and extrinsic
motivational orientations in the classroom: age differences and
academic correlates, Journal of Educational Psychology 97 (2), 184196.
UCLES 2017
ISSUE 65 / FEBRUARY 2017|23
Maccoby, E E and Jacklin, C N (1974) The Psychology of Sex
Differences, Stanford: Stanford University Press.
Mifsud, C L and Vella, L (forthcoming 2017) Pre-school bilingual
education in Malta: The realities and the challenges, in Schwartz,
M (Ed) 21st Century Preschool Bilingual Education: Teachers, Family
and Ethno-linguistic Community Triangle, New York: Springer.
Mills, N, Pajares, F and Herron, C (2006) A reevaluation of the role
of anxiety: Self-efficacy, anxiety and their relation to reading and
listening proficiency, Foreign Language Annals 39, 276295.
Ministry for Education and Employment (2012) A National
Curriculum Framework for All, available online: education.gov.mt/en/
Documents/A%20National%20Curriculum%20Framework%20
for%20All%20-%202012.pdf
Ministry for Education and Employment (2014) National Literacy
Strategy for all in Malta and Gozo 20142019, available online:
education.gov.mt/en/Documents/Literacy/ENGLISH.pdf
Ministry for Education and Employment (2015) Learning Outcomes
Framework, available online: www.schoolslearningoutcomes.edu.
mt/en
Ministry for Education and Employment (2016) A Language Policy
for the Early Years in Malta and Gozo, available online: education.
gov.mt/en/Documents/A%20Language%20Policy%20for%20
the%20Early%20Years%20Consultation%20Document.pdf
Multon, K D, Brown, S D and Lent, R W (1991) Relation of
self-efficacy beliefs to academic outcomes: A meta-analytic
investigation, Journal of Counseling Psychology 38(1), 30.
Organisation for Economic Co-operation and Development (2009)
PISA 2009 Key Findings, available online: www.oecd.org/pisa/
pisaproducts/pisa2009keyfindings.htm
Organisation for Economic Co-operation and Development (2015)
Students, Computers and Learning: Making the Connection, available
online: www.oecd.org/publications/students-computers-and-
learning-9789264239555-en.htm
Oxford, R L and Ehrman, M E (1993) Second language research on
individual differences, in Grabe, W (Ed) Annual Review of Applied
Linguistics XIII, Issues in Second Language Teaching and Learning,
188205.
Progress in International Reading Literacy Study (2011) PIRLS 2011
International Results in Reading, available online: timssandpirls.
bc.edu/pirls2011/downloads/P11_IR_FullBook.pdf
Purpura, J E (2004) Assessing Grammar, Cambridge: Cambridge
University Press.
Slater, S and Collie, J (1987) Literature in the Language Classroom,
Cambridge: Cambridge University Press.
UNDERSTANDING LANGUAGE LEARNING IN MALTA
24|ISSUE 65 / FEBRUARY 2017
An investigation of Portuguese students attitudes to
assessment and Cambridge English exams
JANE LLOYD CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
ELAINE BLAUS CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT, SPAIN
HELDER SOUSA INSTITUTO DE AVALIACAO EDUCATIVA, PORTUGAL
Introduction
This is a report on a multi-phase study which began in
2013, when the Portuguese Ministry of Education and
Science (MEC) made a public commitment to improve
the standard of English language teaching and learning
throughout Portugal, with a focus on Grade 9 pupils. The
MEC made the strategic decision to introduce Cambridge
English: Key for Schools, which targets A2 level, as a
mandatory external exam to be taken by all Grade 9 pupils
at the end of the 2013/14 academic year. The overall aim of
the initiative is the improvement of teaching and learning of
English in secondary school by using exam performance to:
monitor and evaluate language learning in the
Portuguese secondary school system
inform future English language policy and decisions
regarding teacher development initiatives and training
provide pupils with an opportunity to obtain an
internationally recognised certificate of language
proficiency regardless of socioeconomic background.
In order to determine the extent to which these aims are
being met, and to help the MEC achieve its overall long-
term goal to raise standards in English language teaching
and learning in Portugal, Cambridge English undertook an
impact study in collaboration with the MEC and Instituto de
Avaliao Educativa, I P (IAVE).
Year 1 of the study, which took place in the 2013/14
academic year, provided a baseline profile of the different
stakeholder groups (i.e. learners, teachers, school heads,
parents and caregivers). It reported on their attitudes and
perceptions at the beginning of the initiative, following the
introduction of Cambridge English: Key for Schools.
Year 2 of the study builds on the main findings in Year1
of the project. These findings revealed consensus among
head teachers, teachers and parents regarding the
need to improve the teaching and learning situation in
Portugal, and a generally positive reaction to the initiative,
alongside some differing views on the appropriateness of
the exam, which some stakeholders felt had been at too
low a level. Attitudes towards English were very positive
and certification in English was similarly valued, with the
majority of respondents considering it essential to have an
internationally recognised certificate of proficiency. Pupils
and teachers found the English classroom environment
UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES
highly motivating and there was a shared understanding of
the need to prioritise speaking development. Assessment
in general was viewed as an important tool that could
provide diagnostic information to enhance learning in the
classroom. As a result of the findings in Year 1 and the
exam results in 2013/14, Cambridge English: Preliminary
for Schools was introduced as the external assessment
forthe2014/15 academic year, the second year of
theproject.
Research aims
The impact study was expected to take place over three
academic years in total, but was shortened to two years
in the light of a governmental decision to measure
proficiency in maths and Portuguese. It aimed to investigate
stakeholder attitudes to English learning, teaching and
assessment in Portugal in order to answer the following
research question: What is the intended/unintended effect
of MEC and IAVEs strategic decision to ensure the quality
of English language provision through the use of external
assessment, i.e. Cambridge English for Schools exams?
In addition, the impact study attempted to answer the
following question: Do the attitudes and perceptions of the
Grade 9 pupils, their teachers, school directors and parents
towards the introduction of Cambridge English for Schools
exams change over time?
Year 1 of the study, which took place in the 2013/14
academic year, provided a baseline profile of the different
stakeholder groups (i.e. learners, teachers, school heads,
parents and caregivers). It reported on their attitudes and
perceptions at the beginning of the initiative, following
the introduction of Cambridge English: Key for Schools.
Year 2 of the study, which took place during the 2014/15
academic year, concentrated on a detailed analysis
of the Grade9 pupils. The focus was on attitudes to
English, English learning and assessment. It also aimed to
highlight any factors, including demographic, situational
or attitudinal factors, which related to proficiency levels
within the Grade9 cohort, and selected those which
IAVE may be able to influence, either directly or indirectly.
The study did not contain detailed investigation of
factors beyond the control of IAVE, such as gender, for
example, or cityofresidence. It aimed to address the
followingquestions:
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|25

What are the characteristics of the Grade 9 population? Figure 1: Example diagram of multi-phase design (adapted from
Creswell and Plano Clark 2011:118)
What are the characteristics of the learning
environment? Phase 2:
Phase 1:
Mixed
What are common attitudes and perceptions in the Phase 1:
Quantitative
data collection
methods
Quantitative
Grade 9 population? Qualitative & analysis
data
data Procedures: collection &
collection &
Which situational and attitudinal factors relate to Grade analysis
Questionnaires analysis
Tests
9 performance? Procedures:
Candidate
Procedures:
Questionnaire
Overall Semi- Information Tests
This article discusses the findings of the demographic and Initiative structured Form Analysis:
Objective interviews and
attitudinal pupil questionnaire completed in May 2015. focus groups
Analysis: Descriptive
Descriptive analysis
In Year 1, a small percentage of Grade 9 pupils (around Analysis:
analysis Factor analysis
Thematic Regression
3,000) completed the questionnaires. In Year 2, 96% of the analysis
Group
comparisons analysis
population completed the questionnaires, so comparisons Products: Products: Products:
Test scores
must be interpreted cautiously. Major themes Test scores
Frequency %
Frequency %
Mode
Mode
Discussion

Methodology
Research design
Quantitative data: Questionnaires
This study is designed to investigate the impact of the and test data
initiative over time. A sequential transformative research
The quantitative data consisted of:
design formed the basis of the study (see Figure 1). The
research design involved the collection and analysis of demographic data collected from exam candidates on
the qualitative data (from interviews and focus groups) the day of the written exam
and quantitative data (questionnaires) in the first year of
responses to attitudinal questions completed by pupils
the project. These findings informed the refinement of the
on the day of the written exam
instruments for the data collection of the quantitative phase
(i.e. pupil questionnaires) in the second year. As a result, candidate test scores
the research design built up a fuller picture of the effects of
enrolment data for the Cambridge English: Preliminary for
the initiative.
Schools certificate.

Constructs The questionnaire administered to test takers on the exam day

The constructs in Table 1 formed part of the conceptual
framework for the research, and fed into the design of the
instruments for both phases, which is described in the
nextsection.
The constructs in Table 1 were researched in Year 2 of the
study via a scannable questionnaire administered to all
Grade 9 pupils on the exam day and analysed alongside
test scores and enrolment data for the Cambridge English:
Preliminary for Schools certificate.
was used to gather demographic information, attitudes and
perceptions from the whole Grade 9 candidate population.
It consisted of multiple-choice questions in Portuguese,
asking candidates to indicate their age and mother tongue,
for example, and whether they agreed with statements like I
enjoy my English classes and how often they engaged in various
activities, such as using websites in English.
The questionnaire developed in Year 1 was used as the
basis for the Year 2 questionnaire. The attitudinal section
of the questionnaire was developed by selecting validated

Table 1: Research constructs in Year 2

Area of investigation Construct Data type Instrument

Attitude towards English Motivation/attitudes Quantitative Questionnaire to pupils

Characteristics of the home
environment
Characteristics of the learning
environment
Attitude towards assessment
Exposure to English
Exposure to other foreign languages
English ability/proficiency
Test scores
Certificate enrolment data
Motivation/attitudes Quantitative Questionnaire to pupils
Exposure to English
Parental proficiency
Parental involvement in school life
Perceptions of teaching, learning and Quantitative Questionnaire to pupils
assessment
Motivation/attitudes Quantitative Questionnaire to pupils
Certificate enrolment data

UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
26|ISSUE 65 / FEBRUARY 2017
statements from the Cambridge English Questionnaire
Item Bank. The statements included Likert scale items
consisting of a 5-point scale requiring a single response.
The most typical response options were: strongly agree,
agree, disagree, strongly disagree and not sure. In
preparation for Year 2, the Year 1 questionnaire was
reviewed. Questions which were found not to provide
useful information were deleted, and new questions were
added to explore issues raised, such as the frequency of
pair and group work in class. Candidate performance on
the Cambridge English: Preliminary for Schools exam was
analysed, and reported on in September 2015. Candidate
scores were linked to the questionnaire responses via the
candidates unique exam identifier. Enrolment data for the
Cambridge English: Preliminary for Schools certificate was
collected prior to administration of the exam and linked to
the test responses via the candidates unique exam ID.
Study participants
All of the 95,992 pupils who appeared in the exam report
analysis fully or partially completed the initial demographic
questions. In addition, a total of 91,464, or 94.30% of the
Grade 9 exam population, also completed some or all of the
contextual and attitudinal questions. Since the contextual
and attitudinal questions were more likely to yield findings
which could be acted upon, in terms of influencing future
performance, it was decided to concentrate on these
contextual and attitudinal questions for the impact study,
and to report detailed demographic findings (such as gender,
and regional location) in the exam report. Since this group
accounted for 94.30% of the test takers on the day, it was
sufficiently representative of the Grade 9 population as
a whole to allow meaningful generalisations to be made.
Only 82,281 Grade 9 pupils completed all test components.
A higher number of Grade 9 candidates completed the
demographic questions. This indicates that some candidates
completed only the demographic information, but not
the exam papers. For the attitudinal analysis the data
comprises of the 91,464 Grade 9 pupils who completed the
demographic, attitudinal and contextual questions.
Grade 9 participant profile
The Grade 9 exam population seems a homogenous one in
terms of demographics, with the majority of candidates being
Table 2: Distribution of Grade 9 learners by CEFR level and paper
Grade 9 Reading Writing
N % N %
Below 23,523 25.5% 30,702 33.3% 22,951
A2
A2 43,572 47.2% 22,088 23.9% 37,129
B1 19,446 21.1% 27,107 29.4% 21,863
B2 5,775 6.3% 12,419 13.5% 10,414
Total 92,316 100.0% 92,316 100.0% 92,357
B1 + B2 25,221 27.3% 39,526 42.8% 32,277
UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES
Portuguese speakers (91%) who attend public schools (87%).
French is the most commonly taught additional language
(67%) followed by Spanish (22%). Most pupils receive two
(40%) or three (50%) hours of English instruction a week
at school and complete one (60%) or two (23%) hours of
English homework per week. Therefore any variations in exam
scores are unlikely to be due to any significant difference in the
amount of English language provision in schools across the
country or the type of school attended.
On the exam day, some candidates completed the Reading
and Writing sections of the test, but not the Listening paper.
Some did not sit the Speaking test. Table 2 shows Grade
9 performance on each paper, including candidates who
sat only certain components. The first column shows the
Common European Framework of Reference for Languages
(CEFR, Council of Europe 2001) level awarded as an overall
level, which was calculated for the 82,281 candidates who
sat all components.
The highest percentage of candidates for each skill (marked
in bold in Table 2) was at the following CEFR level for each
paper: Reading (A2), Writing (Below A2), Listening (A2),
and Speaking (A2). However, grouping B1 (a Pass or Merit
at this level) and B2 (Distinction) together to show total
percentages at or above the target level, reveals Speaking
to be the strongest skill, closely followed by Writing, with
Reading being the weakest skill. In terms of the awarding of
an overall level for performance, A2 was the most common
(35.2%). However, 38% of Grade 9 pupils were awarded
a Pass, Merit or Distinction, achieving B1 or higher. This
contrast between the most common CEFR level achieved,
of predominately A2, and the percentage of candidates
performing at the pass level or even higher, indicated there
are different test populations within the cohort. This study
aims to identify which factors can be used to separate out
these different populations.
Findings and discussion
This section reports on the findings from the pupil
questionnaire. First, the descriptive data is presented
in a discussion of pupil attitudes towards English,
characteristics of the home learning environment, and of
the learning environment, and finally, attitudes towards
assessment. Secondly, it investigates which of these
attitudinal and situational factors relate to performance in
Reading, Writing, Listening and Speaking.
Listening Speaking Overall
N % N % N %
24.9% 20,510 23.8% 22,045 26.8%
40.2% 28,529 33.2% 28,939 35.2%
23.7% 26,463 30.8% 23,823 29.0%
11.3% 10,541 12.3% 7,474 9.1%
100.0% 86,043 100.0% 82,281 100.0%
34.9% 37,004 43.0% 31,297 38.0%
CAMBRIDGE ENGLISH : RESEARCH NOTES
Data analysis
The analysis was carried out in four stages. In the first
stage, descriptive analysis was carried out to identify overall
patterns and trends in the data. Initially, constructs were
grouped together according to expert judgement.
In the second stage, factor analysis was carried out in order to
uncover the relationship patterns underlying the statements
in the pupil questionnaire. This was done to identify how
statements grouped together most appropriately under the
four umbrella constructs. The analysis was carried out using
SPSS, a statistical software package. The extraction method
was Principal Component Analysis. The rotation method was
Varimax with Kaiser Normalization.
In the third stage, the responses to the pupil questionnaire
were analysed using regression analysis, in order to
investigate the relationship between each construct,
individual questionnaire items and performance in each skill
in English. Analysis was carried out individually on each of
the four skills tested.
In the final stage, the data was analysed to discover how
well the score data and the components highlighted by the
factor analysis were correlated. Some additional analysis
was run to confirm unexpected findings in the data.
The results for each construct are given in Tables 3 and
4, starting with the descriptive statistics. The results
of the factor analysis are given next, giving the number
of components highlighted by factor analysis. This is
followed by the results of the regression analysis. The pupil
responses to statements, such as strongly agree or agree,
were conflated into two categories and recoded: strongly
agree and agree were coded as one and strongly disagree
and disagree as zero. In this way, pupils who reported
higher levels of engagement or frequency of activities such
as reading in English, or a more positive attitude, could be
separated out from those who did not.
The individual responses to the questionnaire statements
were regressed onto the exam scores to determine
which item predicts the most variance in exam scores,
when controlling for variance in other statements. The
results show the relationship between the questionnaire
statements that make up each construct, and the scores
on the tests. The final column indicates whether the
relationship is significant or not. Values above .05 suggest
no relationship when controlling for variance in other
responses in the same section of the questionnaire.
The Unstandardised Coefficients column B gives the
increase in raw score for the pupils who demonstrated more
agreement with the questionnaire statement. Pupils who
indicated they liked English tended to score 2 points more in
Reading (2.194 plus or minus the standard error) compared
to those pupils who indicated they did not like it, when
controlling for variance in other questionnaire statements.
All skills papers in the test have a total score of 25. This
means that a 1 point score in Listening, for example, would
indicate the same increase in overall score as a 1 point
increase in Reading. Items which have a coefficient of under
UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
ISSUE 65 / FEBRUARY 2017|27
0.5 were removed, since this would not round up to one
point, leaving only those which are close to 1 point or higher.
While reading these tables, it is important to note that
the analysis only indicates a significant relationship, but
does not show causality. In other words, the analysis
does not indicate that pupils score more because they
feel positive about English, nor feel more positive about
English because they score more. Only significant findings
are included which meet the threshold of +/-1 point and
.05significance.
Attitude towards English
The academic literature on second and foreign language
acquisition has indicated that learners attitudes towards
learning a language and the extent to which they perceive
the language to be useful can influence learner behaviour,
both in terms of the amount of effort exerted on language
learning and the extent to which they persist with learning
it (Bernat and Gvozdenko 2005, Csizr and Drnyei 2005,
Drnyei 2003, Horwitz 2001, Noels 2001). Therefore
attitude towards English is an important construct to
evaluate. In this section we present findings on Grade 9
attitudes towards the importance of studying English and
attitudes to the language in general.
Enjoyment of English
In order to investigate learners level of engagement with
English, how much they value it and how linguistically
self-confident they feel, pupils were asked to what extent
they agreed with three statements about English, which are
listed in Figure 2. It shows that almost 80% of pupils enjoy
their lessons at school and like English generally. Although
they enjoy the subject, their confidence ratings are not as
high, with just over half the pupils (54.7%) reporting they
are confident using English.
In Year 1 of the study, pupils had a generally positive
Figure 2: Pupils enjoyment of English
I like English 31.7% 44.8% 5
12.5% 5.6%
I enjoy my lessons at school 21.5% 57.1% 4
12.0%4.3%
I am confident using English 13.8% 40.9% 20.9% 9.7%
9.7%13.3%
0% 20% 40% 60% 80% 100%
Strongly Agree Agree Disagree
Strongly Disagree Not Sure Blank
attitude towards the study of English. In the 2014/15
academic year, the pupils who responded reported that
they liked English (81%) and that they enjoyed their lessons
at school (82%), whilst 66% agreed or strongly agreed
that they were confident using English. Compared to the
findings in Year 1, the level of agreement is slightly lower.
However, this current sample is much larger and therefore
more diverse than the group of pupils who responded in
Year 1. The majority of pupils are still positive in their views.
28|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

The importance of studying English Factor and regression analysis

Grade 9 pupils were presented with five reasons why
people study English, and asked how important each of
these reasons was to them, as shown in Figure 3. The
majority of pupils consider getting a good job, travel,
interaction with non-speakers of Portuguese and access
to a good university or college all to be important or very
important reasons for studying English. Almost 80% of
pupils think that English is also important or very important
for accessing information on the internet.
Figure 3: Pupil reasons for studying English
To get a good job 64.9%
To travel to other countries 62.7%
To talk to people who dont speak Portuguese 59.1%
To get into a good university or college 55.2%
To use the internet to get information 30.8%
0% 20% 40%
Very Important Important Not Very Important
Not At All Important Not Sure Blank
In Year 1, the reasons for studying English were ranked in
exactly the same order. Over 90% of pupils indicated that
the first four reasons were important or very important
reasons, and over 80% indicated that using the internet to
get information was an important or very important reason.
Statements were in two related groups: one about
enjoyment of English and a second grouping related
to reasons for studying English. Interestingly, two of
these reasons, using the internet to get information, and
talking to people who dont speak Portuguese, overlap
with the Enjoyment of English grouping. This suggests
that more immediate uses of English, such as using the
internet or interacting with other speakers of English,
contribute to enjoyment in a way that more long-term
extrinsic motivations do not. The aspirational goal of future
proficiency also related to the Enjoyment of English group.
3.
28.9% 3.7% Table 3 shows the relationship between the items that
3.
32.2% 3.0% make up the Attitude to English scale, and the scores on
the tests. The Unstandardised Coefficients column B gives
3.
34.7% 3.5%
the increase (or decrease, a minus number) in score for the
33.8% 7.0%
7 pupils who demonstrated more agreement with the item.
45.8% 19.5%
So for example, a pupil who strongly agreed that they liked
English, also had a 3 point increase in Writing compared to
60% 80% 100%
pupils who did not agree that they like English.
The analysis shows there is a relationship between
enjoyment of and confidence in learning English and better
performance in all four skills. When controlling for variance in
other items, there is also a relationship between feeling that
the ability to speak to non-Portuguese speakers is important
and better performance, which is most apparent in Writing.
There is also a relationship between feeling that English is
important for using the internet and higher scores. This is

Table 3: Attitude to English and scores

Question Skill Unstandardised Coefficients Standardised Coefficients t Sig.

B Std. Error Beta

I like English. R 2.194 .091 .132 24.186 <.001

W 3.857 .119 .172 32.300 <.001
L 2.533 .111 .125 22.821 <.001
Sp 3.379 .111 .173 30.499 <.001
I am confident using R 2.602 .062 .215 41.708 <.001
English.
W 3.536 .082 .217 43.053 <.001
L 3.370 .076 .229 44.188 <.001
Sp 3.114 .075 .222 41.447 <.001
What level of English R 2.559 .052 .240 49.629 <.001
would you like to
achieve? W 3.410 .068 .238 50.239 <.001
L 2.846 .063 .220 45.150 <.001
Sp 2.603 .062 .212 41.972 <.001
It is important to R 1.134 .235 .027 4.816 <.001
learn English to talk to
people who dont speak W 2.080 .310 .037 6.710 <.001
Portuguese. L .827 .288 .016 2.871 .004
Sp 1.618 .296 .032 5.460 <.001
It is important to R .651 .166 .021 3.926 <.001
learn English to use
the internet to get W 1.427 .218 .033 6.533 <.001
information. L .613 .203 .016 3.020 .003
Sp 1.108 .202 .030 5.473 <.001
It is important to learn W 1.386 .233 .032 5.938 <.001
English to get into a good
university or college. Sp .781 .218 .021 3.576 <.001

R=Reading; W=Writing; L=Listening; Sp=Speaking

UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES
more noticeable in Writing and Speaking. The aspirational
level of pupils has a relationship with better performance
in all skills. When controlling for variance in other items,
enjoyment of lessons at school does not have a significant
relationship with better performance in any skill. There is no
significant relationship between the importance of travelling
to other countries or getting a good job and higher scores in
any skill, when controlling for variance in other items.
Engagement with English
Pupils reported how frequently they use English to talk to
others, to read books, magazines or newspapers in English,
to play computer games or use websites in English, or to
listen to songs or watch movies in English in daily life.
The most frequent use of English is listening to songs, or
watching TV programmes and films in English, with over
80% of pupils reporting they do this almost every day, and
a further 11.6% reporting they use English to do this about
once a week. Using English online also ranks highly as a
frequent activity. Over 60% of pupils report using English
to speak to tourists or visitors. This is not as frequent an
activity as listening to songs or going online, but most likely
this is linked to the opportunity to do so. Reading books
and magazines is also a reasonably frequent activity, with
just below 40% reporting reading in English at least once a
month. A further 23.2% report infrequent reading of books,
magazines or newspapers in English. It is possible that this
level of engagement for reading in English may not be that
different from the amount of reading in Portuguese for this
age group, and would also be linked to the availability of
reading material in English outside of school.
In Year 1, 83% indicated they listened to songs, watched
movies or TV programmes in English, and 48% that they
played computer games or used websites. The most
common response for speaking to tourists or visitors in
English was also A few times a year (49%) and never for
reading books, magazines and newspapers (35%).
English proficiency at home
The level of English proficiency at home contributes to
the amount of English pupils are exposed to outside the
classroom. In a household where English proficiency is
higher, there may be an increased likelihood of access to
English in the ways described previously, such as access to
films or books in English, and possibly increased interest
in and discussion of English lessons. Therefore pupils
were asked to match the English language ability of their
parents to brief CEFR descriptors. Since not all pupils have
a mother or father living at home, pupils had the option to
select not applicable for these questions. This question
had a lower response rate than others. It may be that the
pupils were unable to rate the level of their parents English
with any certainty. Looking at the responses where a level
was indicated, the majority of parents were categorised as
having no English ability, or being at A1 or A2 level. In other
words, they are perceived to be below the target level of the
exam. While we do not know the actual level of English for
the parents, it is likely that if pupils perceive their parents
UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
ISSUE 65 / FEBRUARY 2017|29
ability to be low, they will be less likely to request help with
homework, or discuss it in detail, or talk about their English
classes. In Year 1 the most common level selected was No
English for both parents, and the majority of parents were
also perceived to be below the target level of the exam, at
A2 or lower.
Parental attitudes and involvement
Active parental involvement in school life can contribute
towards academic success for pupils and can influence
their childrens attitudes. For language learning, the use of
English at home, and the English language ability of family
and friends is also an important factor. Therefore questions
were included in the questionnaire on parental involvement
with the pupils English study, the use of English at
home or for leisure purposes, and the level of English of
theirparents.
When asked how often their parents discussed their
English classes with them or their homework, the two
most common responses for both homework and classes
were about once a week and never. Both activities were
represented to a similar degree in all the categories, with
the exception of homework which was rated never slightly
more often than lessons. The responses in Year 1 were
similar. Pupils were asked how often they discussed English
classes at home, and the three most common responses
were a few times a year about once a week (both 24%)
and never (21%). Homework was not directly targeted
in Year 1, but a more general question on engagement in
school activities was included. Findings in Year 1 could not
be generalised, so this question was changed to a more
specific one on homework for Year 2.
Factor and regression analysis
Three thematically linked groups were identified. They relate
to 1) use of English at home by the pupils, or engagement
with English; 2) the frequency with which parents talk
to them about classes and homework; and 3) the pupils
perception of parental proficiency in English. Reading in
English and accessing English online are more strongly
associated with engagement with English than listening to
songs, or involvement in other media. Table 4 shows the
relationship between the statements grouped under the
Home environment scale, and the scores on thetests.
The analysis shows there is a relationship between increase
in scores in all skills and regular activity and exposure
to English outside the classroom, ranging from a 0.858
increase in scores in Reading for pupils who report they
regularly speak to tourists or visitors in English, or use
English abroad or on holiday, to an increase of 4.354 in
Writing scores for those who report they listen to songs
in English, or watch movies or TV programmes in English.
There is also a relationship between a higher level of
parental ability in English and a higher score on all skills.
What is notable is the tendency for pupils who report
regular parental involvement, in terms of discussing
homework, and lower scores. When controlling for variance
in other items, this relationship is significant in all skills for
30|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Table 4: The home environment and scores

Question Skill Unstandardised Coefficients Standardised Coefficients t Sig.

B Std. Error Beta

How often do you listen to songs R 2.528 .189 .065 13.410 <.001
in English, or watch movies or TV
programmes in English? W 4.354 .248 .086 17.558 <.001
L 2.742 .227 .059 12.056 <.001
Sp 3.817 .230 .086 16.587 <.001
How often do you speak to R .858 .067 .065 12.886 <.001
tourists or visitors in English, or
use English abroad or on holiday? W 1.243 .088 .072 14.193 <.001
L 1.268 .080 .080 15.816 <.001
Sp 1.366 .077 .095 17.669 <.001
How often do you play computer R 2.555 .077 .169 33.221 <.001
games in English or use websites
in English? W 3.129 .101 .157 30.936 <.001
L 3.123 .093 .171 33.732 <.001
Sp 2.926 .091 .173 32.164 <.001
How often do you read books, R 3.245 .059 .289 55.015 <.001
magazines or newspapers in
English? W 4.241 .078 .287 54.681 <.001
L 3.862 .071 .286 54.380 <.001
Sp 3.426 .069 .277 49.825 <.001
How often do your parents R 1.499 .075 .136 20.001 <.001
talk to you about your English
homework? W 1.615 .099 .112 16.384 <.001
L 1.775 .090 .134 19.679 <.001
Sp 1.420 .088 .117 16.230 <.001
How often do your parents talk to L .565 .092 .042 6.136 <.001
you about your English lessons?
What level of English can your R 1.185 .069 .104 17.194 <.001
mother speak?
W 1.741 .091 .116 19.213 <.001
L 1.320 .083 .096 15.901 <.001
Sp 1.262 .081 .100 15.599 <.001
What level of English can your R 1.339 .068 .120 19.828 <.001
father speak?
W 1.701 .089 .116 19.164 <.001
L 1.534 .081 .114 18.869 <.001
Sp 1.168 .079 .094 14.761 <.001

R=Reading; W=Writing; L=Listening; Sp=Speaking

discussion of homework, and for Listening for discussion
of lessons. As mentioned earlier, it is unusual for parental
involvement to have a relationship with a decrease in
performance. Therefore, in order to investigate this finding
further, correlational analysis was carried out for these
two responses, and the four skills. The results are shown
in Table 5. This shows a significant negative correlation
between parental involvement and scores.
It could be that pupils who are finding English challenging
are the ones talking to their parents about their classes
and homework, which would indicate that they and/or
their parents are taking their studies seriously, and have
some worries, or are being put under pressure by parents
to complete homework. The questionnaire included a
statement I am worried about taking the Preliminary for
Schools exam. In order to investigate whether it was
pupils who indicated they were worried about their
exam who were talking regularly to parents, further
correlational analysis was carried out. When controlling
for the level of worry over the exam however, there was
no large and significant correlation between those pupils
who spoke to their parents, and exam scores (see Table
6). There is therefore not sufficient evidence that it is
only the pupils who are worried about the exam and
talking to their parents about their homework who find
Englishchallenging.
The learning environment
As mentioned earlier, both exposure to and attitude
towards the foreign language has an impact on learning.
The amount of engagement with English partly depends
on how much opportunity pupils have to use the language.
Therefore it is useful to find out how much pair work and
group work is taking place in class, when there are likely to
be increased opportunities to use the language, compared
to when the teacher instructs the class as a whole, and
pupils work individually. It is also useful to investigate the
amount of spoken English in class and the opportunities for
the pupils to use the language, in addition to the teacher.

UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|31

Table 5: Parental involvement and scores

How often do your parents talk to How often do your parents talk to
you about your English lessons? you about your English homework?

How often do your parents Pearson Correlation 1 .758

talk to you about your English
lessons? Sig. (2-tailed) <.001
N 91,085 90,737
How often do your parents Pearson Correlation .758 1
talk to you about your English
homework? Sig. (2-tailed) <.001
N 90,737 90,927
Reading Pearson Correlation .084 .127
Sig. (2-tailed) <.001 <.001
N 90,966 90,807
Writing Pearson Correlation .040 .081
Sig. (2-tailed) <.001 <.001
N 90,977 90,818
Listening Pearson Correlation .091 .130
Sig. (2-tailed) <.001 <.001
N 90,898 90,741
Speaking Pearson Correlation .040 .077
Sig. (2-tailed) <.001 <.001
N 81,236 81,089

Table 6: Parental involvement and scores, controlling for worry

Control variables Reading Writing Listening Speaking How often do your parents talk to
you about your English homework?

I am Reading Correlation 1.000 .778 .831 .720 .094

worried
about Sig. (1-tailed) 0.000 0.000 0.000 .000
taking the df 0 75,641 75,641 75,641 75,641
Preliminary
for Schools Writing Correlation .778 1.000 .752 .770 .063
exam
Sig. (1-tailed) 0.000 0.000 0.000 .000
df 75,641 0 75,641 75,641 75,641
Listening Correlation .831 .752 1.000 .707 .094
Sig. (1-tailed) 0.000 0.000 0.000 .000
df 75,641 75,641 0 75,641 75,641
Speaking Correlation .720 .770 .707 1.000 .054
Sig. (1-tailed) 0.000 0.000 0.000 .000
df 75,641 75,641 75,641 0 75,641
How often Correlation .094 .063 .094 .054 1.000
do your
parents Sig. (2-tailed) .000 .000 .000 .000
talk to you df 75,641 75,641 75,641 75,641 0
about your
English
homework?

Pair and group work Use of spoken English

Pupils were asked how frequently they worked individually,
and how often they worked in pairs or groups during
English lessons. Individual work is the most common, with
the highest percentage of pupils (36.7%) indicating they
work individually most of the time, in contrast to not very
often being the most common response (46.4%) when
asked how often they work in pairs or groups. This was a
new question for 2014/15, addressing a gap identified in
the analysis, so there is no comparable data for Year 1.
In terms of use of English in the classroom, pupils were
asked to indicate how frequently the teacher spoke in
English, and how often pupils spoke to each other. They
were also asked to consider their own use of English in
class generally, which would include interaction with other
pupils, and how frequently they spoke to the teacher in
English, as shown in Figure 4.
Figure 4 shows that teacher use of English in the class
is frequent, both in terms of the teacher using English to

UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
32|ISSUE 65 / FEBRUARY 2017
Figure 4: Use of English in class by pupils and teachers
How often does your teacher speak in
English?
26.7%
How often do you talk to your teacher in
English?
11.1% 21.9%
How often do you talk in English?
19.4%
How often do the pupils speak to each other in
English?
0% 20%
Most of the Time About Half the Time
Never Not Sure
address the pupils and to interact with them individually.
Over 90% of pupils report that their teacher speaks in
English most of the time (72.2%) or about half the time
(21.2%).
When pupils are asked how often they speak in English
generally or to each other in English, the most common
response in these cases is not very often. This indicates
that when the teacher is involved in exchanges with pupils,
it is likely to be in English, but that exchanges between
pupils are not in English as frequently. This is not surprising,
given the information on the infrequent use of pair and
group work.
In Year 1, 94% of pupils reported that their teacher used
English most of the time or about half the time, and as
with this year, when asked how often they speak in English
generally or to each other in English, the most common
pupil response was not very often.
Skills in English
When asked to identify their strongest skill in English,
the two most common responses were Reading (26.4%)
and Listening (25.6%) (see Figure 5). Speaking, Writing,
Grammar and Vocabulary were each selected by
approximately 10% of the pupils.
Figure 5: Strongest skill in English
5.2%
9.8%
26.4%
13.0%
9.3%
10.7%
25.6%
Pupils were also asked to identify the skill they felt was
most in need of improvement. The two skills selected by
the highest number of pupils were Writing (22.1%) and
Speaking (19.8%). When asked what most class time was
spent on, the two most common responses were Grammar
(28.2%) and Writing (19.4%). The two skills with the
largest discrepancy between pupil perception of the need to
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
improve and their perception of what class time was spent
72.2% 21.2% 4.0% on, are Speaking and Grammar. Speaking was selected
as the skill most in need of improvement by nearly 20%
30.9% 31.9% 7.0%
of the pupils, but only 7.1% reported it as the skill most
48.0% 15.1% time was spent on in class. Despite Grammar being the
area the highest number of pupils had selected in terms of
53.0% 14.8%
class time, only 17.4% had selected this as the area most
in need of improvement, which puts it in third place after
40% 60% 80% 100%
Writing and Speaking. This pupil feedback suggests a level
Not Very Often
Blank of anxiety about productive skills, and that pupils would
welcome a more communicative approach to teaching.
In 2013/14, Reading (30%) and Listening (33%) were also
the two most common responses in terms of strongest
skill. In 2014/15, vocabulary and grammar were additional
options available. The two most frequently selected areas
last year in those most in need of improvement were also
Speaking (39%) and Writing (32%).
However, this data needs to be considered alongside the
exam results. In their performance on the Cambridge English:
Preliminary for Schools test, the best performance was
in Speaking, with 43.0% at the target B1 level or higher,
followed by Writing (42.8%). Reading was the weakest
skill, with 27.3% at the target level or higher. This indicates
that pupil perception of ability may not be that accurate.
It would appear that the comparatively less time devoted
to Speaking did not have a negative impact on Speaking
performance. It may be that the time devoted to Grammar,
in addition to that spent on Writing activities, had a positive
impact on pupil performance in Writing. They may also be
doing Speaking or Writing activities to practise grammar.
The comparatively lower amount of time spent on Reading
and Vocabulary may have had a negative impact on
Reading performance, or pupil performance may have been
affected by pupil perception that this was a skill which they
could pay less attention to.
Factor and regression analysis
Three thematically linked areas were identified in the
construct of attitude towards English, with overlapping
Reading items from the questionnaire. The clearest grouping was
the set of questions which focus on use of spoken English
Writing
in class, which included the question on frequency of
Listening pair work and group work. A feature of the second group
Speaking appears to relate to individual preferences and use of time
Grammar
by individuals in class.
The addition of the question regarding what skill most
Vocabulary
time was spent on for this group is because the data for
Blank
these three items was scaled in a different way from the
other questions. Pupils selected a skill from six possible
options (Reading, Writing, Listening, Speaking, Grammar
or Vocabulary) whereas for the other questions they
responded to a Likert scale.
Since the questions in this group are a mixture of scales
and types of question, the regression analysis for this group
was limited to those questions in the first component on
the use of spoken English in the classroom. There was a
clear relationship between higher individual use of English
AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|33

by the pupils and higher scores in all four skills. The largest Diagnostic and motivational aspects of
increase is in Writing (3.353) for pupils who report that assessment
they frequently speak in English to their teacher. However Pupils were asked to comment on three statements related
for pupils talking to each other, and for the teacher speaking to assessment in general (see Figure 6). Over 80% of pupils
in English, there is a relationship with a decrease in score agreed that tests helped them to focus on what they need
for all skills, when controlling for variance in other items. to learn. In terms of the diagnostic use of assessment by
teachers, over 60% of pupils agreed, or strongly agreed that
In addition to regression analysis, a correlational analysis
their teacher gave them information about their strengths
was carried out to determine if there was a correlation
and weaknesses after taking an English test. A similar
between the skill area pupils reported spending most time
percentage agreed that tests motivated them to study. In
on in class, and scores on the four skills. The results are
Year 1, the percentages were almost identical: 82% agreed
shown in Table 7. The first column indicates which skill area
tests helped them to focus on what they need to learn, 60%
pupils reported as the one most time was spent on in class.
agreed their teacher gave them diagnostic information, and
The last four columns show the correlation between the
64% agreed that tests motivated them to study.
pupil response and the score on the test for that skill.
Figure 6: Pupil attitude to test taking in general
The table illustrates several negative correlations, such as
that pupils who respond that they are spending the most 32.0% 52.3% 8.8%
Tests are important because they help me
time on that skill tend to receive lower scores on the skills focus on what I need to learn
than pupils who do not report spending the most time on 18.9% 45.1% 19.6% 7.1%
After taking an English test, the teacher
that skill. gives me information about my
strengths and weaknesses
One possible interpretation, if pupil perceptions are accurate, 17.1% 47.4% 24.3%
Tests are important because they
is that teachers are spending more time on those skills motivate me to study

where the pupils do need more help, and that this is having
0% 20% 40% 60% 80% 100%
a negative impact on other skills, when most time is spent
Strongly Agree Agree Disagree
on Reading, Listening or Vocabulary. Spending most time Strongly Disagree Not Sure Blank
on Writing appears to have little impact on scores. More
time spent on Speaking has a beneficial effect in all skills. It
These results show that assessment is valued for its
appears teachers should spend as much time as possible
diagnostic aspects, and that pupils are generally motivated
on Speaking. Individual use of English by pupils and a
by tests. From this we can conclude that the majority
greater focus on Speaking in class is recommended. Further
of pupils have a positive attitude towards tests and
investigation of the use of pair and group work is necessary
assessment. However, Figure 6 shows that almost 40% of
to discover why this is not linked to an increase in scores.
pupils do not receive diagnostic feedback, and nor do they
find tests motivating. A smaller percentage, just over 15%,

Table 7: Use of class time and scores

Pupil response Reading Score Writing score Listening score Speaking Score

Reading Pearson Correlation .025** .029** .020** .021**

Sig. (2-tailed) <.001 <.001 <.001 <.001
N 83,239 83,249 83,177 74,381
Writing Pearson Correlation .006 .008* .002 .001
Sig. (2-tailed) .095 .023 .567 .779
N 83,239 83,249 83,177 74,381
Listening Pearson Correlation .044** .040** .031** .029**
Sig. (2-tailed) <.001 <.001 <.001 <.001
N 83,239 83,249 83,177 74,381
Speaking Pearson Correlation .080** .074** .083** .084**
Sig. (2-tailed) <.001 <.001 <.001 <.001
N 83,239 83,249 83,177 74,381
Grammar Pearson Correlation .035** .028** .016** .010**
Sig. (2-tailed) <.001 <.001 <.001 .006
N 83,239 83,249 83,177 74,381
Vocabulary Pearson Correlation .036** .039** .035** .035**
Sig. (2-tailed) <.001 <.001 <.001 <.001
N 83,239 83,249 83,177 74,381

** Correlation is significant at the 0.01 level (2-tailed).

* Correlation is significant at the 0.05 level (2-tailed).

UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
34|ISSUE 65 / FEBRUARY 2017
did not agree that tests helped them to focus on what they
need to learn. The different percentages in response to the
statements show that some of these pupils did not receive
diagnostic information from their teachers, but they did
use the test results to assess their own performance. This
would account for the discrepancy between the responses
for the less positive group to the three statements.
Figure 7: Pupil confidence
1.1%
9.6%
14.6%
Very Confident
Quite Confident
Not Very Confident
37.6% Not At All Confident
37.1%
Blank
Moving on to pupil confidence, as shown in Figure 7, when
asked how confident they felt about taking English exams
in general, the pupils fall into two groups of broadly the
same size, with just over half indicating they were quite
confident or very confident about taking tests, and the
remainder indicating they were not very confident or not
at all confident.
In Year 1, the most common response was quite confident
(36%) followed by not very confident (32%). As with this
year, over half indicated they were quite confident or very
confident about taking tests (58%).
Attitudes to Cambridge English: Preliminary for
Schools
A similar level of confidence was reported when pupils
were asked whether or not they were worried about taking
the Cambridge English: Preliminary for Schools test. Just over
half the pupils agreed or strongly agreed that they were
worried, and the remaining pupils indicated that they were
not worried (see Figure 8). It is important to bear in mind
that a certain level of anxiety over a test is normal for test
takers. These results indicate a slightly smaller percentage
of pupils who were worried about taking the Cambridge
English: Preliminary for Schools test, compared to the levels
reported for test anxiety in general.
In Year 1, 60% of pupils agreed or strongly agreed that they
were worried about taking the Cambridge English: Key for
Schools test. The slightly higher percentage (despite the
lower target level of the exam last year) possibly reflects
the unfamiliarity with Cambridge English exams, or external
exams in general in the first year of the initiative. Figure
9 shows to what extent taking the Cambridge English:
Preliminary for Schools exam motivated learners. 57.5%
indicated agreement or strong agreement with the statement
that taking the test motivated them to continue with their
English studies. This is similar to the percentage of pupils
who indicated that they find tests motivating in general.
UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES
Figure 8: I was worried about my performance on the Cambridge
English: Preliminary for Schools test.
3.1%
3.6%
20.6% Strongly Agree
16.8% Agree
Disagree
Strongly Disagree
Not Sure
24.1% 31.9%
Blank
Figure 9: Sitting the Cambridge English: Preliminary for Schools
motivates me to continue to study English.
3.1%
6.5%
18.6%
9.6% Strongly Agree
Agree
Disagree
23.3% Strongly Disagree
Not Sure
38.9%
Blank
The majority of pupils in Year 1 also agreed that sitting the
Cambridge English: Key for Schools test motivated them to
continue studying English.
Certification in English
The majority of pupils agree that it is essential to have an
internationally recognised certificate of English proficiency.
Approximately three quarters of pupils agreed or strongly
agreed that it was important to have a certificate of this
kind. In Year 1, 74% agreed it was important. If we compare
these responses to enrolment rates for the Cambridge
English: Preliminary for Schools certificate, a much lower
percentage of the school population, approximately 15%,
enrolled for a certificate prior to sitting the test. The greater
agreement rates regarding the desire for a certificate in
general, compared to those for this exam, may indicate
more interest in certification at higher levels, or in obtaining
certification at a later stage of the educational cycle, or a
possible lack of awareness that certificates were available.
As can be seen in Figure 10, approximately three quarters
of the pupils, responding to short CEFR level descriptors,
identified B1 (the target level) or above as the highest level
of certification they would like to attain, with C2 the most
common response (35.3%). However, just below a quarter
of pupils indicated they would be happy with A1 (15.7%) or
A2 (8.7%).
The pupils in Year 1 had a similar pattern of responses in
terms of aspirations. The most common response was also
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|35

C2 (46%). In terms of the target level, a larger proportion may be because of the complex relationship between
(86%) identified B1 or higher as the level of certification motivation and test anxiety. General positive attitude to
they would like to attain. assessment has a relationship with increase in scores on
all skills. There is not a significant relationship between
Figure 10: Aspirational level reported by pupils the diagnostic aspects of tests and those pupils who
report that the teacher regularly gives them feedback on
their performance, when controlling for variance in other
A1 items. This suggests that there is a need to investigate in
15.7% moredetail the characteristics of feedback being given
C2 topupils.
35.3% A2
8.7%
Main findings
B1
The main findings here are summarised under the four key
13.8%
investigation points for this phase of the impact study, and
C1 B2 also in terms of the constructs under investigation for each
13.1% 13.5% of the four areas, as detailed in Table 1: Research constructs
in Year 2.

Attitude towards English

Factor and regression analysis
The findings show that many pupils are positive about
The first group of statements analysed relate to the
studying English at school and enjoy using English, and for
usefulness of assessment and its diagnostic aspect. The
those pupils who like English generally there is evidence of
second group is made up of statements which relate to
increased performance. English is perceived as intrinsically
self-confidence. The statement I am worried about taking
and extrinsically useful. There is an increase in scores
the Preliminary for Schools exam was coded to have a higher
in Speaking and Writing for pupils who feel English is
value for Strongly Disagree/Disagree, so that it would be
important to get information from the internet. Enjoyment
a measure of self-confidence, rather than worry. Analysis
of lessons or activities such as travelling abroad or finding
showed it fitted into the first grouping, about the usefulness
a good job, do not have an impact on scores for this group.
of assessment, indicating that there is a relationship
This suggests that pupils may not find these factors
between being worried about exams, and appreciating their
relevant because of a lack of certainty about their future.
value. Table 8 shows the relationship between the items
that make up the Attitude to assessment scale, and the
Characteristics of the home environment
scores on the tests.
The findings show high levels of engagement with English
Pupil motivation and attitude to assessment has both where the pupils have ready access to it. The level of
positive and negative relationships with scores. This English ability of parents is not perceived to be particularly

Table 8: Attitude to assessment and scores

Question Skill Unstandardised Coefficients Standardised Coefficients t Sig.

B Std. Error Beta

Taking the Preliminary for Schools exam W 1.023 .091 .055 11.284 <.001
motivates me to continue to study English.
Sp .847 .084 .052 10.044 <.001
Tests are important because they motivate L .738 .112 .032 6.586 <.001
me to study.
Tests are important because they help me W 1.001 .168 .029 5.961 <.001
focus on what I need to learn.
It is essential to have an internationally Sp .572 .097 .029 5.915 <.001
recognised certificate of English proficiency.
How do you feel about taking English exams R 5.185 .046 .504 112.318 <.001
in general?
W 6.902 .062 .498 111.352 <.001
L 6.430 .056 .514 114.735 <.001
Sp 5.716 .057 .483 99.902 <.001
What level of English would you like to R 1.983 .047 .186 42.364 <.001
achieve?
W 2.792 .063 .195 44.430 <.001
L 2.113 .057 .164 37.201 <.001
Sp 2.131 .058 .173 36.605 <.001

W=Writing; Sp=Speaking; L=Listening; R=Reading

UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
36|ISSUE 65 / FEBRUARY 2017
high by pupils, and the level of parental involvement, in
terms of talking about homework or classes, cannot be
generalised, and most likely varies from household to
household. There is an increase in scores for students
whose parents level of English language proficiency
is perceived to be higher. However, there is an inverse
relationship between parental involvement and scores; that
is, those pupils reporting their parents are involved have
lower scores than pupils who report their parents are not
involved. It may be that pupils who perform well are not so
closely monitored by parents. This needs to be investigated
further in Year 3.
Access to English at home via the internet, the media and
reading material in English, or opportunities to interact with
speakers of English is variable, and there is a relationship
between frequency of activity in English in the home
environment and increased levels of performance.
Characteristics of the learning environment
Although there is evidence of some pair and group work
taking place in class, there is enough evidence to suggest
that classes tend to be didactic. It is also clear that there is
more use of English in class by the teacher than the pupils.
Pupils report that they use English to interact with the
teacher, but that interactions with classmates in English
are less common. Further investigation into the nature
of pair and group work is necessary to discover why this
is not linked to an increase in scores. General liking of
English is related to an increase in scores, but in contrast,
enjoyment of English classes is not linked to an increase
in scores. Investigation into pair and group work may help
to identify why enjoyment of classes also has no link to
increasedscores.
Reading and Listening are the two most common skills
selected as the skills the pupils feel they are strongest in.
Grammar receives comparatively more class time and
Speaking receives comparatively less class time than other
skills. Greater use of English in class by pupils and a greater
focus on Speaking in class are both recommended, since
these are linked to an increase in scores in all skills.
Attitude towards assessment
The findings in this section show that the majority of pupils
find tests and assessment motivating and useful. A positive
attitude to assessment in general has a relationship with
increases in scores, but there is no significant relationship
between increase in scores and feedback. This would
suggest that either the feedback is not something pupils are
able to act on or implement, or that they do not understand
the feedback. More investigation is needed to identify
the characteristics of feedback being given to pupils on
their test performance and why this is not linked to an
improvement in test scores. There is interest in obtaining
certification in English, although this is not reflected in the
enrolment rates for the Cambridge English: Preliminary for
Schools certificate.
UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES
Recommendations
The analysis has shown that pupils confidence and high
parental proficiency in English have a positive relationship
with pupil achievement. However, the recommendations
in this section are limited to actions which IAVE may be
able to influence and the Ministry of Education should
implement. In addition, some of the recommendations
could be addressed and included in teacher training
programmes, either at an initial phase or within in-
servicetraining.
1. Use of English outside the classroom has a relationship
with improvement in scores. All the activities discussed
in this article, such as use of the internet, the media,
real-life interaction in English and reading in the
language should be encouraged by teachers and, where
possible, incorporated into class activities or assigned
ashomework.
2. More individual use of English by pupils and more focus
on Speaking in class are both recommended.
3. Further investigation at classroom level, either through
class observation or teacher focus groups, could help to
identify the characteristics of pair and group work being
carried out in terms of tasks and content. This may help
to identify why there is no identified link between this
type of activity and increases in scores.
4. Further investigation is also recommended to identify
the nature of feedback being given to pupils. This
may help to identify why there is no link between this
feedback and increases in scores.
Conclusions
The findings fall under three main areas. These are
motivation and attitude, access to English and engagement
with English, and, finally, environmental factors. Many of
these findings are positive and echo the views expressed
by the Grade 9 population in the first year of the initiative,
such as a general liking of English, use of English outside
class, enjoyment of lessons and so on. However, given
the comparatively small size of the cohort of Grade 9
respondents in Year 1, only very tentative comparisons can
be made.
Motivation and attitude
The analysis highlighted many positive attitudes to English
and aspects of English which pupils found motivating, such
as their English lessons, the usefulness of English beyond
the classroom, and the importance of English, assessment
and certification. There was a relationship between English
scores and enjoyment of English, positive attitudes to
assessment and aspirational level.
Just over half the pupils (54.7%) reported they are
confident using English.
Almost 80% of pupils enjoy their lessons at school and
like English generally.
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|37

There is a relationship between liking English and better
performance in all four skills.
The majority of pupils consider getting a good job,
travel, interaction with non-speakers of Portuguese and
access to a good university or college all to be important
reasons for studying English.
countries or getting a good job and increased scores in
any skill. This perhaps reflects a more positive attitude
towardsthe usefulness of English for more immediate
needs, rather than long-term ones or those activities which
may not be perceived to be particularly likely or predictable
by some pupils.

80% of pupils agreed that tests helped them to focus on
what they need to learn.
General positive attitude to assessment has a
relationship with increases in scores on all skills.
Three quarters of pupils agreed or strongly agreed that it
was important to have a certificate.
Three quarters of pupils identified B1 (the target level) or
above as the highest level of certification they would like
to attain.
The higher aspirational level of pupils has a relationship
with increased performance in all skills.
Less than 25% of pupils indicated they would be happy
with achieving an A2 (15.7%) or A1 (8.7%) level. However
this may simply be an indication of the level of English
these pupils perceive they will need for the type of jobs,
educational opportunities or way of life they will pursue
after leaving school. It could also be an indication that
students are not fully aware of the meaning of CEFR levels.
Access to and engagement with English
Grade 9 pupils are engaging with English outside the
class, and there is evidence of a relationship between an
increase in scores and use of English outside the classroom.
In the case of Writing, there is also a relationship
between increased scores and a feeling that English is
important for using the internet and speaking to non-
Portuguesespeakers.
The most frequent use of English is listening to songs, or
watching TV programmes and films in English.
Almost 80% of pupils think that English is important
orvery important for accessing information on the
internet.
There is a relationship between increases in scores in
all skills and regular activity and exposure to English
outside the classroom.
There is a relationship between feeling that English is
important for using the internet and increased scores.
This is more noticeable in Writing and Speaking.
Over 60% of pupils report using English to speak to
tourists or visitors.
There is a relationship between feeling that the ability
to speak to non-Portuguese speakers is important
and increased performance, which is most apparent
inWriting.
The analysis did not find any evidence of a significant
relationship between the importance of travelling to other
Environmental factors
The analysis found evidence of supportive factors for
learning and teaching, both at home and at school,
suchasuse of English in the classroom and parental ability
in English.
There is a relationship between a higher level of parental
ability in English and higher scores on all skills.
Pupils report that teacher use of English in the class
isfrequent.
There is a relationship between use of English by the
pupils and increased scores in all fourskills.
Pupils would like to spend more time on improving
Speaking and Writing skills.
Over 60% of pupils agreed that their teacher gave them
information about their strengths and weaknesses after
taking an English test.
There are several findings which help to give a picture of
the learning environment for Grade 9 pupils, and are useful
for considering what further steps can be taken towards
improvement. These are further subdivided into parental
proficiency level, skills in English, classroom dynamics
andassessment.
Parental proficiency
The majority of parents were perceived to be below the
target level of the exam.
A higher level of parental ability in English correlates
positively with higher scores.
There is a correlation between pupils who report regular
discussion of homework, and lower scores. These findings
do not mean that parental involvement causes a decrease
in scores, only that there is a correlation. It contradicts
the findings of other large scale studies, such as TIMSS,
PIRLS and PISA. This section of the questionnaire should
be expanded in future assessments, as this relationship
between parental attitudes and student scores needs
further investigation.
Skills in English
The skills identified by the highest number of pupils as in
need of improvement were Writing and Speaking.
There is a discrepancy between pupil perceptions of
their strengths and weaknesses, and their performance
in different skill areas.
When asked what most class time was spent on, the
two most common responses were Grammar and
Writing.

UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
38|ISSUE 65 / FEBRUARY 2017
Teachers are spending more time on those skills where
the pupils do need more help, perhaps at the expense of
other skills.
Enjoyment of lessons at school does not have a
significant relationship with increase in performance in
any skill.
There is a pupil perception of Writing as a skill in need of
improvement, and that this is a skill regularly worked on
in class. However, pupil perceptions of their strengths and
weaknesses are not necessarily reflected in their exam
performance. It is recommended that teachers look at the
results by skill for their schools and decide which skills to
prioritise for subsequent academic years.
Class dynamics
There is evidence of some pair and group work taking
place, but pupils working alone is most common. Pair and
group work is not currently contributing to an increase in
scores. There is not a great deal of pair and group work
taking place. It may be that it is not currently contributing
to scores because it is not that prevalent. Teachers need to
consider how pair and group work can be used to increase
the amount of English used by pupils in class, and how
to ensure this English is of a sufficient standard, and also
how to increase the amount of pair and group work in their
classrooms. The findings suggest that more controlled
practice, monitoring, correction and feedback are necessary.
Assessment
Over 60% of pupils report receiving diagnostic feedback,
and that they find tests motivating. There is not a
significant relationship between test scores and those
pupils who report that the teacher regularly gives them
feedback on their performance. These findings suggest
that diagnostic feedback is not having a positive impact,
either because it is not taking place, or because the pupils
feel that the feedback they receive does not have any
practical application. Teachers need to consider what type
of feedback is most beneficial to pupils, and what can be
done to make this more accessible to pupils. As a practical
next step, further investigation at classroom level is needed
UCLES 2017 AN INVESTIGATION OF PORTUGUESE STUDENTS ATTITUDES TO ASSESSMENT AND CAMBRIDGE ENGLISH EXAMS
CAMBRIDGE ENGLISH : RESEARCH NOTES
to ascertain the nature of feedback being given to pupils on
their assessment performance, and also what kind of pair
work and group work activities are being carried out. Use
of spoken English in class and engagement with English
outside the class should be promoted and encouraged.
The study so far has highlighted many positive outcomes,
which should not be set aside simply because they are
what might have been predicted, following on from the
findings in the first year. It is clear that positive pupil
attitudes are related to increased proficiency. It is to the
credit of the Grade 9 pupils, their families and teachers that
there is such a positive attitude to assessment and English
generally, and that pupils enjoy their classes at school, and
engage with English when they have the opportunities to
do so.
References
Bernat, E and Gvozdenko, I (2005) Beliefs about language learning:
Current knowledge, pedagogical implications, and new research
directions,TESL-EJ 9 (1), 121.
Council of Europe (2001) Common European Framework of
Reference for Languages: Learning, Teaching, Assessment, Cambridge:
Cambridge University Press.
Creswell, J W and Plano Clark, V L (2011) Designing and Conducting
Mixed Methods Research (2nd Edition), Thousand Oaks, CA: Sage.
Csizr, K and Drnyei, Z (2005) The internal structure of language
learning motivation and its relationship with language choice and
learning effort,The Modern Language Journal89 (1), 1936.
Drnyei, Z (2003) Attitudes, orientations, and motivations
in language learning: Advances in theory, research, and
applications,Language Learning53 (1), 332.
Horwitz, E (2001) Language anxiety and achievement,Annual
Review of Applied Linguistics21, 112126.
Noels, K A (2001) New orientations in language learning
motivation: Towards a model of intrinsic, extrinsic, and integrative
orientations and motivation, Motivation and Second Language
Acquisition 23, 4368.
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|39

Assessing the English language progress of students

in a trilingual education framework in the Basque
Country
BELINDA CERD CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT

ANDREW BLACKHURST CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT

CHRISTINE WALKER CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT

Introduction relatively independently, and to continue their studies or use

In 2011, upon initiation of the Marco de Educacin Trilinge
(the Trilingual Education Framework project, hereafter
referred to as MET), Cambridge English Language
Assessment started working with the Basque Institute for
Research and Evaluation in Education (ISEI-IVEI), which
is maintained by the Basque Governments Department
of Education, Universities and Research. The aim of this
collaboration was to assess the English language progress
of students participating in the MET programme in
order to support ISEI-IVEI in evaluating the effects of the
simultaneous introduction of Spanish, English and Basque
as languages of instruction for content subjects.
This article focuses on the performance of the 2014 primary
and secondary school students who formed the second
cohort taking part in the MET programme (henceforward
MET II). This group of students took an English language
test in writing, reading, listening and speaking and we
firstly compare their performance in 2014 against their
performance in the 2012 English benchmarking test which
they also sat. Secondly, we compare the performance of
these primary and secondary students (the experimental
group) against two control groups which were added to the
MET programme in 2014. Thirdly, we evaluate the strengths
and weaknesses in writing and speaking performance
in the 2014 primary and secondary cohorts and suggest
how their performance can be improved. We conclude by
highlighting some characteristics that contribute to the
success of such programmes based on our experience of
the METprogramme.
Context of the MET programme
Since the 1990s, the Department of Education, Universities
and Research has been promoting Plurilingual Education
in the Basque Country, and in 2011 the MET programme
was initiated to promote trilingual education. Trilingual
education refers to the use of three languages as the means
of instruction for content matter subjects (e.g. Geography,
Maths, Art) and it can support students to achieve sufficiently
high levels of proficiency in each language to allow them
to continue to develop their language knowledge and skills
any of these languages in their future employment.
The Basque Country is a region in the north of Spain close
to the Pyrenees and the French border with a population
of just over 2 million (according to the 2011 census, see
en.eustat.eus/estadisticas/tema_159/opt_0/ti_Population/
temas.html). The official languages are Basque and Spanish,
and English is the most commonly taught foreign language.
All students are taught either in Basque or Spanish and
have English as a foreign language. Not all schools teach
content subjects through the three languages and official
time allocated to each language can vary. According to
Seewald and Beetsma (Eds) (2002) one of the problems in
the past was the weak position of Basque and the Basque
language skills of teachers. This is generally not considered
the case nowadays and one of the current challenges is
related to the lack of a need to speak English outside the
classroom and the varying levels of motivation. However,
parents are felt to be fully supportive of trilingual education
and the Basque Department of Education, Universities and
Research supports the development of trilingual education.
The MET programme, which started in 2011, aimed to be
able, via both the benchmarking tests administered by
Cambridge English for English and the tests administered
by ISEI-IVEI for Spanish and Basque, to evaluate the
effects of the simultaneous introduction of these three
languages as languages of learning and teaching. The
MET programme was designed to provide flexibility and
autonomy as participation in the programme was voluntary
and participating schools were encouraged to develop
their own linguistic approach whilst keeping within certain
minimum requirements. For this reason it was felt more
appropriate to refer to MET as a Trilingual Framework
rather than a model, with the framework promoting Basque,
English and Spanish whilst consolidating bilingualism and
activating English. The following criteria were identified for
MET:
Students would not need to take a language proficiency
test in order to take part in the test and participation
was the decision of the childs family.
Each language (Basque, Spanish and English) had to be
both a language of instruction and a subject in its own

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
40|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

right. In practice this means 5 hours a week in primary
and 6 hours a week in secondary. From there each
school could develop its own linguistic proposal.
Preference was given to schools in different regions and
with different systems in both primary and secondary, in
order to ensure adequate representation.
Preference was also given to schools whose permanent
teaching staff had minimum B2 level in the Common
European Framework of Reference (CEFR, Council
of Europe 2001) or who had obtained a recognised
qualification in teaching a foreign language at
primarylevel.
Participating schools would have access to funding
and support to compensate for the time necessary to
implement and co-ordinate the project.
The objective of the external assessment was not
to assess the performance of individuals but rather
evaluate the cohort as a whole and therefore the project
in order to be able to inform future policy.
The experimental phase has been accompanied by a
plan of linguistic improvement and professional training
for the participating teachers.
A call was therefore made for participation which was
accepted by 25 primary and 15 secondary schools in
MET phase I (henceforward MET I) and 57 primary and
32secondary schools in MET phase II (henceforward
MET II, the focus of this article). Whilst the overall aim
of the MET programme was to evaluate the performance
in Basque, Spanish and English of primary and secondary
school children, MET also aimed to study the factors which
could influence performance as well as provide support for
decisions regarding CEFR attainment levels at the end of
primary and secondary level education.
Benchmarking student performance
group of students. In 2014, in addition to the experimental
group that sat the benchmarking test in 2012, there were
two control groups in each of the primary and secondary
cohorts of students. Control group type 1 students were
students receiving the minimum legal requirement of
English teaching and control group type 2 students were
receiving additional lessons in English. Table 1 details the
students who took part in MET II 2012/14 cohorts which
we report on further in this article.
Research questions
The MET II programme aimed to answer a number of
research questions (RQs), which can be summarised as
follows:
1. What is the CEFR level of the MET II primary and
secondary student populations who took part in the
benchmarking study in 2014 and how do their results
compare with their results from 2012?
2. What is the CEFR level of the MET II primary and
secondary populations in the 2014 control group
split according to whether they are receiving the legal
minimum of lessons in English or whether they receive
additional lessons and how does it compare with the
experimental group?
3. How does the performance across all four skills compare
between the 2014 experimental, control type 1 and
control type 2 groups?
4. What are the strengths and weaknesses of the speaking
and writing performances in the 2014 experimental,
control type 1 and control type 2 groups?
This article presents a summary of the findings, along
with some of our observations and recommendations that
followed from this research. The following section describes
the English language tests that the MET II participants sat
in 2014.

Since 2011, groups of primary and secondary school

students have taken an English language test each year
within the MET programme outlined above. In 2011 and
Cambridge English tests
2012 reading, writing and listening skills were tested, which
Cambridge English test: Primary Reading and
were joined by speaking in 2013 and 2014. A control group
Writing
was included in MET I in 2011 and 2013, whilst in 2012 the
This test consists of 51 items (30 reading and 21 writing
benchmarking test focused exclusively on the experimental
items) measuring from A1 to B1 in the CEFR. The

Table 1 Participating students information for MET II 2012/14

Year Level Number of students Age range Type Benchmarking tests taken

2012 Primary (4th year) 1,998 910 Experimental Reading, listening

2012 Secondary (1st year) 1,127 1213 Experimental Reading, writing, listening
2014 Primary (6th year) 1,880 1112 Experimental Reading, writing, listening
2014 Primary (6th year) 403 1112 Control group type 1 Reading, writing, listening
2014 Primary (6th year) 418 1112 Control group type 2 Reading, writing, listening
2014 Secondary (3rd) 911 1415 Experimental Reading, writing, listening
2014 Secondary (3rd) 127 1415 Control group type 1 Reading, writing, listening
2014 Secondary (3rd) 222 1415 Control group type 2 Reading, writing, listening

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|41

reading section (Parts 14) focuses on a range of skills:
understanding lexis, grammar and cohesion in a variety of
different text types (from dialogues to adapted magazine
articles) through different task types (from matching
prompt sentences, to notices and 3-option multiple
choice). The writing section (Parts 58) focuses on lexical
knowledge and accuracy using a variety of interactive task
types. The main writing task tests the candidates ability
to produce a short text; employing correct grammar, the
correct level of appropriacy and attention to detail, with
the emphasis on the effectiveness of the communication
over lexical and grammatical accuracy. Table 2 shows the
contents of the test.
Cambridge English test: Primary Listening
This test consists of 25 items measuring CEFR A1 to B1
levels. The test focuses on a range of listening skills and on
understanding of lexis, grammar and cohesion; and uses
a variety of different text lengths and interactions, ranging
from short dialogues to a medium-length monologue; and
a range of different task types, from multiple choice with
pictures to note-taking (see Table 3).
Cambridge English test: Primary Speaking
This test consists of two parts in which an assessor
examines two candidates for 810 minutes. The test is
designed to assess candidates at CEFR A1 to B1 level. The
test focuses on each candidates ability to produce a range
of accurate grammar and lexis, and on their pronunciation
and ability to interact, both with the interlocutor and each
other (see Table 4). The examiner submitted more detailed
feedback on each pair of candidates (based on a Cambridge
English brief) in order to provide quotes on speaking
proficiency for the report.

Table 2: Cambridge English test: Primary Reading and Writing

Part Task type and format Task focus Number of items and
range of difficulty

Reading
1 Matching five prompt sentences to eight notices (preceded Gist understanding of real-world notices. Five items at A1 level
by an example). Reading for main message.
2 Five gapped sentences with three multiple-choice options Reading and identifying appropriate vocabulary. Five items at A1 to A2
for each gap. The sentences are linked by a storyline. level
3a Eight discrete 3-option multiple-choice items. Understanding functional language in verbal Eight items at A1 to A2
exchanges. Reading and identifying appropriate level
responses.
3b Five matching items (plus an integrated example) in a Understanding functional language in verbal Five items at A1 level
continuous dialogue. For each item, selecting from eight exchanges. Reading and identifying appropriate
possible responses. responses.
4 Three short texts adapted from authentic magazine articles. Reading for detailed understanding and main Seven items at A2 level
Seven 3-option multiple-choice items. ideas.
Writing
5 Five dictionary-definition-type sentences (plus an example) Reading and identifying appropriate lexical Five items at A1 to A2
requiring accurate completion of appropriate word. items and producing them with correct spelling. level
6 Narrative text of kind that candidates could be expected Reading and identifying appropriate lexical Ten items at A1 to B1
to write. Ten gaps (plus integrated example) to fill items and producing them with correct spelling, level
appropriately and accurately with one word. with focus on structure and lexis.
7 Two short authentic-style input texts to prompt completion Reading and writing down appropriate words or Five items at A1 to A2
of five spaces in a notes section output text with words or numbers with focus on content and accuracy. level
numbers (plus an integrated example).
8 A short input text (email) with three questions to prompt a Writing a short email of 2535 words using an Targeted at A2 level
written response (also email) in answer to those questions. appropriate range of lexis and grammar, and
including appropriate writing conventions for
the audience and text type.

Table 3: Cambridge English test: Primary Listening

Part Task type and format Task focus Number of items and
range of difficulty

1 Five short neutral or informal dialogues, with five Listening to identify key information (times, Five items at A1 to
accompanying discrete 3-option multiple-choice items prices, days of week, numbers etc.). A2 level
with pictures, plus one example.
2 A longer informal dialogue with five accompanying matching Listening to identify key information. Five items at A1 to
items (plus one integrated example) and eight options. A2 level
3 A longer informal dialogue, with five accompanying 3-option Listening to identify key information. Five items at A1 to
multiple-choice items (plus one integrated example). A2 level
4 A longer neutral dialogue, with accompanying notes which Listening and writing down specific information Five items at A1 to
have five gaps to fill with one or more words or numbers. (including spelling of names, places etc. as A2 level
dictated on recording).
5 A longer neutral dialogue, with accompanying notes which Listening and writing down specific information Five items at A1 to
have five gaps to fill with one or more words or numbers. (including spelling of names, places etc. as A2 level
dictated on recording).

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
42|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Table 4: Cambridge English test: Primary Speaking

from short dialogues to a longer monologue, through a
Part Task type and format Task focus Range of range of different task types, from multiple choice with
difficulty
pictures to note-taking (see Table 6).
1 The interlocutor asks Using language A1 to B1
candidates questions normally level
on familiar topics associated with Cambridge English test: Secondary Speaking
(school, family etc.). meeting people for This test consists of four parts in which an interlocutor exa-
the first time and
giving information mines two candidates for 1012 minutes. The test is designed
of a factual to assess candidates at CEFR A2 to B2 level and focuses
personal kind,
including spelling on candidates range and accuracy of grammar and lexis;
where appropriate. pronunciation; discourse management; and interaction, both
2 The interlocutor Using factual A1 to B1 with the interlocutor and each other, summarised in Table 7.
explains the activity information of level An assessor was present in a representative number of both
using a standard rubric, a non-personal
then issues candidates kind related to primary and secondary exams.
with prompt cards from daily life, such as
which they ask and price, address and
answer questions about available facilities.
real-world notices. Test marking and scoring
Clerical marking of items and examiner marking of
Cambridge English test: Secondary Reading and extended texts was carried out by Cambridge English.
Writing For the objectively scored test components (Reading and
This test consists of 25 items for reading and two items Listening) cut-off scores for CEFR levels were arrived at
for writing measuring from CEFR A2 to B2. The reading using a Rasch Ability table based on item difficulty values
section focuses on a range of skills; on understanding that were anchored to the Cambridge English Common
lexis, grammar and cohesion in a variety of different text Scale. The cut-off scores were criterion based (i.e. based on
types, ranging from short notices to adapted magazine established proficiency levels). For the subjectively scored
articles; and a range of task types, from 3-option multiple test components (Speaking and Writing) writing responses
choice to matching information in one text to another. The were marked by trained Cambridge English examiners using
writing section focuses on a variety of skills, from lexical assessment scales linked to the CEFR and Speaking tests
and grammatical range and accuracy to knowledge of were conducted by trained Cambridge English Speaking
appropriate register and writing conventions, through two Examiners using assessment scales linked to the CEFR. The
different task types: writing an email and writing a letter mark schemes were based on the analysis of the features
(see Table 5). of performance at different levels, with marks awarded
according to these mark schemes, and cut-off scores being
Cambridge English test: Secondary Listening criterion based in relation to these mark schemes. Overall
This test consists of 25 items measuring CEFR A2 to B2 cut-off scores for CEFR levels were based on a combination
levels. The test focuses on a range of listening skills and of the cut-off scores for the objectively and subjectively
understanding of lexis, grammar and cohesion with a scored elements. Candidate performance was reported in
variety of different text lengths and interactions, ranging terms of CEFR levels.

Table 5: Cambridge English test: Secondary Reading and Writing

Part Task type and format Task focus Number of items and
range of difficulty

Reading
1 Five short texts (notes, labels, emails,
signs) with accompanying discrete
3-option multiple-choice items.
2 Five items in the form of descriptions of
people to match to eight short adapted-
authentic texts.
3 An adapted-authentic long text with 10
true/false items.
4 An adapted-authentic long text with five
4-option multiple-choice items.
Writing
1 Candidates are prompted to write a short
informal email. The prompt takes the
form of a rubric with three bullet-pointed
items to include.
2 Candidates are prompted to write a
longer informal letter. The prompt
takes the form of an extract from a
letter containing three questions to be
answered.
Reading real-world notices and other short texts for the main Five items at A2 to
message. B1 level
Reading multiple texts for specific information and detailed Five items at A2 to
comprehension. B1 level
Processing a factual text. Scanning for specific information while Ten items at A2 to
disregarding redundant material. B2 level
Reading for detailed comprehension; understanding attitude, opinion Five items at A2 to
and writer purpose. Reading for gist, inference and global meaning. B2 level
Writing an informal email of 3545 words focusing on Targeted at B1 level
communication of three specific content points, using appropriate
lexis, grammar and writing conventions to convey the message.
Writing an informal letter of around 100 words focusing on Targeted at B1 level
answering three questions from the prompt, using appropriate
lexis, grammar and writing conventions. Candidates are assessed
using assessment scales consisting of four subscales: Content;
Communicative Achievement; Organisation; and Language.

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
CAMBRIDGE ENGLISH : RESEARCH NOTES
Table 6: Cambridge English test: Secondary Listening
Part Task type and format
1 Seven short neutral or informal dialogues or monologues
with seven accompanying discrete 3-option multiple-choice
items with pictures.
2 A longer interview with one main speaker, and six 3-option
multiple-choice items.
3 A longer monologue, with accompanying notes which have
six gaps to fill with one or more words.
4 A longer informal dialogue, with six true/false items.
Table 7: Cambridge English test: Secondary Speaking
Part Task type and format
1 The interlocutor asks candidates questions on familiar topics
(school, family etc.).
2 The interlocutor explains the task through a standard rubric.
Candidates are given a visual stimulus and a task, then asked to
work together to complete it.
3 A colour photograph is given to each candidate in turn and they
are then asked to talk about it for approximately a minute. Both
photographs relate to the same topic.
4 Using a standard rubric, the interlocutor explains that
candidates should speak together on the same topic as in Part 3.
Test population
The school students taking the benchmarking tests in
2014 comprised 4,026 students selected by ISEI-IVEI. Of
these, 2,701 primary school students and 1,260 secondary
students have results for Reading, Writing and Listening,
with the remaining candidates being absent for one or
more components. A subset comprising 420 primary
and 378secondary students also completed a Speaking
component. The students were split into three categories:
experimental trilingual project students (MET II)
control group type 1 students taking statutory English
lessons
control group type 2 students taking additional English
lessons.
Control groups 1 and 2 were selected by ISEI-IVEI to
represent the MET II sample as closely as possible
according to their results in a diagnostic evaluation in 2013,
the linguistic model they were following and the socio-
economic status of their families. Information supplied by
ISEI-IVEI indicated some differences in the composition of
these three groups which might affect their performance.
For example, 77% of control group type 2 schools are
subsidised, rather than state schools, compared to 48% in
the trilingual and 43% in control group type 1. Furthermore,
UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
ISSUE 65 / FEBRUARY 2017|43
Task focus Number of items and
range of difficulty
Listening to identify key information. Seven items at A2 to
B1 level
Listening to identify specific information and Six items at A2 to B1
detailed meaning. level
Listening to identify, understand and interpret Six items at A2 to B1
information. level
Listening for detailed meaning, and to identify the Six items at A2 to B2
attitudes and opinions of the speakers. level
Task focus Range of difficulty
Giving information of a factual personal kind. The A2 to B2 level
candidates respond to questions about present
circumstances, past experiences and future plans.
Using functional language to make and respond A2 to B2 level
to suggestions, discuss alternatives, make
recommendations and negotiate agreement.
Describing photographs and managing discourse, A2 to B2 level
using appropriate vocabulary, in a longer turn.
The candidates talk together about their opinions, A2 to B2 level
likes/dislikes, preferences, experiences, habits etc.
the families of students in control group type 2 tend to
have a higher educational level than the other two groups
(this applies both to the families whose children are in
state schools, as well as to those which are in subsidised
schools). Given that higher levels of education may well
translate into higher socioeconomic status, this might
contribute to stronger learning outcomes for this group.
In addition, the data also reveals that children in control
group type 2 have greater exposure to English outside class
via extra-curricular activities. Again this may indicate that
students in the different groups vary in their motivation or
are differently placed to take advantage of the additional
tuition in English, and this in turn would explain some of the
variability in their performance in the benchmarking study
in 2014.
We now summarise the results according to the research
questions posed above.
Results
This section reports on test performance based on the 2014
test administration and provides a picture of proficiency
levels of students according to the CEFR. The analyses
were carried out on data from experimental, control
group type 1 and control group type 2 students. We firstly
answer RQ1: What is the CEFR level of the MET II primary
 44|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

and secondary student populations who took part in the
benchmarking study in 2014 and how do their results
compare with their results from 2012?
Overall performance in 2014
Overall performance: Primary
Figure 1 shows that for all primary students, 4.44%
achieved B1 CEFR level; just over a quarter (26.14%) are at
A2 level; 40.43% are at A1 level; and 28.99% are below the
A1 CEFR level.
Figure 1: Cambridge English test: 2014 distribution of primary
student scores by CEFR level (N=2,701)
consisted of Reading, Writing and Listening in both years).
In 2012, the CEFR levels reported for primary candidates
were A1 and A2 and above, whilst for secondary the CEFR
levels reported were A2 and B1 and above.
Comparative performance 201214: Primary
The primary students comparative performance
(N=1,816) is shown in Figure 3, which shows that
overall the percentage of primary students achieving a
Level A2 and above increased from 6.66% to 33.20%.
Correspondingly, the percentage of students at a
level below A1 on the CEFR has dropped from 48.51%
to20.32%.

45%
40.43% Figure 3: Cambridge English test: Distribution of primary student
40% scores by CEFR level in 2012 and 2014
Percentage of primary students

35%
30% 28.99%
26.14%
25% A2 & above 6.66% 33.20%

20%
15%
10%
5% 4.44%
A1 44.82% 46.48%
0%
B1 A2 A1 Below A1

Overall performance: Secondary

Figure 2 shows that over a third (34.76%) of all secondary
Below A1 48.51% 20.32%
students achieved B2 level. The majority (42.94%) are at
Level B1; 21.43% are at Level A2; and 0.87% are at Level A1
and below.
2012 2014

Figure 2: Cambridge English test: 2014 distribution of secondary

student scores by CEFR level (N=1,260) Of those experimental primary students who were at Level
42.94% Below A1 in 2012, 67.88% had improved by one or two
45%
CEFR levels by 2014; and 48.16% of those at Level A1 in
Percentage of secondary students

40%
34.76% 2012 had improved by one CEFR level.
35%
30%
Comparative performance 201214: Secondary
25% The secondary students comparative performance is
21.43%
20% shown in Figure 4 (N=863) based on their Reading
15% and Listening scores in 2012 and 2014. Figure 4 shows
10% that the percentage of secondary students achieving
5% Level B1 and above has increased from 32.79% to
0.87%
0% 77.06%. Correspondingly, the percentage of students
B2 B1 A2 A1 & below at Level A1 and below has dropped from 28.74% to
0.81%.
Having summarised the MET II participants overall
Of those experimental secondary students who were at
performance in 2014, we now address the second part of
Level A1 and below in 2012, 97.18% had improved by one or
RQ1: How do MET II students 2014 results compare with
two CEFR levels by 2014; and 84.64% of those at Level A2
their results from 2012?
in 2012 had improved by one CEFR level.

Comparative overall performance 201214
There were 1,816 primary and 863 secondary experimental
students who had results from both 2012 and 2014 tests. In
2012, the primary benchmarking test consisted of Reading
and Listening papers. In order to make a comparison
between the years only the Reading and Listening results
from 2014 were used (as the secondary benchmarking test
To summarise the findings in relation to the first research
question, in 2014 the results showed that in primary,
33.20% of students are Level A2 and above, 46.48% are A1
and far fewer than in 2012 are below A1, 20.32%. In 2012,
6.66% were A2, 44.82% were A1 and 48.51% were below
A1. In primary, the strongest skill was speaking with 53%
of the experimental group at Level A2. The weakest skill

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|45

Figure 4: Cambridge English test: Distribution of secondary Figure 5: Cambridge English test: Distribution of primary student
student scores by CEFR level in 2012 and 2014 scores by CEFR level for each category
MET I Control type 1 Control type 2
experimental
B1 4.52% 0.74% 7.66%
B1 & above 32.79% 77.06%

A2 27.55% 10.67% 34.69%

A2 38.47% 22.13%

A1 39.36% 47.89% 38.04%

A1 & below 28.74% below A1 28.56% 40.69% 19.62%

0.81%

2012 2014
To compare the performance of the experimental students
was writing with 38% of the experimental group below A1.
and the two control groups, independent t-tests were run
This is not a surprising result as we would expect primary
for each component which showed that the differences
students to make slower progress in third language learning
between the experimental group and the control group
than in second language learning and slower progress in
type1 group was statistically significant by p>0.05 in
primary than in secondary. Neither is the skill performance
all three skills, with the experimental group consistently
surprising as in primary the focus is on speaking and
achieving higher mean scores (e.g. reading mean was
listening (in L1, L2 and L3) and so we would expect primary
12.40 over 10.25, writing 10.58 over 7.58). The differences
school students to perform best in speaking and struggle
between the experimental group and the control group type
most with writing.
2 were statistically significant for all three components,
At secondary level, the results showed an even greater with the experimental group achieving lower mean scores
improvement in performance than at primary level. Of in all three skills.
the 863 secondary experimental students who had
results for both 2012 and 2014, in 2014, 77.06% of Overall performance by grouping: Secondary
students were at Level B1 and above by the third year of Of the 1,260 secondary school students included in the
secondary school with most of the remainder at Level A2 analysis, 911 were in the experimental group, 127 were in
(22.13%). Compared to 2012 findings, we can see that control group type 1 and 222 were in control group type 2.
the improvement has been significant: in 2012, 32.79%
Figure 6 shows that the overall performance of the
of students were at B1 and above, 38.47% at A2 and
control group type 2 students was stronger than both
28.74% A1 and below. In 2014 the secondary students
the experimental students and the control group type 1
strongest skill is writing with 92% of the experimental
students, with a higher percentage of students achieving
group achieving Level B1 and above; the weakest skill is
CEFR Levels B2 and B1 (90.99% compared to 77.17% and
speakingwith 51% of the experimental group at Level B1
58.26% respectively).
and above.
Figure 6: Cambridge English test: Distribution of secondary
Next, we report on RQ2: What is the CEFR level of the student scores by CEFR level for each category
MET II primary and secondary populations in the 2014
MET I
control group split according to whether they are receiving experimental
Control type 1 Control type 2
the legal minimum of lessons in English or whether they
receive additional lessons and how does it compare with
B2 33.04% 14.17% 53.60%
the experimental group?

Overall performance by grouping: Primary

Of the 2,701 primary school students included in the
B1 44.13% 44.09% 37.39%
analysis, 1,880 were in the experimental group, 403 were
in the control group type 1 and 418 were in the control
group type 2. Figure 5 shows that the overall performance
of the control group type 2 students was stronger than
A2 22.06% 38.58% 9.01%
both the experimental students and the control group type
1 students, with a higher percentage of students achieving
CEFR Levels B1 and A2 (42.35% compared to 32.07% and
11.41% respectively).
A1 & below 0.77% 3.15%

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
46|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

To compare the performance of the experimental students Figure 8: Cambridge English test: Distribution of control group
and the two control groups, independent t-tests were run type 1 primary student scores by CEFR level for each skill
for each component, which showed that the differences B1 A2 A1 Below A1
between the experimental group and control group type1 3%
were statistically significant for all three components Speaking 24% 31% 42%
(p>0.05), with the experimental group achieving higher 3%
mean scores in all three skills (e.g. listening mean was Listening 18% 49% 30%

15.64 over 13.46, reading mean 19.60 over 17.27). The

Writing 13% 31% 56%
differences between the experimental group and control
group type 2 were statistically significant for all three 2%
Reading 11% 64% 23%
components, with the experimental group achieving lower
mean scores in all three skills. Now we focus on RQ3:

0%

0%
10

20

30

40

50

60

70

80

90
How does the performance across all four skills compare

10
between the 2014 experimental, control type 1 and control
Figure 9: Cambridge English test: Distribution of control group
type 2 groups? type 2 primary student scores by CEFR level for each skill

Performance by skill and grouping B1 A2 A1 Below A1

Performance by skill: Primary Speaking 15% 31% 43% 11%

Performance by skill for each grouping is shown in Figures
7, 8 and 9. Figure 7 shows that for the experimental primary Listening 17% 33% 39% 11%
group the strongest skill was speaking with 53% of the
students achieving a level of A2 or above, and the second Writing 4% 37% 28% 31%

strongest skill was listening (41% at A2 or above). The

Reading 14% 28% 42% 16%
most difficult skill for this group was reading, where only
30% of students achieved a level of A2 or above. However
in writing, 33% achieved a level of A2 and above and a
0%

0%
10

20

30

40

50

60

70

80

90

10
significant number, 38%, were at below A1 (compared
to 18% in reading and similar percentages in speaking Performance by skill: Secondary
andlistening). Performance by skill for each category is shown in
Figures 10, 11 and 12. In Figure 10 we can see that for the
Figure 7: Cambridge English test: Distribution of experimental
primary student scores by CEFR level for each skill
experimental secondary group the strongest skill was
writing (92% at B1 and above), followed by listening (68%
B1 A2 A1 Below A1 at B1 and above) and then reading (65% at B1 and above).
The weakest skill in this group was speaking, with 51% at B1
Speaking 7% 46% 30% 17%
or above and 13% at A1 and below.

Listening 13% 28% 40% 19% Figure 10: Cambridge English test: Distribution of experimental
secondary student scores by CEFR level for each skill
2%
Writing 31% 29% 38% B2 B1 A2 A1 & below

Reading 8% 22% 51% 18% Speaking 6% 45% 36% 13%

2%
Listening 41% 27% 30%
0%

0%
10

20

30

40

50

60

70

80

90

10

1%
Writing 38% 54% 7%
Figure 8 shows that for control group type 1 the strongest
2%
skill was speaking (27% at A2 or above) followed by Reading 32% 33% 32%
listening (21% at A2 or above). In writing, 13% of students
were at A2 level and in reading, 11% were at Level A2 and
0%

0%
10

20

30

40

50

60

70

80

90

10

2% at B1. The most difficult skill for control group type 1

students was writing as the majority (56%) were at below
A1 level.
Figure 9 shows that for the control group type 2 primary
students the strongest skill was listening with 50% of
candidates achieving a level of A2 or above, closely
followed by speaking (46% at A2 or above). As with the
control group type 1, their weakest skill was writing with
41% of candidates achieving a level of A2 or above and 31%
achieving A1 and below.
In Figure 11 we can see that for the control group type 1
secondary group the strongest skill was also writing (80%
at B1 or above), with listening the second strongest (52%
at B1 or above). In this group the weakest skill was again
speaking with 41% of candidates achieving a level of B1 or
above and 18% at A1 and below.
In Figure 12 we can see that for the control group type 2
secondary candidates the strongest skill was writing, with
98% achieving a Level B1 or above. Listening was their

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|47

Figure 11: Cambridge English test: Distribution of control group Student performance in productive
type 1 secondary student scores by CEFR level for each skill
skills
B2 B1 A2 A1 & below
2% Our analysis of the productive test components highlighted
Speaking 39% 41% 18% the strengths and weaknesses in performance in writing
1% and speaking among Basque primary and secondary
Listening 22% 30% 47% students. In terms of the strengths and weaknesses in
speaking, we found that whilst strong primary students
Writing 19% 61% 12% 8%
were able to ask questions and respond confidently,
sometimes initiating exchanges and developing their
Reading 20% 26% 50% 5%
answers, weak students struggled with question forms
and with maintaining basic conversation. Based on our

0%
%

%
%

%
%

%
%
0%

observations, we made some recommendations which

40

60

80

90
20

50
30

70
10

10
include practice with question forms, information gap
second strongest skill, with 89% achieving B1 or above, and activities and opportunities to improve confidence.
their weakest skill was speaking (71% at B1 or above).
At secondary level we found that strong students were
Figure 12: Cambridge English test: Distribution of control group able to use a variety of lexis and grammatical forms and
type 2 secondary student scores by CEFR level for each skill developed the interaction whilst weaker students lacked
confidence, were unable to produce extended language and
B2 B1 A2 A1 & below
they also struggled at times with tenses and verb forms.
Speaking 7%
For secondary students, we recommended that role play
64% 22% 6%
activities would work well alongside activities which involve
Listening 59% 30% 11%
obtaining information from a partner and reporting back
to the group, along with developing attention to functional
1%
Writing 56% 42% language to develop confidence. With practice, secondary
1%
students would be able to contribute to accurate and
Reading 55% 28% 16% extended speech which is necessary for all students to
achieve levels B1 and above in speaking.
0%

0%

In terms of strengths and weaknesses in writing, we found

10

20

30

40

50

60

70

80

90

10

that strong primary candidates included all the content

points and used basic structures correctly but weaker
When we compare the performance across the three
students struggled, sometimes writing only 1- or 2-word
groups in both primary and secondary in 2014 we see that
answers , and they lacked an awareness of basic phrases
the strongest group overall was control group type 2 and
which would have helped them to get their message
the weakest group was control group type 1. In primary,
across. Based on our analysis we recommended some
34.69% of control group type 2 students are at Level A2,
more exposure to written texts which could come before
27.55% of the experimental group are at Level A2 and
or after a speaking activity and practice in writing short
10.67% of the control group type 1 students are at LevelA2.
emails or notes to focus on basic phrases on topics relevant
If we look at secondary, the situation is similar with again
to students daily lives. If we turn to secondary level
control group type 2 being the strongest of the three
writing, we found that strong candidates organised their
groups. In secondary, 53.60% of control type 2 students
writing into paragraphs with some evidence of linking and
are achieving B2 compared to 33.04% of the experimental
cohesive devices. They were more adventurous in their use
group and 14.17% of the control group type 1. Analysis of
of language and included relevant content which showed
the characteristics of the control group type 2 suggests
that they were reading and processing the rubric and input
that in addition to the hours dedicated to language
material. Weaker candidates did not consider the target
learning in school, the socioeconomic status of families,
reader or check their work and used basic structures and
the increased exposure to the target language outside the
lexis, sometimes incorrectly. We recommended that paying
classroom and the motivation and positive attitudes of
attention to the process of writing, planning and checking
teachers and students all contribute to positive results in
their own work and drawing attention to text cohesion and
second and third language learning. This may help explain
coherence during reading activities would help secondary
the performance of control group type 2 especially in
students to improve their writing, which are necessary for
comparison to the experimental group.
students to achieve levels B1 and above in writing.
In the following section we will summarise the students
strengths and weaknesses in their spoken and written
performances in the 2014 MET II cohort, to answer RQ4: Summary of results
What are the strengths and weaknesses of the speaking
In providing a summary of this study, we refer back to the
and writing performances in the 2014 experimental, control
key research questions which it aimed to investigate.
type 1 and control type 2 groups?

UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
48|ISSUE 65 / FEBRUARY 2017
RQ1: What is the CEFR level of the MET II primary
and secondary student populations who took part
in the benchmarking study in 2014 and how do their
results compare with their results from 2012?
The benchmarking study showed that almost half of those
students, who took part in the benchmarking study in 2014,
were A1 (40.43%), while over a quarter (28.99%) were below
A1 and 26.14% were A2. The results in 2014 show that the
primary students in the experimental group were strongest in
speaking, followed by listening, and then writing and reading.
This is not unusual in primary students as there tends to be a
focus on speaking at this level, and young learners in general
tend to perform better in speaking. However, while control
group 1 shows the same pattern, control group 2 actually did
best, on average, in the Listening test.
In order to be able to provide a valid comparison between
the results in 2012 and the results in 2014 we removed the
Speaking and Writing tests as in 2012 the primary students
only took a Reading and Listening test. It should be noted
that in 2012 there were no control groups. The benchmarking
shows that between 2012 and 2014 there has been a
significant increase in the number of primary candidates at
Level A2 and above. In 2012, 6.66% of students were A2
and above, while in 2014 this has increased to 33.20% and
although a significant percentage continue to be A1 in 2014
(46.48%), far fewer (20.32%) are below A1. Based on these
results, an attainment level of A1 was recommended as the
most realistic target for primary students.
The benchmarking study showed that over a third have
achieved Level B2 (34.76%) by the third year of secondary
but that the majority (42.94%) are at Level B1. The results
in 2014 show that secondary students in contrast to the
primary students were strongest in writing followed by
listening and reading, with speaking as, on average, the
weakest skill.
The benchmarking shows that between 2012 and 2014
there has been a significant increase in the number of
secondary candidates at Level B1 and above. In 2012,
32.79% of students were B1 and above, while in 2014 this
has increased to 77.06%. Also, whereas in 2012 28.74%
were A1 and below in the first year of secondary, by 2014, in
a very positive development, only 0.81% of students remain
at that level. The recommended future attainment level
for secondary students was thus set at B1, with a view to
increasing to B2.
RQ2: What is the CEFR level of the MET II primary
and secondary populations in the 2014 control
group split according to whether they are receiving
the legal minimum of lessons in English or whether
they receive additional lessons and how does it
compare with the experimental group?
The strongest group in primary in 2014 was the control
group type 2 (34.69% at Level A2) followed by the
experimental group (27.55% at Level A2) and finally the
control group type 1 (10.67% at Level A2). While for these
three groups we see that in the sixth year of primary the
majority of students are A1, closer analysis shows that for
UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
CAMBRIDGE ENGLISH : RESEARCH NOTES
control group type 2 there is an almost equal number at
Level A2 with 34.69% at A2 and only marginally more at
A1 (38.04%). For the experimental group, the difference
between A1 and A2 is more significant with 39.36% at A1
and 27.55% at A2.
If we compare the performance therefore between control
group type 1 and control group type 2 we can see that,
not surprisingly, increased exposure to English at school is
having a noticeably positive effect on English performance.
The fact that a higher percentage of control group type2
students are achieving A2 by the end of primary than those
in the trilingual group would suggest that there are particular
characteristics of the control group type 2 whichare
contributing to their success. These might includethe
educational level of one or both of their parentswhich in turn
would have an effect on the socioeconomic status of the
family, the number of hours of English and the type ofschool
attended i.e. state run versus private or subsidised.
If we look at the differences in performance between the three
secondary groups who took part in the study in 2014 we see
a similar pattern to the results from primary, with a higher
percentage of students (53.60%) from control group type 2
achieving B2 compared with only 14.17% from control group
type 1 and 33.04% from the experimental group. If we compare
the performance therefore between control group type 1
and control group type 2 we can see that, not surprisingly,
increased exposure to English at school is having a significantly
positive effect on English language performance.
The fact that a higher percentage of control group type 2
students are achieving B2 by the third year than those in
the trilingual group would suggest that there are particular
characteristics of the control group type 2 which are
contributing to their success.
In terms of skills, we see that for all three groups in 2014 the
strongest skill in secondary is writing, the second highest
performance is in listening followed by reading and finally
speaking. As with primary, the control group type 2 achieved
the highest score in all four skills.
Students following an increased exposure to an English
curriculum where they are receiving up to 33% of the
curriculum in English are four times more likely to achieve
Level B2 in the third year of secondary, and we would
reasonably expect that if the trend were to continue, by
the end of compulsory secondary education (fourth year),
perhaps 60% or even slightly more might achieve Level B2.
RQ3: How does the performance across all four
skills compare between the 2014 experimental,
control type 1 and control type 2 groups?
Results for primary in 2014 show that for the experimental
and control group type 1 groups speaking is clearly the
strongest skill. As mentioned earlier, this is not unusual at this
age where there tends to be a focus on speaking and primary
school children in general tend to perform better in speaking.
In the experimental group, 53% achieved A2 or above and in
control group type 1, 27% achieved A2 or above. In the control
group type 2 listening was slightly stronger than speaking:
CAMBRIDGE ENGLISH : RESEARCH NOTES
50% achieved A2 or above in listening compared to 46% in
speaking. The weakest skill for all three groups in primary was
writing. In the experimental group, 38% were below A1. In
control group type 1, 56% were below A1 and in control group
type 2, 31% were below A1.
Results for secondary in 2014 show that across all
three groups writing is clearly the strongest skill. In the
experimental group, 92% achieved B1 or above. In control
group type 1, 80% achieved B1 or above and in control group
type 2, 98% achieved B1 or above. The weakest skill for all
three groups in secondary was speaking. In the experimental
group, 51% were at B1 or above and 13% at A1 and below. In
control group type 1 41% were at B1 or above and 18% at A1
and below. In control group type 2 71% were at B1 or above
and 6% were at A1 and below.
RQ4: What are the strengths and weaknesses of
the speaking and writing performances in the 2014
experimental, control type 1 and control type2
groups?
Speaking
A selection of primary and secondary candidates were
observed throughout the 2-week speaking window. In the
primary cohort, analysis of scores and observations showed
that the strong candidates were able to produce grammatically
correct and extended utterances. This meant they developed
the conversation and strong and medium candidates in
the experimental group were able to form questions. Weak
candidates in the experimental group struggled with forming
questions and struggled to maintain simple exchanges. When
we compared control group type 1 and control group type 2,
we noted that control group type2 were generally stronger
than control group type 1, scoring higher on Pronunciation and
Interactive Communication. The observers comments did not
identify clear differences between control group type2 and
the experimental group, although overall control group type 2
did secure marginally higher ratings in three of four categories
(Grammar and Vocabulary, Interactive Communication and
Global Achievement (i.e. the mark reflecting the examiners
impression of the candidates overall performance)). It is
recommend that students practise speaking in class with
a partner in activities which require exchanging or finding
out information. Practice with question forms via games or
mingling activities will not only help students develop towards
A2, but will also increase motivation and especiallyconfidence.
At secondary level, analysis of scores and observations
shows that strong candidates demonstrated accuracy and
a range of grammar and vocabulary. They were able to
respond appropriately and develop the conversation. Weak
candidates produced isolated words with little linking and
were unable to extend the conversation. Analysis does show,
however, that they tended to score consistently higher on
Pronunciation than on the other criteria. If we concentrate
on the control groups, we see that again control group type 2
were generally stronger overall and that the strong candidates
in this group scored consistently high on all parts of the test.
Weak candidates in both groups showed excessive hesitation
and the use of short and disconnected utterances. Again,
the observers comments did not identify clear differences
UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
ISSUE 65 / FEBRUARY 2017|49
between control group type2 and the experimental group,
although control group type 2 secured higher ratings in all
analytic categories. In order to improve the performance of
secondary students in speaking, we would recommend that
a number of skills are integrated into the language classes
so that students progress across all skills. Students at this
age should be encouraged to take part in role play activities
related to topics and issues relevant to their lives.
Writing
A selection of the writing scripts for primary and secondary
were analysed in order to evaluate the strengths and
weaknesses in their performance. Scripts were split for
each group into strong, medium and weak performances.
At primary level, strong candidates structured their scripts
well with appropriate opening and closing phrases. Strong
candidates also remembered to include all content points
and were therefore reading the instructions. Weak students
did not read the instructions and did not organise their text
in an email format.
If we look at the difference between the control groups we
see that control group type 2 were clearly stronger. Weak
candidates in control group type 2 were stronger than the
weak candidates in control group type 1. However, some
candidates in both groups struggled with using grammatical
forms correctly and lacked the vocabulary to be able to
communicate their points clearly. It is recommend that
teachers move towards integrating two or all skills into each
activity towards the end of primary and support the spoken
form with written consolidation. If not used already, the
classroom physical environment can be used to consolidate
what is presented orally first. Students should be encouraged
to read the instructions first and practise reading so that
rubrics and tasks are clearly understood. Finally, we would
recommend that students are encouraged to re-read and
check their work so that some errors such as simple spelling
mistakes can be avoided.
At secondary level, analysis of the writing scripts showed
that strong candidates in the experimental groups included
all relevant content points and that these were well
communicated with the appropriate tone. Strong candidates
organised their email well using cohesive devices to help
guide the reader. They also used a range of vocabulary
although there was some inconsistent use of tenses.
Weak candidates showed a lack of awareness of the target
reader and the genre. This meant that the email was not
organised appropriately with little cohesion and linking.
Some candidates did not use paragraphing or full stops and
students in the weak groups may not have checked their
work before submitting, as evidenced by the greater number
of minor mistakes.
If we compare control group type 1 with control group type2
we see a similar pattern to primary. Control group type 2
were stronger overall than control group type 1. Generally
control group type 2 used a greater variety of grammar and
vocabulary more accurately. Medium candidates in control
group type 2 also avoided repetition and sentences were more
effectively linked. Errors in control group type 1 tended to be a
little more basic than those in control group type 2.
50|ISSUE 65 / FEBRUARY 2017
If we compare the control groups with the experimental
groups we see that performance between control group type
2 and the experimental group was very similar except among
the medium candidates. Students in the medium group in
control group type 2 tended to achieve B2 level compared to
B1 level among students in the experimental group. Control
group type 1 were the weakest of the threegroups. In order
to improve the performance of all secondary candidates
in writing and enable all students to acquire B1 or B2 level
English, students should be exposed to more varied lexis
and grammatical forms via reading texts and support should
be given, in class, for organising and planning a piece of
written text which can be consolidated at home. Attention
to linking words and cohesive devices as well as exposure
to fixed expressions and functional language will also raise
awareness as well as improve confidence and motivation.
Conclusions
The study findings point to the success of the MET trilingual
programme in improving standards of English in the Basque
Country. Over the course of the four years during which the
study took place, students in the experimental programme
in both MET I and MET II have made significant progress.
Overall, 68.52% of secondary students in MET I in 2013
were at levels B1 and above (in 2011, this was 31.48%). For
the MET II students in secondary the results were even more
positive. In 2012, 32.79% of secondary students were B1 and
above. In 2014, this has increased to a significant 77.06%.
Based on these results, the attainment level for the fourth
year of secondary school was recommended to be set at B1,
with this being increased to B2 in future.
If we look at the results for primary, we can also see that
the trilingual programme is having a positive effect on the
Appendix 1: Writing tasks
Primary writing task
Read the email from your English friend, Jacky.
From: Jacky
To:
Im so happy you can come to my house for lunch. What
time will you get here? What kind of food do you like?
What shall we do after lunch?
Write an email to Jacky and answer the questions.
Write 25-35 words.
Write the email on your answer sheet.
UCLES 2017 ASSESSING THE ENGLISH LANGUAGE PROGRESS OF STUDENTS IN A TRILINGUAL EDUCATION FRAMEWORK IN THE BASQUE COUNTRY
CAMBRIDGE ENGLISH : RESEARCH NOTES
language level of students. 40.37% of primary students in
MET I in 2013 were at levels A2 and above (in 2011, only
9.42%). For the MET II students the results in primary are
equally positive, with 6.66% of primary students at A2 and
above in 2012, rising to 33.20% in 2014. Based on these
results, a realistic attainment level for the majority of students
at the end of primary school was recommended to be set at
A1 level, which was viewed as an interim target to be revised
upwards in the future. Whilst setting A1 as a target exit level
for primary students might seem low, we had to consider
that the rate of progress for young learners who speak or are
learning three or more languages will often in their primary
years be slower than can be seen in secondary years.
In terms of future research we envisage further studies in
order to evaluate the MET programme, monitoring progress
over time, and ensuring its positive impact by identifying
and fostering good practice. The trilingual context makes
this especially interesting given ISEI-IVEIs exploration of
not only the progress of the English language level but also
progress in Spanish and Basque. The progress in English that
we have observed over time has been encouraging and we
hope to apply our experience of this trilingual programme to
the challenges and opportunities of other bi-and trilingual
learning environments.
References
Council of Europe (2001) Common European Framework of Reference
for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge
University Press.
Seewald, S and Beetsma, D (Eds) (2002) Trilingual Primary Education
in Europe. Inventory of the Provisions for Trilingual Primary Education in
Minority Language Communities of the European Union, Leeuwarden:
Fryske Akademy.
Secondary writing task 1
You went to a concert last night which you enjoyed.
Write an email to your British friend, Charlie. In your email,
you should:
tell Charlie who played at the concert last night
explain why you enjoyed it
invite Charlie to go to another concert with you.
Write 35-45 words on your answer sheet.
Secondary writing task 2
This is part of a letter you receive from an English friend.
I really like sport. Which sports do you like to
play and which ones do you prefer to watch? Do
you think sport is important? Why?
Now write a letter, answering your friends questions.
Write your letter in about 100 words on your answer
sheet.
CAMBRIDGE ENGLISH : RESEARCH NOTES
Which factors affect English language attainment?
Astudy of school students in Chile
AGNIESZKA WALCZAK CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
GRAEME HARRISON CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT, ARGENTINA
MERCEDES MURATORIO CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT, MEXICO
CAROLINA FLORES AGENCIA DE CALIDAD DE LA EDUCACIN, CHILE
SOFIA BRUNNER AGENCIA DE CALIDAD DE LA EDUCACIN, CHILE
COREEN DOCHERTY CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
Project background and aims
The Chilean Ministry of Education has made a
commendable investment in improving English language
learning outcomes for over a decade. In order to better
understand the impact of the Ministrys initiatives and as
part of a strategy to improve learner proficiency in English,
the Simce Ingls exam was introduced in 2010 to provide
an objective measure of the English ability of 3rd grade
secondary school students in Chile on a biennial basis.
Since 2012, Cambridge English has provided Simce Ingls
(taking over from the Educational Testing Service), which
reports on Reading and Listening performance from A1 to
B1 levels of the Common European Framework of Reference
for Languages (CEFR, Council of Europe 2001). The findings
in this article are based upon the data gathered from the
2012 and 2014 administrations of the Simce Ingls test,
including candidate results and background socioeconomic,
geographical and educational data from questionnaires
administered to students, parents and teachers. The
questionnaire data provides a profile of English language
learning and teaching and when combined with the test
data, the results can be used to identify factors which may
be associated with different learning outcomes.
The aim of this research project was to investigate how
these background factors affected achievement in the 2014
Simce Ingls test and also to compare levels of achievement
between the test administrations in 2012 and 2014. The
results reported here can be used by the Ministry, at a
national level, to inform strategic decisions related to
education policy and practice designed to support their
objectives of raising standards in English language learning.
The results can also facilitate data-driven decisions and
recommendations to sustain conditions for success, and,
crucially, help to identify any areas for improvement.
Educational context
The educational system in Chile can be roughly divided into
three sectors: public (municipal) schools, voucher-subsidised
private schools and private schools. A unique feature of
the Chilean education context is that the majority of pupils
attend voucher-subsidised private schools rather than public
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
ISSUE 65 / FEBRUARY 2017|51
schools. English language education varies by provider across
sectors; however, the public and voucher-subsidised schools
are advised to deliver their curriculum as described below.
English language education in Chile begins in what is
termed the 5th year of the basic cycle (5th year of primary
school). The number of English language hours in the
5th and 6th year curricula was raised in 2011 from two to
three per week to bring them into line with the rest of the
compulsory education system. There are also long-term
plans to extend English classes to the first four years of
primary school and introduce English Content and Language
Integrated Learning (CLIL) in maths and science subjects.
In 2013, the curriculum for English language teaching
underwent a complete review and a new curriculum based
on communicative language learning methodology was
produced. However, research conducted by PIAP (Ministry
of the General Secretariat of the President, Ministry of
Education and Ministry of Economy, Promotion and
Tourism 2014) found that 17% of state English teachers
(public and voucher subsidised) had no teacher training. It
is therefore likely that many English teachers do not have
the methodological training to effectively implement this
new curriculum. To combat this, there are now plans to
standardise teacher training in higher education institutions.
PIAP also found (Ministry of the General Secretariat
of the President, Ministry of Education and Ministry of
Economy, Promotion and Tourism 2014) that a third of a
representative sample of teachers tested had an English
language level of B1 or below in the CEFR, which could
indicate that these teachers may be limited in their ability
to support learners in developing high levels of English
proficiency. In a bid to rectify this, PIAP is currently offering
language tuition to English teachers. Furthermore, PIAP
offers various other programmes to students and teachers,
such as English language summer camps and scholarships
for study abroad.
In order to gauge the effectiveness of the above
interventions designed to improve English teaching and
learning, it is essential to have a standardised measurement
of English proficiency, from which results-based decisions on
educational policy and practice can be made. In this context,
the biennial evaluation of students with the Simce Ingls test
52|ISSUE 65 / FEBRUARY 2017
is an important part of this general drive to raise English
standards. It serves as a census of language ability by which
the long-term effects of interventions can be measured.
As the Simce Ingls test is linked to the CEFR, part of the
impact of the test should be that classroom practice is
more focused on students achieving the skills described in
this communicative language framework. This means that
students are learning real-life language skills at an optimum
level for improvement. Furthermore, as candidates who
achieve a level of A2 or above in Simce Ingls receive an official
Cambridge English certificate which has some recognition
within Chile, another impact of the test is to provide a source
of motivation for both teachers and students.
In 2012, 18% of students achieved A2 or above in the Simce
Ingls test. This was an improvement on 2010, when only
11% achieved the same level. This would suggest that some
of the interventions mentioned above have contributed to a
positive effect on teaching and learning English, although it
is difficult to say which factors specifically are responsible,
and how those factors interact with the wider social context.
In order to support the Ministry in investigating these
factors, Cambridge English undertook this research project.
Research questions
The research design combined test data, questionnaire data
and contextual information in order to answer the following
research questions:
1. How did 3rd grade secondary school students perform
in the 2014 Simce Ingls test in terms of the CEFR?
2. How did performance of Chilean learners in the Simce
Ingls test change from 2012 to 2014?
3. Which factors play a role in English language attainment
and in particular in achieving a CEFR level of A2 or B1 as
measured by the Simce Ingls test?
The first research question is answered by analysing the
general profile of Chilean learners in terms of English
language attainment against the CEFR and exploring how
student performance compares across variables such as
gender, school type, socioeconomic profile of the school,
location (urban or rural) or geographic region.
Changes in performance between 2012 and 2014, the second
research question, are investigated for all Chileanlearners
and across contextual variables identified above.
Finally, this article explores which factors affect the
chances of Chilean learners achieving an A2 or B1 level in
English, focusing on variables which have been identified
in educational research as most likely to affect attainment
and are most amenable to change through policy
directives (e.g. onset of English language instruction).
In addition, variables which are beyond the control of
the Ministry but are of a particular concern in Chile
(e.g. socioeconomic status) were also investigated. The
variables selectedforinvestigation fall under two broad
categories as follows:
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
CAMBRIDGE ENGLISH : RESEARCH NOTES
Learner-related factors
Learner background and socioeconomic characteristics
Learner exposure to English at school
Motivation to learn English
Use of English language in free time
School-related factors
School-specific factors such as school type and
socioeconomic profile of the school
Geographic factors such as school location (urban vs rural)
and region
Methodology
In order to address the research questions of this study
we implemented a research design which combines
information from a number of data sources. The first
building block of this research design is information about
the level of English language attainment among Chilean
learners and how it changed over time using the Simce
Ingls test results for 2012 and 2014. Although the tests
for each year contained different items, the specifications
and the level of difficulty, calibrated using Item Response
Theory, were the same; therefore, the results for both the
2012 and 2014 cohorts can be considered comparable and
used to provide insights into whether the level of English
language proficiency achieved among 3rd grade secondary
school students in Chile changed over time.
The second building block of this research design is
information about possible factors that affect student
attainment in English. As such, information was gathered
from questionnaire data on a wide range of factors which
may be affecting English language learning. In order
to better understand why some learners in 3rd grade
secondary schools in Chile have achieved an A2 or B1
level in English, it was important to consider factors from
a wider context those related to student background
characteristics, exposure to English language, motivation
to learn English and use of English in students free time,
teacher qualifications and teaching methods and practices,
school characteristics and geographical differences in the
country. Figure 1 shows the multi-layered factors that are
brought together in this study to explain the differences in
English language attainment among 3rd grade secondary
school students in Chile.
Data collection instruments
This study brought together a number of data sources in
order to answer the research questions outlined above on
current language levels of the 3rd grade cohort, changes
in performance between 2012 and 2014 at this grade
and factors associated with attainment. The following
sections provide a brief description of the test and the
questionnaires which formed the basis of this study.
CAMBRIDGE ENGLISH : RESEARCH NOTES
Figure 1: Research design
English language attainment among
3rd grade secondary school students
in Chile
(benchmarking)
SCHOOLS
LEARNERS TEACHERS
Background Teaching School type
qualification
Socioeconomic Socioeconomic
characteristics Perception of status
preparedness to
Exposure to teach
English at school
Teaching practices
Motivation to
learn English
Use of English in
free time
Simce Ingls test
The Simce Ingls test was developed by Cambridge English
Language Assessment in collaboration with the Instituto
Chileno Britanico de Cultura and the Agencia de Calidad de
la Educacin (the latter is referred to as the Agencia in this
article).
The test was designed to measure Reading and Listening
skills of 3rd grade secondary school students and can
report on performance from Level A1 to B1 of the CEFR.
All items in the Simce Ingls test were selected from the
Cambridge English item bank, which contains calibrated
items produced under strict guidelines and put through
an extensive pretesting programme. This rigorous quality
assurance and measurement process ensures the validity,
reliability and fairness of all items used in this test.
The Reading section of the Simce Ingls tests in 2012 and 2014
contained seven parts with 50 items in total. The format of
the tasks was matching, multiple choice and multiple-choice
cloze with a time limit of 45 minutes. The Listening section of
the tests consisted of five parts with a total of 30 items, with
matching, multiple-choice and gap-fill task types and a time
limit of 30 minutes. The tests were administered in November
2012 and in NovemberDecember 2014. After administration
of the tests, papers were marked and each candidates
standardised score was calculated, giving equal weighting to
the Reading and Listening components. Cut scores were then
determined for A1, A2 and B1 levels. Candidates that achieved
an overall CEFR level of A2 or B1 received certificates.
Although tests administered in each year contained
different items, the specifications and the level of difficulty,
calibrated using Item Response Theory, were the same. As
such, the results for both the 2012 and 2014 cohorts can be
considered comparable.
Questionnaires and contextual data
In 2014, the Agencia surveyed 3rd grade secondary
school students in Chile, and their parents and
teachers. The parent and teacher questionnaires were
distributed a week before the students sat the Simce
Ingls test and collected on the day of the test while
Household income was a self-reported variable.
1
UCLES 2017
ISSUE 65 / FEBRUARY 2017|53
students completed their questionnaires just after they
completed the test. In addition, the Agencia compiled
contextual data related to all schools; the structure and
focus of boththequestionnaires and contextual data is
describedbelow.
GEOGRAPHIC
LOCATION
Urban/rural
The student questionnaire contained 19 questions about
Region reasons for learning English, exposure to English at
school and outside of school, teaching practices in the
classroom, learner attitude to English classes and English
teachers, parents level of English language knowledge,
motivationtolearn English and use of English in students
free time.
The parent questionnaire consisted of 17 questions about
the learners family (including socioeconomic status,
income and educational level), practices at home regarding
learning and exposure to English, and parents perceptions
regarding the motivation of their children for learning
English, whether they thought this learning was important,
and what learning practices their children used.
The teacher questionnaire contained 22 questions about
teaching qualifications that teachers have, their teaching
experience, teaching practices (including resources used
in classroom, providing feedback in class, use of English in
class), self-assessment in terms of preparedness to teach,
and attitudes of the school towards English teaching.
The contextual dataset contained information, among others,
on the type of school attended by learners, the socioeconomic
profile of schools and their location (region or a broader rural
or urban area). Selected questionnaire responses were used
to explore the factors affecting performance on the Simce
Ingls test, as described in the next section.
Constructs
In order to explain why some Chilean learners obtained
an A2 or B1 level of English, while others did not, we
chose a number of constructs to explore. The selection
of constructs was informed by the literature on English
language acquisition and insights from previous studies
conducted by Cambridge English Language Assessment.
In addition, constructs which were within the control of
the Ministry and could be modified through policy were
prioritised over those beyond the control of the Ministry.
The main exception to this is the selection of background
variables which could provide information on issues of
equity within the education system, as this has been
identified as an area of concern.
Background and socioeconomic characteristics
of learners: This construct is measured with three
variables gender of the learner, fathers and mothers
educational level, fathers and mothers ability to
speak, read and write English, and income of the
household in which a learner lives1; these variables are
WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
54|ISSUE 65 / FEBRUARY 2017
operationalised using questions from the student and
parentquestionnaires.
Learner exposure to English at school: This construct
is measured by a number of variables starting age of
English instruction, number of English lessons per week
that candidates have at school, language spoken in class;
these variables are operationalised using questions from
the student questionnaire.
Learner motivation to learn English: This construct
spans students attitude to learning English and attitude
to the Simce Ingls test; this construct is operationalised
using questions from the student questionnaire.
Use of English language in free time: This construct
is measured with a number of questions tapping
into everyday practices of using English language
such as watching TV or movies in English, reading
books, magazines or comics in English, browsing
English webpages, speaking English with other people
and using online chat in English; this construct is
operationalisedwith questions from the student
questionnaire.
School type and socioeconomic profile of the school:
These factors have been chosen here as they tap into
the educational structure and socioeconomic status
of schools in Chile; the contextual dataset provides
information on both.
Location of the school in terms of urban/rural area or
geographic region: Information on these is contained in
the contextual dataset.
Teacher qualifications: This construct is measured
with teaching qualifications, both general and related to
English language; it is operationalised using questions
from the teacher questionnaire.
Teacher perception of preparedness to teach: This
construct is measured with a number of questions on
the perception of teachers of how well they are prepared
to teach various aspects related to English; the questions
here come from the teacher questionnaire.
Teaching practices: This construct is measured with
variables tapping into the variety of resources teachers
use in classroom, the frequency and sort of feedback
they provide in classroom and the language they
speak during class and during individual meetings
with students; this construct is operationalised with
questions from the teacher questionnaire.
Data analysis
Test data analysis
All items included in the Simce Ingls tests in 2012 and
2014 had been previously calibrated using a typical CEFR
A2 level population as a sample. A post-live calibration
(conducted after the test administration based on live
response data) was conducted to take into account
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
CAMBRIDGE ENGLISH : RESEARCH NOTES
the features of Simce Ingls specific candidature, and
to improve grading accuracy. In both tests, candidate
results for the individual components of Reading and
Listening were reported in standardised scores. The use
of standardised scores enables direct comparison of the
results from different versions of each component.
Questionnaire data analysis
Three analytical techniques were employed in this article
to answer the research questions. Firstly, descriptive
statistics were used to compare language proficiency of
learners across school types, socioeconomic status of
schools, urban and rural locations and across regions.
In addition, descriptive statistics are used to compare
performance of Chilean learners in the Simce Ingls in
2012 and 2014. Secondly, multilevel modelling was
conducted to understand the role that various factors play
in affecting the chances of Chilean learners obtaining an
A2 or B1 certificate in the Simce Ingls test conducted in
2014. Multilevel (or hierarchical) modelling is a statistical
technique which is appropriate when the structure of the
data is hierarchical or nested (e.g. students nested within
schools, schools nested in geographic regions). This study
deals with information that pertains to various analytical
levels. Factors such as learner background characteristics,
exposure to English language, income of the households
in which the learners live and educational level of parents
are individual-level factors pertaining to learners. There
are also factors that are specific to schools, such as
the type of school or the socioeconomic status of the
school. Moreover, there are factors pertaining to schools
geographic locations. Those geographic locations analysed
in this article are urban/rural areas and regions. As a result,
we have information on three levels of data individual,
school and geographic level and the data will be analysed
accordingly using multilevel modelling techniques. The
dependent variable in these models is binary learners
that obtained an A2 or B1 certificate in Simce Ingls were
coded with 1, while learners that achieved A1 level or below
were coded with 0. Therefore, multilevel modelling with a
logit link function was employed to account for the binary
dependent variable and the multilevel data structure.
The modelling was conducted in the R statistical package
using glmer. The estimation procedure in glmer optimises a
function of the log likelihood using penalised iteratively re-
weighted least squares, and the log likelihood is evaluated
using the Laplacian approximation (Steele 2009).
Regression analysis was conducted to understand how
factors such as teacher qualification, teacher perception of
preparedness to teach, teaching practices such as use of
complementary material at school, the kind and frequency
of feedback provided in class or use of English in class are
related to performance of Chilean learners on the test.
In this project we conducted statistical analysis on big
data the number of respondents is close to the total
population of 3rd grade secondary school students in
Chile. Learners in Chile can be seen as a sample of a wider
population of learners. This allows us to draw conclusions
CAMBRIDGE ENGLISH : RESEARCH NOTES
on certain factors that may be generalisable beyond the
national context of Chile such as to other Latin American
countries.
Learner profile
In Chile, there were 225,685 3rd grade secondary school
students enrolled in 2014. In both 2012 and 2014 learners
whose native language was English, learners with a
disability or those not participating in classes based on
medical reasons were exempted from the Simce Ingls
test. As a result, in 2014 a total of 154,097 students took
thetest.
The information on candidate score and CEFR levels
was linked with data from student, teacher and parent
questionnaires as well as contextual data about the school
(such as region, urban/rural area). The linking of this
information was possible because there were common
variables included in all datasets such as student and
school identification number. The linking was undertaken
by the Agencia.
After linking student score, questionnaire and contextual
data, a dataset with N=151,515 candidates for whom the
score and CEFR level was reported remained. This amounts
to 98% of the candidates who sat the test. Approximately
50% of learners in this dataset are male and another
50% are female. The descriptive statistics provided in
this article are based on the number of candidates with a
reported CEFR score as opposed to those who answered
individualquestions.
The majority of learners in our dataset come from voucher-
subsidised private schools (65%); 26% of learners attend
public (municipal) schools and 9% of learners attend
private schools.
The majority of learners (32%) in our dataset come from
schools with a medium-low socioeconomic profile, 25%
come from schools with a medium socioeconomic profile
and 17% from a school with a low socioeconomic profile
(see Figure 2). A much smaller percentage of learners
come from schools with a higher socioeconomic profile
16% of learners come from schools with a medium-
high profile, while 10% attend schools with a high
socioeconomicprofile.
When considering the distribution of respondents by
location, 97% attend schools in urban locations, while 3%
of respondents attend schools in rural areas. Although only
3% of respondents come from a rural area, the number of
respondents in this category is substantial and amounts to
4,709 respondents.
Main findings
The main findings of the analysis are presented with a
description of the overall performance of the 2014 cohort,
which is followed by a comparison between the 2012 and
2014 cohort. We then turn to the factors which affect
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
ISSUE 65 / FEBRUARY 2017|55
Figure 2: Distribution of respondents by school socioeconomic
group (2014)
10%
17%
16% Low
Medium-low
Medium
Medium-high
32%
High
25%
English language attainment on the 2014 Simce Ingls test,
focusing first on learner-related factors, then school-level
and geographical variables. It is important, however, to
address some of the limitations.
Firstly, the analyses in this study have been conducted
for candidates for whom we have information on the
total score and the corresponding CEFR level from the
Simce Ingls test which accounted for 82% of candidates
in 2012 and 98% in 2014. Given that information on a
number of candidates is missing, analyses were conducted
for candidates for whom scores and grades in the Simce
Ingls test are available, treating information about other
candidates as missing at random.
Secondly, ideally we would like to have included the school-
level variables related to teacher qualifications, teacher
perception of preparedness to teach and teaching practices
in class in the multilevel models with a binary dependent
variable. This would allow us to tease out the effects of
those variables controlling for individual characteristics
of learners (such as background and exposure to English
language instruction) and higher level variables such as
location of schools in an urban/rural area or region. Instead,
we decided to split the analysis into:
1. Multilevel models with a binary dependent variable
(achievement of A2 or B1 level in Simce Ingls vs
achievement of pre-A1 or A1 level) including individual-
level factors, two school-level variables (school type and
socioeconomic profile of the school) and geographic
factors (location and region).
2. Regression analyses expressing the dependent variable
as the proportion of learners in a school that obtained
A2 or B1 in Simce Ingls and expressing the explanatory
variables as a proportion of respondents in a school who
gave a certain response.
The reason for this approach is two-fold. Firstly we
wanted to avoid a kitchen-sink model, where all possible
explanatory variables are introduced, thus making it difficult
to decipher which variables are of importance in explaining
learner performance in the test. We chose a parsimonious
56|ISSUE 65 / FEBRUARY 2017
model with two school-level variables and two geographic
factors which we considered most important to control for.
As a result, our modelling approach on the multilevel data
starts with investigating the influence of individual-level,
then school-level variables are added to the analysis, and
finally geographic factors are included in the final model.
The second reason for our approach is that we would have
reached the limits of computer memory running a three-
level multilevel model with a binary dependant variable
with approximately 150,000 observations with around two
dozen explanatory variables, many of them categorical.
Although there are both theoretical and practical reasons
for splitting the analysis the way we did, we do recognise
that in order to have a full insight into the role of school-
level variables related to teacher qualifications, teachers
perception of preparedness to teach and teaching practices,
it would be advisable to control for characteristics of
learners and geographical factors.
Overall performance
National profile in 2014
The results of the 2014 Simce Ingls test show that a
considerable number of learners in 3rd grade of secondary
school are below A1 level (54%), while 22% of learners
demonstrated the A1 level. A much smaller number of
learners achieved a higher level of English 12% of learners
are at the A2 level and 12% of learners are at (or above) the
B1 level.2 Figure 3 shows the distribution of CEFR levels for
the 2014 cohort.
Figure 3: Distribution of CEFR levels (2014)
12%
12%
54%
22%
Change in national profile 201214
We now compare performance on the Simce Ingls test
of the 2012 and 2014 cohorts.3 Here, calculations of
2
Note that as the test measured up to B1, some candidates may be above this level. The descriptive statistics presented here are based on N=151,515 candidates for whom we have
information on the CEFR level obtained in the Simce Ingls test.
3
For comparison of CEFR level distribution between 2012 and 2014, percentages were calculated based on the total number of respondents who obtained a score and a CEFR level
in the Simce Ingls test (candidates with missing information were excluded). Similarly, for the comparison of CEFR level distribution across categories, percentages are based on the
total number of candidates in each category who obtained a CEFR level.
4
The cumulative percentage for A1 indicates the percentage of candidates who achieved A1 plus the percentage of candidates who achieved above A1. Similarly, the cumulative
percentage for A2 indicates the percentage of candidates who achieved A2 and above A2. For pre-A1, the absolute percentage is reported percentage of candidates who did not
achieve A1 or above.
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
percentage difference for A1, A2 and B1 are based on
reverse cumulative percentages for each year (2012
and 2014), the percentage of candidates at a CEFR level
is expressed as the percentage of candidates at this level
and above. The difference for below A1 is calculated as the
absolute percentage.4 Percentages are calculated this way
to enable comparison with 2012 data. Figure 4 presents
the comparison of test results between 2012 and 2014.
A positive difference means that a higher percentage of
candidates from the 2014 cohort obtained a CEFR level,
while a negative difference means that a lower percentage
of candidates from the 2014 cohort obtained a CEFR level.
Figure 4: Difference in distribution of CEFR levels between 2012
and 2014
B1 4%
A2 6%
A1 2%
pre-A1 2%
0%
%
%
%
4%
6%
8%
0%
2%
4
8
10
1
As can be seen in Figure 4, a slightly higher percentage of
learners in the 2014 cohort achieved A1 (2%), A2 (6%)
and B1 (4%) when compared to the 2012 cohort, while
fewer learners scored below A1 (the difference amounting
to 2%). This small, positive trend may be a by-product of
recent changes to English language contact hours and the
curriculum. As mentioned previously, there has been an
increase in the number of hours given to English language
instruction in Chile and in 2013 a new curriculum was
introduced which emphasised a communicative language
teaching methodology. These changes, along with teachers
increased familiarity with the Simce Ingls test, which has
pre-A1 been administered twice before (in 2010 and 2012 with its
A1
current specifications), may have had a positive impact on
learner performance. This suggests that continued use of
A2
the Simce Ingls test would be useful in order to determine if
B1
this trend is maintained and improved upon over time.
Factors affecting English language
attainment
This section explores the factors which affect learner
outcomes as measured by the Simce Ingls test.
Learner-related factors such as learner background and
socioeconomic status (parents education, household
WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
CAMBRIDGE ENGLISH : RESEARCH NOTES
income and parents ability to speak English), learner
exposure to English at school, motivation to learn English
and use of English in learners free time are investigated
first. Then, the effect of school-related variables such as
school type, socioeconomic profile of the school and school
location (urban/rural and geographic region) are explored.
Finally, factors associated with teachers within schools such
as their qualifications, preparedness to teach and classroom
practices are investigated. In exploring the influence of
various factors on English language proficiency, information
from different types of analyses is presented:
1. Descriptive statistics of CEFR grade data from the Simce
Ingls test.
2. Descriptive statistics of questionnaire responses.
3. Multilevel and regression analyses findings.
Results from the multilevel analyses are expressed as odds
ratios and are interpreted accordingly. An odds ratio is a
way to quantify how strongly the presence of a certain
factor or trait is associated with a certain result (in our
context, scoring A2 or B1 in the Simce Ingls test). If the
odds of achieving A2 or B1 on the Simce Ingls test for
learners with a certain trait (e.g. those who receive 5 hours
of English instruction per week) are higher than the odds
of learners who do not have this trait (e.g. those who have
2 hours of English instruction per week), then this means
that the former are more likely to achieve A2 or B1 on the
test than the latter. In the context of this article, we will
use odds and chances interchangeably. We will also
express the increase of odds for one group versus another
in percentages.
It is important to point out that significant findings reported
below should not be interpreted as evidence of causation
but rather that the two things are positively correlated.
For example, if using the internet in English is positively
associated with language learning outcomes, it is not
appropriate to conclude that using the internet more will
lead to improved outcomes because it may be that only
those learners who have reached a higher language level
are able or willing to use the internet in English.
Learner-related factors
Background and socioeconomic factors
Language learning is a complex phenomenon which is
influenced by many factors. Learner motivation and parental
attitude to language learning are two factors which can
influence a learners willingness to engage with language
learning and persist with learning when it becomes
challenging (Bartram 2006, Czisr and Drnyei 2005).
Additionally, the socioeconomic status of the household
and education level attained by parents has also been
shown to be associated with non-language-related learning
outcomes on international tests such as PISA (Organisation
for Economic Co-operation and Development 2011).
Therefore, in this section, we investigate variables related
to learners family background to determine the extent to
which they predict language learning outcomes. As equity
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
ISSUE 65 / FEBRUARY 2017|57
within the education system is important in this context, we
also look into performance on the test by gender.
Among the background and socioeconomic factors
investigated here, two stand out as important predictors
ofachievement.
Firstly, household income emerges as a very strong predictor
of English language achievement. The higher the household
income of a learners family, the higherthe chances of the
learner obtaining an A2/B1 level in the Simce Ingls test.
Statistical analysis shows that the odds of achieving an
A2 or B1 certificate for learners from households with a
medium-low income are approximately 6% higher than
the odds of candidates who come from a low-income
household. The odds of success increase as the household
income increases. The odds for learners from households
with medium-high income are around 13% higher than the
odds of a low-income household, while the odds for learners
from a high-income household are approximately 50%
higher. This effect is statistically significant, controlling for
all background and socioeconomic factors and variables
related to English language instruction.
Secondly, if the father or mother of a learner can speak
English, the learner has a significantly higher chance of
attaining an A2/B1 level of English language proficiency,
but we did not see a similar effect for the variables linked
to whether the father or mother can read and write English.
A potential explanation for this pattern could be that the
ability of parents to communicate in English may lead
to increased exposure to the language outside school
through travel or interactions with English-speaking people.
Additionally, parents are important role models and their
ability to speak English may influence learners willingness
to use English when given the opportunity, which has been
found to be linked to increased proficiency (Jones 2013).
For the learners who stated their fathers speak English
quite a lot, the chances of achieving a A2/B1 level in the
Simce Ingls test are approximately 12% higher than the
chances of a learner whose father does not speak English
at all. Similarly, if the learners father speaks English a lot,
the chances of this learner obtaining an A2/B1 level of
English are 30% higher when compared to learners whose
fathers cannot speak English at all. Similarly, when the
learners mother speaks English a lot, the learner has an
approximately 20% higher chance of obtaining an A2/B1
level in Simce Ingls than a learner whose mother does not
speak English at all.
Parents educational level is also related to their childs
English language attainment in our analysis. The analysis
shows that there is a positive relationship between
a mothers and fathers educational level and English
language attainment of their child, but the effect is not
statistically significant. The higher the educational level
of the father or the mother, the higher the chances of the
learner obtaining an A2/B1 level in the Simce Ingls test.
For example, learners whose fathers attended institutions
of higher education have approximately an 18% higher
chance of obtaining A2/B1 in Simce Ingls than learners
58|ISSUE 65 / FEBRUARY 2017
whose fathers did not. Similarly, for learners whose mothers
attended higher education the odds of achieving an A2/B1
level of English language are approximately 30% higher.
For gender, the analysis shows that male learners seem to
have slightly higher chances of obtaining an A2/B1 level of
English proficiency and this positive relationship persists
after controlling for individual and school-level factors.
When we control for all individual-level characteristics
of learners and school-level variables, the effect of
gender emerges as statistically significant taking all
explanatory variables into account, the chances of male
learners obtaining an A2/B1 level in English language
are approximately 12% higher than the chances of
femalelearners.
The association between the socioeconomic status of
learners and their parents educational level with student
English language learning outcomes is not unexpected
as these factors have been identified as contributing to
differences in results for other skills in Chile (Ministry of
Education 2015, Organisation for Economic Co-operation
and Development 2010). Much of the explanation for
this situation relates to the structure of the education
system which has resulted in higher variation in learner
performance between schools than within schools
(Organisation for Economic Co-operation and Development
2010). Therefore, the school a learner attends in Chile
makes a big difference in their learning outcomes. The
impact of school choice on learning outcomes will be
investigated further in the section on school-related
factors; however, continued monitoring of the influence of
socioeconomic status on achievement would be a useful
activity in order to determine whether the gap between the
economic strata in society is shrinking over time.
The difference in learning outcomes between male and
female learners is an area in need of further exploration.
Cambridge English has not found a consistent trend
concerning language attainment of males and females
throughout the world (Walczak and Geranpayeh 2015).
Variation in English language attainment for each gender by
country is the norm; however, as a strategic objective of the
Ministry is to reduce learning gaps, this is an area that may
require further monitoring and investigation to ensure that
there are no systemic issues which may be influencing the
chances of female learners to attain high levels of English.
Exposure to English in school
A characteristic of the Chilean educational context is that
the grade in which English is first introduced, the number of
hours of instruction on offer and the amount of instruction
that takes place in English (rather than in Spanish) can vary
by school and/or school type. As these variables, unlike
socioeconomic and parental background variables, are most
amenable to change it is important to investigate whether
they affect attainment in the Simce Ingls test. Therefore, we
will look at each factor in turn below.
5
The descriptive statistics in this chapter show percentages which are calculated based on the total number of learners in the 2014 dataset. Please note B=primary school,
M=secondary school.
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
Onset and frequency of English language learning
Figure 5 shows the percentage of learners who started
learning English in different grades at school.5 Although
English is officially introduced as a subject in the 5th year
of primary school (5B) in public schools, 41% reported
learning English before this grade suggesting there is
variation both within and across school types. It should,
however, be kept in mind that approximately a third of
learners did not respond to this question.
Figure 5: In what grade did you receive your first English lessons?
Nursery school or kindergarten 11%
1B 17%
2B, 3B or 4B 13%
5B 11%
6B, 7B or 8B 6%
1M, 2M or 3M 1%
Does not remember 8%
Not responded 33%
The results show that the onset of English instruction is a
significant predictor of achievement of English language
proficiency. Learners who started learning English in
nursery school or kindergarten have a considerable
advantage over learners who started English language
instruction later. The odds of achieving a Simce Ingls
certificate for learners that started English instruction in 1B
are approximately 15% lower than for learners that started
English instruction in nursery school or kindergarten. The
same effects can be observed for learners who started
English instruction in the 2B, 3B and 4B and 5B grades.
Learners who started learning English in 6B, 7B or 8B
have a 17% lower chance of achieving A2/B1 in the Simce
Ingls test, while the odds of learners who started English
instruction in 1M, 2M or 3M are 13% lower when
compared to learners who started English instruction
muchsooner.
Turning now to the number of English language contact
hours, Figure 6 shows the breakdown of how many English
classes learners in Chile have in school. Almost half of the
learners (45%) have between 2 and 3 hours of instruction
per week, while 20% have more than 4 hours.
Figure 6: How many English classes do you have a week?*
Does not have English lessons 0%
1 hr 1%
2 hrs 21%
3 hrs 24%
Between 4 and 6 hrs 19%
7 hrs or more 1%
Not responded 33%
*Percentages may not sum due to rounding
WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
CAMBRIDGE ENGLISH : RESEARCH NOTES
The number of English classes per week that students have
in school is also positively correlated with performance
in English. The higher the number of English classes per
week,the higher the chances are that learners obtain an
A2/B1 level in English. Learners that have 7 or more hours
of English classes per week have an approximately 15%
higher chance of obtaining a Simce Ingls certificate than
learners with just 1 hour of classes per week. Learners
who have between 4 and 6 hours of English classes per
week have an approximately 7% higher chance to obtain
an A2/B1 level in English language. The analysis shows
a positive correlation, but the effect does not emerge as
statistically significant given all the other individual-level
explanatoryvariables.
One way of improving performance in the public sector
would be to bring the introduction of English and the
number of contact hours per week into line with that of
voucher-subsidised and private schools, in the instances
where there is a difference. This may help in reducing the
achievement gap seen between the sectors and ensure that
all students have an equal opportunity to learn English.
Use of English in the classroom
In the European Survey on Language Competences
(European Commission 2012), exposure to a foreign
language within the learning environment and greater use
of it in the classroom was found to be positively associated
with higher levels of proficiency (Jones 2013). As such, the
English language use of learners and teachers in Chilean
schools was investigated to determine whether this
relationship holds in this context.
Questionnaire results indicate that a minority of students
in Chile speak English to their teachers during English
lessons. As seen in Figure 7, only 2% of students in Chile
speak English to their teachers throughout the class. The
vast majority of students speak Spanish or mostly Spanish
during their English classes (27% and 28% respectively)
which in itself should be a cause for concern if the aim of
English language learning is to develop communicative
competence. Although the majority of teachers in Chile
speak English during English classes according to learners
(see Figure 8), a considerable number of teachers (around
one in five) speak mostly in Spanish.
Figure 7: What language do you speak to your teacher during
English lessons?
Always in English 2%
Mostly in English 10%
Mostly in Spanish 28%
Always in Spanish 27%
Not responded
Not surprisingly, the analysis indicates that the use of
English by teachers and learners is an important factor
when considering learner achievement. Learners who speak
to teachers in English during their classes achieve better
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
ISSUE 65 / FEBRUARY 2017|59
Figure 8: What language does your teacher speak during English
lessons?
Always in English 7%
Mostly in English 42%
Mostly in Spanish 16%
Always in Spanish 2%
Not responded 33%
results than learners who speak their native language or
alternate between their native language and English. The
odds of obtaining the Simce Ingls certificate for learners
who always speak Spanish in class are approximately
18% lower than for learners who always speak English in
class. For learners who speak mostly Spanish in class the
chances are approximately 23% lower. Interestingly, the
chances of attaining an A2/B1 level of English are lower
even for students who speak a bit of English in class
when compared to those learners who report that they
always speak English in class. These effects are statistically
significant. Thus, there is strong evidence that speaking
English throughout the class has an important impact on
learning. When looking at the role of English language use
by the teacher there seems to be a positive relationship
between the language a teacher speaks in class and
learner performance in English, but our analysis does not
show a statistical significance for this variable. A potential
explanation for this might be that the variable language
which learners use during English classes might have a
predominant effect which overshadows the effect of the
other variable. It may be that classes in which learners
spend the majority of their time speaking are more learner
centred and the language used by the teacher is of less
significance because learners are the ones speaking
during class time. Therefore, developing teachers skills
in designing tasks which require learners to engage in
purposeful communication in English may help improve
learning outcomes.
Motivation to learn English
Learners attitudes towards language learning and the
extent to which they perceive the language to be useful can
influence their behaviour, both in terms of the amount of
effort exerted on learning and their willingness to continue
learning when faced with challenges (Csizr and Drnyei
2005, Drnyei 2003). Similarly, the introduction of tests
can also influence learner motivation as well as approaches
to teaching and learning, depending on how stakeholders
perceive the assessment and its use within a particular
context (Saville 2012). Learner motivation to learn English
and attitudes towards the Simce Ingls test, therefore, are
investigated below to determine whether these factors are
related to learner outcomes.
33%
As seen in Figure 9, the vast majority of learners in Chile are
motivated to study English and value receiving the Simce
Ingls certificate. For many Chilean learners, achieving a
good grade in the Simce Ingls test is also very important.
The opinions were split, however, over whether the Simce
60|ISSUE 65 / FEBRUARY 2017
Ingls test motivated learners to study English more 29%
of learners agreed that the test motivated them to study
English more, while 35% disagreed.
Figure 9: Motivation to learn English
Obtaining the Simce Ingls 18%
certificate is important for me
Simce Ingls motivated
me to study English more
Having a good grade in Simce 14%
Ingls is important for me
Learning English is very important for me 5%
Strongly agree or agree Strongly disagree or disagree
The statistical analysis shows that the motivational aspects
do not explain why some learners achieved an A2/B1level
while the others did not. The estimates for these measures
of motivation are inconsistent and they are not statistically
significant. One possible reason for this is that in our
analysis we controlled for a number of background and
socioeconomic variables, exposure to English in school and
outside school. The economic and educational factors
household income and parents ability to speak English
and the onset of English language instruction are the
prevailing predictors of learner performance and they may
overshadow the motivational aspects.
Use of English outside school
Just as the use of English in the school environment has
been found to be positively associated with language
learning outcomes, so too has learners exposure to and
willingness to use English in their daily lives (Jones 2013).
This section reports on the extent to which learners
use English in their free time and whether this factor is
associated with attainment.
The results show that a third of Chilean learners watch TV
or movies in English on a daily or weekly basis (see Figure
10), whereas only one tenth of them responded that they
read books, magazines or comics in English daily or weekly.
Even fewer (8%) speak English with other people (either
over the internet or in person) on a daily or weekly basis. It
is important to note that this finding may be more strongly
associated with the extent to which they have opportunities
to speak English with other people rather than their
willingness to do so. Interestingly, 18% of learners do report
using online chat in English either daily or weekly and 24% of
them browse English websites. The results shown in Figure
10 suggest that Chilean learners are willing to engage with
English in their free time when given the opportunity, which
could be harnessed to improve language learning at school.
Statistical results give some indication that use of English
in free time is related to the attainment of English language
proficiency. Learners who watch TV or movies, read
books, magazines or comics in English or who browse
English language webpages several times a week or every
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
CAMBRIDGE ENGLISH : RESEARCH NOTES
day have slightly higher chances of achieving the A2/
B1 Simce Ingls certificate than learners who never do it.
However, the effects for these variables are not statistically
significant, most probably because our statistical model
contains a number of control variables, among them the
highly statistically significant effects of learner background
48%
variables and starting age of English language instruction,
34%
which might be overshadowing the effect of other variables.
29%
35%
36% Figure 10: Use of English language in free time
51%
5% 8% 20% 25% 7% 34%
35% In my free time, I watch TV or movies in
English (with or without subtitles in English)
61% 17% 11% 14% 15% 9% 34%
In my free time, I browse English
34% webpages (incl. playing online games)
34% 11% 7% 16% 2% 34%
In my free time, I use online chat in English
Not responded
30% 15% 12% 7% 2% 34%
In my free time, I read books,
magazines or comics in English
33% 14% 10% 6% 2% 30%
In my free time, I speak English with
other people (over the internet or in person)
A few times a year
Never A few times per month
Several times a week Every day Not Responded
School-related factors
Having considered learner-related variables, we now turn to
the effect of school-level factors on attainment. In this section,
performance by school type, the socioeconomic profile of the
school and the location of the school are investigated.
There is evidence that the type of school a learner attends
is a major predictor of learner attainment in non-language-
related subjects (Organisation for Economic Co-operation
and Development 2010). As mentioned previously, this has
been attributed to differences in quality between the school
sectors and the extent to which schools are academically
and socially segregated (Organisation for Economic Co-
operation and Development 2013a). Consequently, to
understand the effect of school choice on language learning,
this section investigates these factors. The results in this
section can shed light on the degree of equality within the
education system in terms of English language provision.
School type
In Chile, learners in private schools performed very well in
the Simce Ingls test, while performance of learners in public
schools and voucher-subsidised private schools was weaker.
As Table 1 shows, the vast majority of learners in private
schools (63%) achieved a B1 CEFR level. In contrast,
the majority of learners in public schools and voucher-
subsidised private schools are at the lower end of the
CEFR scale in terms of English language proficiency. In
public schools, as many as 71% of learners are at a pre-A1
level while 54% of learners in voucher-subsidised private
schools are at a pre-A1 level and 26% are at an A1 level.
The majority of learners at private schools, however, are at
a B1 level.
When we compare performance of two cohorts of Chilean
students (2012 and 2014) across different school types,
the following picture emerges (see Figure 11). For all three
CAMBRIDGE ENGLISH : RESEARCH NOTES
Table 1 Distribution of CEFR levels by school type profile (2014)
Level Public Voucher-subsidised private Private
pre-A1 71% 54%
A1 17% 26%
A2 7% 13% 20%
B1 4% 8% 63%
school types, the percentage of candidates who obtained
A2 or B1 was higher in 2014 when compared to 2012.
For B1, private schools showed the largest change
(7%higher), followed by voucher-subsidised private schools
(4% higher) and public schools had the smallest increase
(2% higher). However, when looking at changes at the A2
level, voucher-subsidised private schools had the largest
increase (6% higher), followed by public schools (5%
higher) and private schools (2% higher). For Levels A1 and
below A1 there is either no change or the change is minimal.
The descriptive statistics show a marked difference in
learner attainment depending on the type of school. In
order to explore the effect of school type on chances to
obtain an A2/B1 level of English language proficiency, we
included this variable in a hierarchical model, alongside
individual-level information on Chilean candidates. If the
school type variables come out as significant then it would
mean that it is an important predictor of achievement in
English language on top of the individual factors.
Our results show a considerable difference in performance
of learners across different types of schools, controlling
for all individual factors related to student background
and motivation. Learners who attend voucher-subsidised
private schools are approximately three times more likely
to achieve an A2/B1 level in Simce Ingls than learners from
public schools. Interestingly, learners from private schools
are much more likely to achieve a higher level of English
around 145 times more likely. This shows that private
schools achieve much better results than other school
types in Chile.
Interestingly, when we introduced the school type variable
into the model, the statistical significance of important
individual level predictors disappeared this is the case for
variables such as household income, starting age of English
Figure 11: Difference in distribution of CEFR levels 20122014 by school type
Public
B1 2% B1
A2 5% A2
A1 0% A1
pre-A1 0% pre-A1
10% 0% 10% 10%
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
ISSUE 65 / FEBRUARY 2017|61
instruction in school, and whether a mother or a father
of the learner can speak English. The positive correlation
of these variables and performance of learners in English
4% language stays, which means that they are positively
12% related to learner performance in English language.
However, the effect of school type becomes predominant
once this variable is introduced in the statistical analysis.
Socioeconomic profile of the school
As we have seen, the household income of learners
is a strong predictor of attainment; therefore, we also
investigated the impact of the socioeconomic profile
of the school on language learning. Schools with a low
socioeconomic profile have been found to perform
worse on international tests than those with a higher
socioeconomic profile (Organisation for Economic Co-
operation and Development 2015). This finding is attributed
to differences in a variety of factors such as resources
available, teacher quality and parental support for learning.
That is, schools classified as lower on the socioeconomic
range face more challenges in these areas which can
negatively impact learning. This variable consists of a
number of indicators, namely: the educational and income
level of parents whose children attend the school as well
as the degree of vulnerability of the school i.e. whether
children in these schools come from impoverished families
needing assistance from governmental or nongovernmental
organisations.
Performance of learners in the Simce Ingls test varies
considerably according to the socioeconomic profile of the
school the learners attend. The best performers in the Simce
Ingls test come from schools with a high socioeconomic
profile: 62% of learners from schools with a high
socioeconomic profile achieved a B1 level in Simce Ingls,
while only 4% are at the pre-A1 level. Learners from schools
with a medium-high socioeconomic profile performed well
in Simce Ingls but visibly worse than learners from schools
with a high socioeconomic profile (see Figure 12).
The lower the socioeconomic status of the school, the
worse the learners performed in Simce Ingls. In schools
with a medium socioeconomic profile, 45% of learners are
below A1 level and 32% are at A1 level, while in schools
with medium-low socioeconomic status, 74% of learners
Voucher-subsidised Private
private schools
4% B1 7%
6% A2 2%
0% A1 1%
0% pre-A1 1%
0% 10% 10% 0% 10%
62|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Figure 13: Difference in distribution of CEFR levels 20122014 by

scored pre-A1 and 19% achieved A1 level. Learners in socioeconomic profile of school
schools with a low socioeconomic profile performed the
worst in the Simce Ingls test. The vast majority of learners Low Medium-low
in these schools (89%) are below A1 in terms of English
B1 0% B1 1%
language proficiency. The discrepancies in performance of
learners across schools with different socioeconomic profile A2 1% A2 3%
are visualised in Figure 12.
A1 3% A1 1%
Overall the picture is consistent with findings from previous
pre-A1 3% pre-A1 1%
Figure 12: Distribution of CEFR levels by school socioeconomic
group profiles (2014) 10% 5% 0% 5% 10% 10% 5% 0% 5% 10%
89%

Medium Medium-high
74%

62%
B1 4% 7%
B1
45%

A2 7% 8%
A2
32%

32%
26%
22%

21%
20%
19%

A1 1% 1%
15%

13%
A1
10%

8%

4%
5%
2%
1%
0%

pre-A1 1% pre-A1 1%
Low Medium-low Medium Medium-high High
10% 5% 0% 5% 10% 10% 5% 0% 5% 10%
pre-A1 A1 A2 B1

High
research (Organisation for Economic Co-operation and
Development 2010, 2013a) fewer learners attending B1 6%

schools with a low or medium-low socioeconomic profile A2 2%

are achieving A2 or B1 level. These differences highlight
A1 1%
variations in the quality of education by school type.
pre-A1 1%
Having looked at overall performance in 2014, we now
investigate changes in performance between 2012 and 10% 5% 0% 5% 10%

2014 across the socioeconomic status of the school.

The comparison shows that the percentage of learners
who achieved B1 or A2 across the medium-low, medium,
medium-high and high socioeconomic profiles of schools
has risen slightly, but there is no uniform pattern for
other CEFR levels. An area of concern relates to the small
increase in the percentage of learners achieving pre-A1
and an equally small decrease in the percentage achieving
A1 for schools which fall into the low socioeconomic
profile. Although it is important to note that these
changes are very small, it is important to monitor future
performance to ensure that this is not a trend which would
result in a widening gap between schools with different
socioeconomic profiles. The changes are presented in
Figure 13.
Subsequently, we investigated what impact the
socioeconomic status of the school has on attainment of
English language proficiency. Statistical analysis shows that
this variable is a very important predictor of attainment.
The higher the socioeconomic status of the school, the
better learners performed in the Simce Ingls test, thus the
better their attainment of English language proficiency is.
Learners from schools with medium-high socioeconomic
status are over 90 times more likely to obtain A2/B1
level in English, while learners from schools with a high
socioeconomic status are over 300 times more likely to
achieve a higher level of English language proficiency.
Learners from schools with a medium-low socioeconomic
status are approximately five times more likely to achieve
an A2/B1 level in English than learners from schools with a
low socioeconomic status.
The effect of the socioeconomic status of a school is
huge and points to the fact that the status of the school
matters a lot in determining student performance. Once
this variable is entered in statistical models, the statistical
significance of individual-level variables (household
income, whether father or mother of the learner speak
English, starting age of English language instruction)
and of the school type variable disappears. The effect of
school type loses statistical significance as school type is
strongly positively correlated with socioeconomic status
of the school private schools tend to have a much higher
socioeconomic status. This means that the effect of the
school socioeconomic status overshadows the effects
of other variables and is crucial in explaining learner
outcomes. This finding is not surprising considering the
challenges faced by Chiles education system which
tends to be highly stratified by socioeconomic status
(Organisation for Economic Co-operation and Development
2011).
School location: Urban or rural
The next variable that we investigate is school location in
terms of being located in either an urban or rural area. In
many countries, schools in rural areas tend not to have
the same access to resources or the ability to attract and
retain good teachers. As a result, learning outcomes can
be negatively affected by this situation (Monk 2007).

UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|63

Chile faces shortages in qualified teachers in rural areas teaching and learning practices are much more amenable to
even though it has introduced several initiatives over the change through policy interventions than systemic factors
last two decades to support rural schools (Organisation are (Hattie 2009), the focus here is on those areas that are
for Economic Co-operation and Development 2013b, likely to have the greatest impact on learning outcomes in
2015). Therefore, an area of interest is the extent to which a relatively short period of time. Finally, the areas covered
there is an urbanrural gap in English language learning in this article are inevitably linked in a common ecological
outcomes as these findings can be used to evaluate the system, where changes to one affect the others; therefore,
success of these projects and shed light on equity issues the following recommendations should be viewed holistically
in termsofpupil access to quality education regardless of rather than as discrete recommendations.
location.
Increasing learner exposure to English
When we look at performance of Chilean learners in the
There was a positive correlation found between onset of
2014 Simce Ingls test, we see that there is a difference in
English instruction and hours of instruction; therefore, the
performance between learners coming from urban and rural
Ministry could consider the feasibility of introducing English
areas. As Table 2 shows, in urban areas learners received
at an earlier grade in primary school and/or increasing the
slightly better results in Simce Ingls than learners in rural
number of hours of instruction. Differences in attainment
areas. In urban areas, 25% of learners are either at A2 or
began to be noticed at 4 hours a week of instruction.
B1 level (12% and 13% respectively) compared to 5% of
learners in rural areas (with 3% at A2 and 2% at B1 level).
Improve teachers language proficiency and
pedagogy skills
Table 2 Distribution of CEFR levels by location (2014)
As teacher quality has been identified as an important
Urban Rural factor for learning outcomes generally (Schleicher 2011)
pre-A1 53% 82% and our findings indicated that teachers English proficiency,
qualifications and confidence in teaching are associated
A1 22% 12%
with learning outcomes, it is important to identify ways of
A2 12% 3%
improving in-service teacher quality. An integrated solution
B1 13% 2%
that addresses both proficiency and pedagogy is essential.
Courses that focus specifically on improving the English of
Comparison of performance of the 2012 and 2014 cohorts teachers may be particularly useful to help large numbers
across urban/rural school location shows that for schools of teachers to quickly improve their language levels whilst
located in urban areas the percentage of learners with allowing the Ministry to engage in longer term, more
higher levels of English language proficiency increased systemic changes.
between 2012 and 2014. In turn, in the rural areas the
Similarly, flexible training programmes specifically designed
percentage of learners who are below A1 is higher in the
for primary and secondary language teachers to improve
2014 cohort when compared to the 2012 cohort. Figure 14
their teaching methodology are recommended. Again,
below shows these changes.
an example of a Cambridge English course offering is the
Figure 14: Difference in distribution of CEFR levels 20122014 by Certificate in English Language Teaching Primary (CELT-P) and
location Certificate in English Language Teaching Secondary (CELT-S);
such courses can provide targeted help in improving teacher
Urban Rural
confidence and effectiveness. Again, these courses can be
B1 4% B1 0% offered fully online or as blended learning.6
A2 7% A2 0.4%
Encourage teacher reflection
A1 2% A1 2.9% In addition to the points above, we recommended an
expansion of the teacher network already in place in Chile
pre-A1 2% pre-A1 2.9%
to allow for wider sharing of expertise, peer mentoring
10% 5% 0% 5% 10% 10% 5% 0% 5% 10% and critical reflection, all of which would be beneficial for
teachers. This could include access to self-access materials

Recommendations such as the webinars, teaching tips, resources and forums

A premise recognised in educational reform is that a key
characteristic of the educational process is that student
learning is influenced by many small factors rather than a
few large ones (Chapman, Weidman, Cohen and Mercer
2005:526). Additionally, as research has indicated that
available on Cambridge English Teacher which support
instructional improvement.7 A tool such as the Cambridge
English Teaching Framework, a profiling grid describing
teacher competencies, could also be used to promote
teacher self-reflection which could lead to better awareness
of areas that are in need of further training.8

6
www.cambridgeenglish.org/teaching-english/teaching-qualifications
7
www.cambridgeenglish.org/teaching-english/cambridge-english-teacher
8
www.cambridgeenglish.org/teaching-english/cambridge-english-teaching-framework

UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
64|ISSUE 65 / FEBRUARY 2017
Upskilling activities like those described above will benefit
teachers and learners by improving teacher quality and
confidence and increasing English use in the classroom.
Courses could be targeted at public and rural schools in
the first instance to try to support the teachers in these
locations in order to try to reduce the learning gap identified
in this article.
Improving quality of pre-service English language
teacher education
Identifying ways of ensuring that teachers who enter the
profession have the qualifications and training necessary
to be effective teachers is important. This can be done by
establishing realistic English language targets for teachers
and providing them with teaching qualifications that
address methodological considerations in a standardised
format. For example, the Teaching Knowledge Test (TKT) is
a modular English language teaching qualification which
provides teachers with the teaching knowledge necessary
to be an effective English teacher, and has a practical
component which can be used to ensure teachers have the
classroom skills necessary to be effective teachers.9
Incorporating external assessment of both teacher language
proficiency and pedagogic competence could provide an
element of standardisation across providers which may
lead to more consistent educational input and instructional
practices across school sectors.
Improving access to and quality of resources
As there are currently a number of teachers in the system
who have low levels of English proficiency and limited
confidence in their ability to prepare effective lessons, it is
important that teachers have access to quality resources
and are trained on effective use of these resources. A range
of appropriate resources can support teachers in providing
learners with the input they need to progress in English.
However, the provision of these resources needs to be
coupled with training on their effective use otherwise their
impact will be limited.
Monitoring equity of English language provision
through Simce Ingls
The Simce Ingls results show that school choice can
determine a learners chances of reaching the national
English language targets, which suggests that some
students are being disadvantaged. It is important that the
Ministry continues to monitor this situation and identify
opportunities to reduce this achievement gap. The Simce
Ingls test provides a systematic method of monitoring
equity and fairness within the system and the effect of
Ministry interventions. The results reported here suggest
that the test can play an important role in supporting
education policy decisions and it is hoped that the Ministry
continues this testing programme.
www.cambridgeenglish.org/teaching-english/teaching-qualifications/tkt
9
UCLES 2017 WHICH FACTORS AFFECT ENGLISH LANGUAGE ATTAINMENT? ASTUDY OF SCHOOL STUDENTS IN CHILE
CAMBRIDGE ENGLISH : RESEARCH NOTES
Conclusion
Simce Ingls is a unique, and in many respects, unparalleled
project that demonstrates the Governments desire
to improve learning outcomes in Chile. The combined
testing and collection of questionnaire data has provided
the Agencia with a wealth of data to support evidence-
based policy decisions and monitor the effects of these
decisions. The findings from the data analyses also can help
inform language education reform beyond Chile due to its
comprehensiveness and quality.
Controlling for a number of individual and school-level
factors, we investigated whether the urban/rural location
of the school affects attainment of English language
proficiency. Statistical analysis shows that learners from rural
areas are less likely to score A2/B1 on the Simce Ingls test
than learners from urban areas (the odds are 45% lower for
the former than for the latter). This difference is statistically
significant. Interestingly, when the urban/rural location of
a school is included in the statistical analysis, the effect of
the socioeconomic status of schools remains statistically
significant. This again points to the latter variable as being a
strong predictor of English language attainment.
These findings suggest that learners in rural areas are
disadvantaged when it comes to English language learning.
As equity within the education system is a strategic goal
for the Ministry, identifying additional ways to support
language learning in rural areas is important. The Ministry
already offers financial incentives for teachers in rural
areas, so it may be useful to review the current policy and
consider other complementary initiatives that may improve
the recruitment and retention of qualified teachers in rural
areas. For example, expanding the teacher network already
in place to include peer mentoring and wider sharing of
expertise, and incorporating other value-added activities
for English language teachers in rural and disadvantaged
schools such as the opportunity to receive international
qualifications may improve not only instructional quality in
these areas but also teacher job satisfaction.
References
Bartram, B (2006) An examination of perceptions of parental
influence on attitudes to language learning, Educational Research
48, 211221.
Chapman, D W, Weidman, J, Cohen, M and Mercer, M (2005) The
search for quality: A five country study of national strategies to
improve educational quality in Central Asia, International Journal of
Educational Development 25, 514530.
Council of Europe (2001) Common European Framework of
Reference for Languages: Learning, Teaching, Assessment, Cambridge:
Cambridge University Press.
Csizr, K and Drnyei, Z (2005) The internal structure of language
learning motivation and its relationship with language choice and
learning effort, The Modern Language Journal 89 (1), 1936.
CAMBRIDGE ENGLISH : RESEARCH NOTES
Drnyei, Z (2003) Attitudes, orientations and motivations in
language learning: Advances in theory, research and applications,
Language Learning 53 (1), 332.
European Commission (2012) First European Survey on Language
Competences. Final Report. European Commission, available online:
ec.europa.eu/languages/policy/strategicframework/documents/
language-survey-final-report_en.pdf
Hattie, J (2009) Visible Learning: A Synthesis of Over 800 Meta-
analyses Relating to Achievement, New York: Routledge.
Jones, N (2013) The European Survey on Language Competences
and its significance for Cambridge English Language Assessment,
Research Notes 52, 26.
Ministry of Education (2015) Entrega de Resultados de aprendizaje
2014, available online: archivos.agenciaeducacion.cl/resultados-2014/
Presentacion_Entrega_Resultados_2014_8_II_III.pdf
Ministry of the General Secretariat of the President, Ministry of
Education, and Ministry of Economy, Promotion and Tourism
(2014) National English Strategy 20142030, available online:
www.economia.gob.cl/wp-content/uploads/2014/03/140307-
Presentacion-CS-lanzamiento-ENI-v3.pdf
Monk, D H (2007) Recruiting and retaining high-quality teachers
in rural areas, Future Child 17, 155174.
Organisation for Economic Co-operation and Development
(2010) OECD Economic Surveys: Chile 2010, available online:
www.oecd-ilibrary.org/docserver/download/1010011e.
pdf?expires=1473328157&id=id&accname=oid007055&check
sum=D007B8B71FF07D0C650B8B5CA1DF7D9C
The impact of the Antioquia English in the Park
initiative on language teaching and practice
DANIEL BROOKER CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
JANE LLOYD CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
MARTIN ROBINSON CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
SERGI CASALS BRITISH COUNCIL, COLOMBIA
Introduction
The province of Antioquia, Colombia has a strategic
objective to strengthen and improve English language
teaching and learning in the province. As such the
Education Secretariat of Antioquia established the English
in the Park initiative in July 2015 which aims to create a
language-friendly environment in the Education Parks
in Antioquia and to provide opportunities for practising
communicative language skills in state schools. There
are 80 parks (or parques) in the province. Their main
purpose is to improve the quality of education in order to
UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
ISSUE 65 / FEBRUARY 2017|65
Organisation for Economic Co-operation and Development (2011)
Maintaining Momentum: OECD Perspectives on Policy Challenges in
Chile, available online: www.oecd.org/chile/47496802.pdf
Organisation for Economic Co-operation and Development
(2013a) Education Policy Outlook: Chile, available online: www.
oecd.org/edu/EDUCATION%20POLICY%20OUTLOOK%20
CHILE_EN.pdf
Organisation for Economic Co-operation and Development
(2013b) OECD Reviews of Evaluation and Assessment in Education:
Teacher Evaluation in Chile, available online: www.oecd.org/chile/
OECD Review Teacher Evaluation Chile.pdf
Organisation for Economic Co-operation and Development (2015)
OECD Economic Surveys: Chile 2015, available online: www.oecd.org/
eco/surveys/economic-survey-chile.htm
Saville, N (2012) Applying a model for investigating the impact of
language assessment within educational contexts: The Cambridge
ESOL approach, Research Notes 50, 4-8.
Schleicher, A (2011) Lesson from the world on effective teaching
and learning environments, Journal of Teacher Education 62,
202221.
Steele, F (2009) Module 7: Multilevel Models for Binary Responses
Concepts, available online: www.bristol.ac.uk/cmm/learning/
module-samples/7-concepts-sample.pdf
Walczak, A and Geranpayeh, A (2015) Gender Gap in English
Language Proficiency? available online: www.cambridgeassessment.
org.uk/Images/genderdifferences-cambridge-english.pdf
enhance economic, social and cultural development and
opportunities in Antioquia.
The initiative includes two types of sessions: language
lessons for primary and secondary school teachers and
pupils, and methodological sessions for English language
teachers. Other components are a teacher and student
web portal containing self-access English language learning
material, teacher professional development supported
by the Open University, and a media campaign. The
introduction of these five strands is aimed to:
1. Motivate the educational community (teachers and
students), in both urban and rural environments to take
66|ISSUE 65 / FEBRUARY 2017
an interest in learning English, recognising it as a useful
tool for accessing individual work/study opportunities
and, in a broader sense, for regional development.
2. Enable teachers to develop their classroom teaching
skills in order to improve students level of English, and
promote a broader understanding and the practical use
of English among learners.
3. Improve English language competency in young
people by increasing their motivation, and making
communication skills relevant to their interests, local
contexts and work/study opportunities.
This project was implemented in collaboration with the
British Council, who is providing the training and academic
support by means of the design and implementation
of language sessions for students and teachers in five
targeted municipalities, and Cambridge English Language
Assessment who is developing self-access web portals
and providing a language placement test in English for
pupils and teachers to support the above-mentioned goals.
The material in the portals will be provided by the British
Council, Cambridge English Language Assessment and
other providers. Cambridge English Language Assessment
was commissioned to provide an evaluation of the language
and methodology lessons in the five targeted municipalities
where they have been offered. The evaluation focused on
investigating teacher and learner perceptions of the English
in the Park initiative, including an evaluation of their English
language levels. This article gives an overview of the initiative
and the attitudes and perceptions of the pupils and teachers
so far, and the performance of a sample of the target
population in the Cambridge English Placement Test.
English in the Park initiative
Language sessions
English language sessions for members of the education
community were delivered in five target municipalities
(La Unin, El Carmen de Viboral, Guarne, Tmesis and
Santa Rosa de Osos) by English language trainers called
Champions. These language Champions are native English
speakers who hold the Cambridge CELTA qualification
(Certificate inTeaching English to Speakers of Other
Languages). This qualification ensures that trainers have an
understanding of the principles of effective, communicative
language teaching, including a range of practical skills for
teaching English to young learners and adults. There are
two Champions assigned to each municipality. Language
lessons are available not only to teachers and pupils, but
also the wider education community, such as headmasters,
head teachers, directors, academic co-ordinators, and
parents of students. The aims of the training are to
introduce communicative teaching techniques and digital
training, and to improve competence and confidence in
speaking and listening. The sessions are flexible in format,
and are a mix of drop-in, regular, and scheduled sessions for
specific groups of students of different ages and in-service
UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
CAMBRIDGE ENGLISH : RESEARCH NOTES
teachers. All linguistic sessions are 90-minute sessions,
delivered by the English Champions.
All state school pupils and teachers in the target
municipalities were offered English language training. This
included English teachers and teachers of other subjects in
both primary and secondary schools. Classes for pupils are
on specific topics and areas of the English language, within
the framework of the activities of the standard school day,
the extended school day and the school holidays in each
targeted municipality. The sessions also include a formative
assessment of English language competence to teachers
and students by means of self-access resources which
promote strategies for improving English.
Methodology sessions
Methodology sessions are offered to teachers of English
in order to increase the use of communicative approaches
to teaching and improve student language learning in
their schools. At this point in the initiative, methodology
sessions have been carried out in three municipalities (El
Carmen de Viboral, Santa Rosa de Osos, Tmesis). So far,
42 teachers (19 in El Carmen, 21 in Santa Rosa de Osos
and two in Tmesis) have received methodology training.
Methodology sessions for English teachers are being
scheduled in the other municipalities, provided there are
enough teachers interested and available.
Methodology sessions are presented in three or four blocks
of 5 hours, and cover some of the following topics: teaching
approaches; teaching vocabulary; classroom management and
interaction patterns; games for children and teenagers; using
songs; using videos; and activities and strategies to learn and
improve (a) pronunciation, (b) vocabulary and (c) fluency.
Evaluation aims
This evaluation has three aims. Firstly, for pupils, we want
to investigate the effect or impact of this initiative on
their attitude towards English or their interest in learning
English. For teachers who do not teach English, the aim
is to determine whether the project has increased their
interest in learning English. Finally, for teachers of English,
the evaluation is focused on the impact of this initiative on
teaching practices and language development.
Research questions
1. What impact has the initiative had on pupils attitudes
towards and motivation to learn English?
2. Are teachers of other subjects more interested in
learning English as a result of this initiative?
3. What has been the impact of the initiative on the teaching
practices and language development of English teachers?
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|67

Methodology on the impact of the sessions on teaching practices and

language development.
In addition to measuring English language levels of students
and teachers, the evaluation also investigated the context
of learning, and both pupil and teacher attitudes and
motivation. A primarily quantitative approach was taken
Data collection
due to the limited timeframe available for data collection. The CEPT was used in order to have a snapshot of English
Questionnaires and the Cambridge English Placement Test language proficiency levels in the selected municipalities.
(CEPT) were the instruments used to collect the main data CEPT is an online adaptive test of General English, testing
for analysis. However, within the questionnaires, open- the skills of reading, use of English and listening. It has been
ended comment boxes were included to allow respondents designed to be short and flexible, and to meet the needs
an opportunity to provide additional information. This data of users who want to place English language learners at all
formed a smaller qualitative data strand which was analysed levels of the Common European Framework of Reference
separately and the findings were then used to inform the for Languages (CEFR, Council of Europe 2001), from pre-A1
findings from the quantitative data strands, thus allowing for to C2. Each test taken is virtually unique. As the candidate
a more in-depth understanding of the effect of the project on progresses through the test, each item is chosen on the
participants. In this respect, the research design incorporated basis of the candidates response to the previous item. The
a mixed method approach as shown in Figure 1. test becomes progressively easier or more difficult until a
consistent level of ability is achieved, and the candidates
Figure 1: Research design (based on Creswell 2009)
level of English can be identified. CEPT features a variety of
Quantitative data Quantitative data accents and texts sourced from a range of English-speaking
collection: analysis:
countries. The test is secure and fast, taking on average 30
Placement tests for Mapping of test results
pupils and teachers onto CEFR levels minutes to complete.
Multiple-choice Descriptive statistics of
questionnaire questionnaire
responses for pupils responses (frequency, Attitudinal and demographic data
and teachers % and mode) Integration
and Pupils and teachers perceptions of the project were
Interpretation
collected via an online questionnaire. For pupils the
Qualitative data Qualitative data
collection: analysis: questionnaire was administered in Spanish, and the
Comments in Thematic analysis of questionnaire for teachers was administered in English and
questionnaires for questionnaire
pupils and teachers comments Spanish. It consisted of multiple-choice questions asking
participants demographic and attitudinal information. For
example, participants were asked to indicate where they
lived, and what training they had attended, and whether
Areas under investigation they agreed with statements such as I feel it is important
for me to learn English and how often they engaged in
The constructs in Table 1 formed part of the conceptual
various activities, like speaking to pupils in English. The
framework for the research, and fed into the design of the
attitudinal section of the questionnaire was developed
questionnaires.
by selecting validated statements from the Cambridge
The key areas under investigation for student participants English Questionnaire Item Bank. The statements included
are: general attitudes towards English, their motivation to Likert scale items consisting of a 5-point scale requiring a
learn and their learning preferences rather than specific single response. The most typical response options were:
aspects of the initiative. For teachers, the focus is primarily strongly agree, agree, disagree, strongly disagree and

Table 1: Constructs

Area of investigation Construct Data type Instrument

Exposure to English Use of English in the classroom Quantitative Questionnaire to pupils

Attitude towards English
Learning approach
Learning oriented assessment
Perception of the English
lessons at the Park
Attitude towards the initiative
Language progression
Parental/family members ability in English
Motivation/attitudes Quantitative and Qualitative Questionnaires to pupils and
Confidence/perceptions teachers
Enjoyment of English
Use of English in the classroom Quantitative Questionnaires to pupils and
Use of technology in the classroom teachers
Learning or teaching preferences
Teacher collaboration Quantitative Questionnaire to teachers
School culture
Assessment literacy
Motivation/attitudes Quantitative and Qualitative Questionnaires to pupils and
teachers
Motivation/attitudes Quantitative and Qualitative Questionnaires to pupils and
Interest in future training teachers
Value of current training
English ability/proficiency Quantitative Test scores

UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
68|ISSUE 65 / FEBRUARY 2017
not sure. Additional project-specific statements were
developed, where necessary, by a team of researchers, and
in consultation with the British Council.
The pupil and teacher questionnaires went through an
internal review before being submitted to the British
Council and project team for feedback and final approval.
The Spanish translation for the teacher questionnaire was
provided by a native Spanish speaker with a translation
and teaching background. The Spanish translation for
the pupil questionnaire was provided by a native Spanish
speaker who is a Colombian national. Both the individuals
reviewed each others work. The Spanish versions of both
questionnaires were reviewed and edited where necessary
by the British Council. A final consultation between the
research team and the British Council and project team
took place to check the content and language of both
questionnaires before distribution.
Collation of the data
The data was collected between 9 and 17 November
2015. The British Council organised and administered
the questionnaires and CEPT in each Education Park. The
initial intention was to select a representative sample
of 100 teachers, and a sample of 100 pupils with equal
numbers from each municipality to take CEPT, and for all
participants to complete the questionnaire. However, the
CEPT was administered to additional pupils because it was
not possible to reach the desired sample of teachers due
to availability issues as the testing period coincided with
the end of the academic year. Champions administered the
test and questionnaires to participants. Administration was
therefore based on who was present during the collection
time frame for questionnaire and test completion, and the
participants in the questionnaires and the test were thus
selected through convenience sampling.
There is widespread consensus that questionnaires are not
generally given to children under the age of 8 because they
are still at an early stage of their linguistic and cognitive
development, which can make it difficult to ensure the
validity and reliability of their responses (Borgers, Leeuw
and Hox 2000); therefore, the questionnaires were not
distributed to learners below Grade 4 in primary.
Study participants
A total of 188 primary pupils and secondary pupils and
teachers took the Cambridge English Placement Test.
Details of individual test performance were provided in a
separate confidential summary document, and as individual
performance reports for distribution to test takers.
A total of 242 pupils and 69 teachers completed the pupil
and teacher questionnaires. The largest group of pupils
are in Grade 4 of primary school, but all school grades are
represented. Pupils and teachers from all five municipalities
answered the questionnaire. Two thirds of the pupil
respondents are female, and the rest male, with a similar
gender split in the teacher respondent numbers.
UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
CAMBRIDGE ENGLISH : RESEARCH NOTES
Teacher profile
Teachers who completed the questionnaire all have
higher education qualifications. The highest level
qualifications they have are either a Bachelors degree
(68.1%) a postgraduate diploma (21.7%) or a Masters
degree (10.1%). Of the 69 teachers who completed the
questionnaire, 25 (36.2%) are English teachers and 44
(63.8%) teach other subjects. The majority teach English
for 5 hours or less per week, and the two largest groups
have taught English from 13 to 510 years.
Pupil profile
As parental knowledge of English can positively influence
English language learning outcomes for their children
(Jones 2013), pupils were asked firstly, how well their
primary caregiver (i.e. mother/father/grandmother, nanny,
etc.) is able to speak English and secondly, how well other
members of their family are able to speak English. As can
be seen in Table 2, there is a reasonable level of English
within the family, but a lower level of English in terms
of parents or grandparents. This may be a reflection of
the higher proficiency of other family members such as
brothers and sisters still in the education system.
Table 2: Perceived ability to speak English
Perceived English ability Primary caregiver Other family
members
He/She cant speak English 50.0% 20.8%
... not very well 33.3% 30.0%
... moderately well 7.1% 17.1%
... very well 7.5% 29.6%
Not sure/Not applicable 2.1% 2.5%
Findings and discussion
This section reports on the key findings. As this study is
a post hoc investigation into the effects of the initiative,
findings must be interpreted cautiously as they are derived
from participants perceptions of the project and its
impact. However, the information reported here provides a
snapshot of the current situation with regard to language
education in Antioquia, which can be used to monitor the
progress of the project into the future.
The findings are presented according to the key areas the
initiative was designed to affect. First, attitudes towards
English are discussed, followed by an overview of the
learning approach used in school, perceptions of learning
oriented assessment, the perception of the English lessons
taking place at the respective parques, the general attitude
towards the initiative and finally, perceptions of language
progression and an overview of the CEPT results.
Attitude towards English
As a key objective of the project is to improve interest and
confidence in language learning, English teachers were
asked to respond to a series of statements about English.
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|69

Figure 4: Responses to I am confident that I can learn English

These were I like teaching English and whether or not their
pupils were capable of learning it, whether it was important 2%
to their pupils to learn it, and whether their pupils enjoyed 2%
7%
learning it. These statements provide an indication of the
value placed on language learning, which is most strongly
associated with the willingness to engage in it and persist Strongly agree
with it even when it becomes difficult (Csizr and Drnyei Agree
2005). Their responses are shown in Figure 2 (note: none
31% Disagree
of the teachers selected strongly disagree for any of the 58%
Strongly disagree
statements).
Not sure
Figure 2: English teachers' attitude to English language learning

I like teaching English. 57 39 4

I believe that the majority of my students are

capable of learning English. 43 52 4

Learning English is important to my students. 48 39 4 9 Learning approach

In order to investigate learners attitude towards
My students like learning English. 13 61 17 9
communicative and self-directed approaches to learning,
0% 20% 40% 60% 80% 100% they were asked a series of statements about classroom
Strongly agree Agree Disagree Not sure interactions and learning preferences.

Teacher responses show a very positive attitude to English. Figure 5: Learning approach
None of the teachers disagreed with the statement I like Our teacher encourages us to speak and
53 36 6 5
teaching English. Pupils were also asked to what extent they interact with other students in English.
Students in my class help each other to learn. 38 49 7 6
agreed with the statement I feel it is important for me to learn
Using technology in class helps me learn
English. Figure 3 summarises the responses. Three quarters better. 45 43 6 7
Working with others in pairs or groups helps
of the pupils (75%) strongly agreed, and a further 18% me learn better.
37 48 6 8

agreed, which is a slightly higher positive response rate than I know what to do to improve my English. 35 47 6 15
teacher expectations. Less than 8% disagreed or were unsure Working on my own (individually) helps me
30 38 17 10 10
learn better.
whether it is important to learn English. The large number
0% 20% 40% 60% 80% 100%
suggests that English is important to pupils regardless of age,
Strongly agree Agree Disagree Strongly disagree Not sure
gender or which municipality they come from.

Figure 3: Pupil responses to I feel it is important for me to learn

English
1%
0%
6%
18%
75%
Pupils self-confidence in learning English was also
investigated. As Figure 4 shows, the majority of pupils
(89%) agreed, or strongly agreed that they were confident
they could learn English. Although teachers appear to be
more confident in their pupils ability to learn English than
the pupils themselves (93% versus 89% respectively),
almost two thirds of the pupils (58%) strongly agreed with
this statement, indicating they are very confident in their
capacity to learn English.
As can be seen in Figure 5, around 80% of the pupils
agreed or strongly agreed that they knew what to do to
improve their English, that they spoke and interacted with
other pupils in English, and that pupils helped each other to
learn. More than 80% of pupils also agreed that technology
helped them to learn.
Strongly agree
In terms of learning preferences, over 80% of pupils felt
Agree
that working in pairs or groups helped them to learn. Over
Disagree 60% of pupils also felt that working on their own was also
Strongly disagree beneficial. These responses suggest that learners recognise
Not sure their role in the learning process and are willing to engage
in learning actively. Although it is not possible to determine
whether the project has positively affected these attitudes,
the learning approaches and preferences reported by pupils
here may support subsequent phases of the project which
include self-directed learning activities.
Learning oriented assessment
Assessment has multiple functions in education, including
the measurement of achievement and the provision of
feedback to learners on their strengths and weaknesses.
When assessment practices are linked closely with learning
goals, it has the ability to both motivate learners and
improve learning (Carless 2009). Therefore, pupils were
asked about the feedback they receive from their teacher,
and whether it helps them to learn better. They were also

UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
 70|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

asked if they were able to judge the quality of their own Perception of the English lessons at the Park
work and that of other pupils, and finally if it was clear to Having looked at general attitudes to learning English, we
them what they were supposed to learn. Their responses will now look at the perception of what was happening
are summarised in Figure 6. during the English lessons, before moving on to look at
attitudes to the initiative.
Figure 6: Feedback and assessment: Pupil attitudes
As a main objective of the project was to improve listening
The feedback helps me understand things better
and improve my ways of learning.
61 30 4 6 and speaking skills, pupils were asked to comment on
It is clear to me what I am supposed to learn. 44 46 7 7 the use of English in their lessons and the opportunities
I am able to judge the quality of my own work. 26 39 18 9 14 to interact with others in class. The responses for pupil
After taking an English test, the teacher gives me
feedback on my strengths and weaknesses.
21 41 18 9 16 perception of use of English are summarised in Figure 8.
I am able to judge the quality of others' work. 12 26 26 23 17
Figure 8: Use of English during the English in the Park lessons
0% 20% 40% 60% 80% 100%

Strongly agree Agree Disagree Strongly disagree Not sure The teacher encourages us to
76 18
speak in English.

Over 90% of pupils reported that they receive feedback The teacher primarily speaks
74 19
English.
from the teacher which helps them to understand things
better and that it is clear what they are supposed to learn, I primarily speak to the teacher in
20 47 19 11
English.
although a lower percentage (62%) reported they received
feedback on their strengths and weaknesses. Over 60% I primarily speak to other pupils
15 35 25 24 2
report they are able to judge the quality of their own work, in English.

but a lower number (38%) feel they can judge the quality 0% 20% 40% 60% 80% 100%
of their classmates work.
Very often Sometimes Not very often Never Not sure
Teachers were also asked to respond to a series of
statements on assessment and feedback in order to The results show frequent use of English by the teacher,
determine the extent to which they are engaging in with 74% of pupils reporting the teacher speaks in English
assessment for learning practices. Their responses are very often, and a similar percentage (76%) report that
summarised in Figure 7. their teacher encourages them to speak in English very
often. Although the frequency of speaking in English is also
Figure 7: Feedback and assessment: Teacher attitudes lower for pupil interaction, with the highest percentage
I ask students to evaluate their own work or the
(35%) reporting they sometimes speak in English to other
23 68 5
work of others.
I gather information about my students' progress 9
pupils, the percentage of pupils speaking English to their
77 9
in English using a variety of methods. teacher is reasonably high, with 20% reporting they do
I use test results to decide what to teach next. 36 50 9 5
this very often, and a further 47% reporting they do this
I tell my students about the criteria on which
they will be evaluated.
45 36 9 some of the time. Pupils were also asked to indicate how
I regularly give my students feedback on their 18
strengths and weaknesses.
59 14 9 often they worked in pairs, groups and individually as this
I feel confident in designing my own English tests 18
or quizzes.
55 14 9 can also indicate the extent to which a communicative
I feel confident in gathering information about 14
students' ability in order to assess their progress.
59 14 14 approach to learning is being encouraged. The responses
0% 20% 40% 60% 80% 100% are summarised in Figure 9.
Strongly agree Agree Disagree Strongly disagree Not sure Figure 9: Class dynamics during the English in the Park lessons

The teacher responses indicate that teachers feel they are I work with other pupils in pairs. 47 39 4
engaging in a variety of assessment practices and also
I work with other pupils in groups. 35 46
46 9
involve their pupils in these processes. The statement with
the most agreement was I ask students to evaluate their I work on my own (individually). 27 33 17 16
own work or the work of others which 91% of teachers
agreed or strongly agreed with. The three statements with a 0% 20% 40% 60% 80% 100%
lower rate of agreement than the others, which are all above Very often Sometimes Not very often Never Not sure
85%, were two about confidence, such as I feel confident
ingathering information about students abilityin English Comparing working individually, in pairs or in groups,
in order to assess their progress and I feel confident in the most common dynamic in the classes is pair work,
designing my own English tests or quizzes which both with 90% of pupils reporting they work in pairs very
have 73% agreement, and a third where 77% of teachers often or from time to time. Although the use of pair
agree or strongly agree with the statement I regularly give work is positive as pupils are potentially being given an
my students feedback on their strengths andweaknesses. opportunity to communicate with others, the fact that
This is a higher rate of agreement than the pupil perception; only 47% of pupils report speaking to classmates in
however, it does indicate that teachers may value additional English suggests that more work needs to be done either
training in assessment for learning practices. to ensure the pair work activities encourage English use

UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|71

Figure 12: Responses to I am pleased that I was given the

or that learnersarepushedto use English with classmates opportunity to participate in the English in the Park initiative
morefrequently.
Pupils 72 20 5
As one aspect of the project is to encourage the use of
technology in learning, pupils were asked to indicate how Teachers 70 17 13
often they used technology in the English in the Park
lessons. The responses are summarised in Figure 10. 0% 20% 40% 60% 80% 100%

Strongly agree Agree Disagree Strongly disagree Not sure

Figure 10: Frequency of use of technology in class*
The generally positive attitude to the project was also
evident in the additional comments received in the
questionnaire, such as this comment from a Grade 11 pupil
3% in Tmesis:
11%
Very often I thank you for this great opportunity to learn English in
32% Sometimes the educational park, I congratulate the teacher for the
14%
Not very often methodology and the excellent management of the systems,
this makes us learn and enjoy the lessons. I hope to be able
Never
to continue with this process next year and that such an
Not sure interesting and productive process does not stop. Thousands
36% and thousands of thanks for bringing to Tmesis such an
excellent programme and teachers.

Similarly, comments made by teachers also indicated a

positive attitude to the project; for example, a non-English
*Percentages may not sum due to rounding
teacher from El Carmen de Viboral made this comment:
Use of technology in the lessons is quite frequent with over
After these classes in the Park, I feel more confident when I
70% of pupils reporting they use technology very often or
have to express myself in English.
from time to time.
The majority of pupils agreed that they enjoyed using
In order to investigate the extent to which pupils were
English more, and enjoyed English lessons more as a result
encouraged to develop self-directed learning strategies,
of the initiative, as can be seen in Figure 13.
pupils were asked how often they were asked by the
teacher to judge the quality of their own work and the work Figure 13: Positive impact of the English in the Park initiative
of their classmates during the English in the Park lessons.
The pupil responses are summarised in Figure 11. I enjoy using English more than before. 66 27

Figure 11: Assessment during the English in the Park lessons I enjoy English lessons more than before. 69 23

0% 20% 40% 60% 80% 100%

The teacher asks me to judge the quality of
5 12 8 63 12
work of other pupils.
Strongly agree Agree Disagree Strongly disagree Not sure

The teacher asks me to judge the quality of

12 14 14 48 12
my own work. This positive attitude was also evident in the comments in
0% 20% 40% 60% 80% 100%
the questionnaire, such as this one from another Grade 11
pupil from Tmesis:
Strongly agree Agree Disagree Strongly disagree N/A

I thank you for this opportunity that we are given to learn

The figures show that the majority of pupils are asked to
assess their work or the work of their classmates during the
lessons. Only 26% of pupils are asked to judge the quality
of their work with any frequency, and only 17% are asked to
judge the quality of the work of their peers.
English, I find that the methodology used by the teacher is
extraordinary and what I want the most is that this process
continues next year because I can see progress and I do
not want to waste everything I have achieved until now.
Thousands and thousands of thanks for bringing to this region
this opportunity.

This pupil from Santa Rosa de Osos commented:

Attitude to the initiative
Having looked at general attitudes to learning English, and I think the lessons of English in the Park have made my
learner perceptions of what was happening in the English learning faster.
lessons, we will now focus on participant perceptions of the
There were, however, less positive views such as this one
initiative and the impact of it on learning English. Although
on the level of teacher expertise, made by a pupil in Guarne,
attendance levels varied, the attitude to the initiative
but these were in the minority:
was very positive, with the majority of pupils (92%)
indicating they were pleased to participate, as can be seen I would like teachers to be better trained, but in general the
in Figure12. Teachers were also positive about being able lessons are good and I appreciate that English is having a
toparticipate. greater presence in educational centres.

UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
72|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Of the 114 comments left by pupils, 86 (75%) were Methodology sessions

positive, 7 (6%) were mixed, 12 (11%) were negative, and English in the Park also included sessions for teachers
6 (5%) were about their school and not the initiative. The focusing on improving their teaching practices. Teachers
positive view such as this one from a pupil in La Unin is were asked to comment on the usefulness of the
typical of the comments received, which praise the initiative methodology sessions and how interested they would be in
and hope it will continue: workshops in the future. Their responses are summarised in
Figure 16.
I think the teaching is excellent, I hope they will keep it like this.
Figure 16: Usefulness of the methodology sessions
Teachers were asked if they would like to attend language
training in the future, and as can be seen in Figure 14, the Activities and strategies to improve 41 45 14
vocabulary
response was very positive. Activities and strategies to
38 48 14
improve fluency
Teaching vocabulary 31 55 14
Figure 14: Responses to I would like to attend English language
training in the future* Teaching approaches 31 55 14
Classroom management and interaction 38 45 17
patterns
2% Activities and strategies to improve
pronunciation 41 41 3 14
3%
6% Using videos 34 45 3 17

Games for children and teenagers 38 41 7 14

Strongly agree Using songs 34 41 7 17
17% Agree 0% 20% 40% 60% 80% 100%

Disagree Very useful Quite useful Not very useful N/A

Strongly disagree
73% Not sure The first four workshops, on teaching vocabulary, teaching
approaches, and those on strategies to improve fluency
and vocabulary were rated most highly, although all
the workshops were rated as useful by the majority of
teachers. No workshops were rated as not at all useful,
*Percentages may not sum due to rounding
although this was one of the options. The N/A option was
Teachers also commented on the importance of this initiative selected by those who had not attended that particular
and the need to increase the length of the training programme, workshop. The quality of the training was commented on
as can be seen in the selection of comments below: by teachers:

Its very important and good that these opportunities are
provided as not everyone has the opportunity to learn as
English classes are very expensive and not everyone can
affordthem.
These classes should have begun at the beginning of the year
and should have been for more time at least a year and
according to progress, students can move level.
Teachers were also very positive about the effect the
initiative had on their teaching, as can be seen in Figure 15.
The majority of teachers reported that the initiative had a
very positive or positive effect. Although a small percentage
indicated it had no effect (7%), no teachers indicated it had
a negative effect even though this was one of the options.
Figure 15: Effect of teaching of the English in the Park initiative
7%
A very positive effect
48% A positive effect
45%
No effect
Its very important to continue with native teachers to improve
and strengthen the process of teaching and learning, and share
experiences with peers.
For me, its necessary to continue with training in new
methodology for English teaching. Particularly for the teachers,
since we are the people in charge of passing on knowledge to
students. Please, continue with the teachers Lee, Selina and
Tom in Parque Neurona del Municipio de Santa Rosa de Osos.
Its really important.
Although the majority of the teachers were positive, there
were some recommendations made in the comments
related to the planning and scheduling of lessons:
Its a good opportunity. Its difficult to find the time due to the
working day and the location. It would be easier for me if it
were closer. Thanks.
I would like the rural teachers to have a full English course
from beginners level in order to learn and be able to teach this
language in the classroom. Bear in mind that many of us travel
to the municipal headquarters on Fridays.
It would be good if the number of classes during the week was
increased and other methodological strategies used.
English teachers were then asked about the impact of the
English in the Park sessions on their teaching practices, and
knowledge of and confidence in English language teaching.
Figure 17 summarises responses to those statements
relating to knowledge, understanding and abilities.

UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|73

Figure 17: Teacher improvement in abilities and knowledge

Language progression
have improved my ability to adapt
English lessons for different learners' 31 59 10 The most common levels for primary pupils were pre-A1
needs and abilities
have improved my knowledge of how
and A1 depending on the region. Overall the most common
31 59 10
foreign languages are learned CEFR level for primary pupils was A1. For secondary pupils,
have helped me to improve my practical
28 59 10 the most common levels were A1 and A2 depending on the
classroom skills
region, and overall A1 was the most common.
have improved my ability to plan better
24 59 14
English lessons
Looking at teacher performance, because of the small
have helped me to understand how
English is taught in other schools
31 41 3 24 numbers, English and non-English teachers were counted
0% 20% 40% 60% 80% 100% together. For teachers, the most common scores were A1
and A2 depending on the region. The most common level
Strongly agree Agree Strongly disagree Disagree N/A
overall was A1.

The responses show that teachers feel their ability to adapt Overall the range in pupil and teacher scores shows the
and plan lessons have both improved. They also feel their variety in performance in each region. Pupils who took the
practical classroom skills and their theoretical knowledge test ranged in age from 9 to 18, and only a limited number
in terms of how languages are learned have improved. of teachers were available to participate in the test. This
Over 80% of teachers agreed with the statements about limits the conclusions which can be drawn from the results.
these areas of improvement. In fact, all teachers who Nevertheless the results help to build a picture of the
indicated they had attended the workshops indicated they context at this point in the initiative.
had improved in their ability to adapt lessons and their
Pupils were also asked in the questionnaires how much
knowledge of language learning. 72% of teachers agreed
they felt their English had improved since the beginning
that the sessions helped them to understand how English is
of the academic year. Overall pupils felt that they have
taught in other schools.
improved most in listening over the academic year, followed
Teachers were also asked about the extent to which sharing by speaking, and least in grammar (see Figure 19).
and collaboration within their schools had improved as a
Figure 19: Pupil perceptions of improvement in different areas of
result of the English in the Park initiative. The responses English
are summarised in Figure 18.
Listening 43 30 25 5
Figure 18: Teacher collaboration at school as a result of the
initiative Speaking 32 34 26 5 8

Share resources 28 52 4 17 Writing 36 28 29 7 5

Talk to each other about how to improve
our teaching 31 48 4 17 Reading 30 35 29 6 5
Talk to each other about our class'
progress 38 38 9 17 Vocabulary 38 24 29 7 7
Discuss our lesson plans with each other 21 55 9 17
Grammar 21 30 29 10 15
Work with each other to try out new ideas 28 41 10 24
0% 20% 40% 60% 80% 100%
Collaborate to prepare tests 17 45 17 28
A lot A moderate amount Only a little bit Not at all Not sure
Have watched each other teaching 14 38 33 31
0% 20% 40% 60% 80% 100%
This comment, from a pupil in Guarne, illustrates that pupils
A lot more than before The same as before
are aware of changes in their English proficiency as a result
A bit more than before N/A
of attending the lessons:

In these months that I have been attending the lessons in the

None of the teachers selected less than before. For the
programme English in the Park in Guarne I have been very
teachers who selected 'N/A', it is assumed they are either happy because it was very useful, as support for my school
unsure, or do not carry out these activities in their school. lessons, and it was a great opportunity to learn and improve
The three areas where the highest number of teachers my knowledge. I know that if I make an effort and practise, I
indicated they collaborated more than before were sharing am going to be able to progress more in the language. Onthe
resources, talking about how to improve teaching, and other hand, I would like to have this course next year too.
discussing class progress. None of the teachers indicated Congratulations! This was of great help to me. Thank you
that the level of collaboration and sharing had decreased. verymuch.
Use of English in class by pupils and the teacher, and use
However, the improvement cannot be solely attributed to
of technology were felt to have increased more than before
the English in the Park lessons alone, as we asked how
by 93% of the respondents. Working in pairs and groups
much they had improved over the year. Therefore English
was felt to have increased by 90% of teachers. None of the
classes at school are also a factor. Nevertheless, pupil
teachers felt that any of the activities were occurring less
comments indicate that they feel the English in the Park
often than before, with the exception of asking students to
lessons have contributed to their improvement.
work individually, where 7% reported they were doing this
less often, and for 14% there was no change.

UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
74|ISSUE 65 / FEBRUARY 2017
Teachers were also asked to comment on the areas where
pupils had improved. Vocabulary and grammar were the
two areas where the greatest amount of improvement was
noted by teachers, as can be seen in Figure 20.
Figure 20: Teacher perceptions of improvement in different areas
of English secondary student scores by CEFR level for each skill
Vocabulary 27 45
Grammar 5 55
Speaking 18 23 45
Writing 5 36 41
Listening 18 9 64
Reading 9 18 64
0% 20% 40% 60%
A lot A moderate amount
Not at all Not sure
Questionnaire responses also show that teachers feel they
have improved in all areas, and the most popular response
was in the area of vocabulary, followed by listening. Teachers
commented on the motivational aspects of the training:
The teacher is excellent, brilliant, but my skills are poor. I love
learning.
The programme has encouraged our enthusiasm for second
language teaching and our desire to learn more and master it.
I would quite like this initiative to continue with the foreign
teachers as their way of teaching is different and you learn
a lot more with them. I really appreciate the opportunity to
learnEnglish.
Conclusions
The first question the evaluation seeks to answer is what
impact the initiative has had on pupils attitudes towards
and motivation to learn English, and secondly whether
teachers of other subjects are more interested in learning
English as a result of taking part. The findings are that all
three stakeholder groups, namely pupils, teachers of other
subjects and teachers of English, are pleased to be given
the opportunity to participate in the initiative. Learners
and teachers enjoyed the English lessons and found them
useful in improving their speaking and listening skills,
and commented positively on the variety of interactional
patterns, such as interacting with the teacher and with
other learners, and the predominant use of English in
lessons. The lessons appeared to motivate learners to use
English more than before and improved their enjoyment of
learning English. In fact, learners appear to be engaging in
self-directed learning, which is a positive sign.
The other area of investigation is the impact of the
initiative on the teaching practices and language
development of English teachers. The feedback from
English teachers has shown that all workshops have
been considered useful to teachers. In the near future,
UCLES 2017 THE IMPACT OF THE ANTIOQUIA ENGLISH IN THE PARK INITIATIVE ON LANGUAGE TEACHING AND PRACTICE
CAMBRIDGE ENGLISH : RESEARCH NOTES
the workshops teachers are most interested in are firstly,
games for children and teenagers, and secondly, how to
improve pronunciation. The methodology training has had
a positive impact on teaching practices, with teachers
reporting that they are engaging in more communicative
approaches to teaching English in their classrooms than
before, for example using more group and pair work, and
27
increased use of English in the classroom. They also report
they are better able to adapt lessons for learner needs.
27 14 These improvements are also evident in their attitude
9 5 to their work, with teachers reporting that the training
has also improved their motivation and confidence when
9 9
teaching English. Training benefits are also evident at
9 the school level, with teachers reporting that they are
9 collaborating more often with their colleagues by talking
about their classes, sharing resources and working with
80% 100%
each other to try out new ideas.
Only a little bit
Test data shows a range in performance across the regions
with the majority of participants at a basic level of English.
The pupils who took the CEPT test ranged in age from 9
to 18. We would recommend testing the English skills of a
cohort of pupils from one age group, or one school year, so
that meaningful conclusions can be drawn.
Recommendations
Simply put, everyone would like to continue studying
English and English teachers would like to continue taking
methodology sessions. As one teacher commented:
An excellent initiative which should be continued for a
longerperiod.
We recommend continuation of the project, and expansion
of the training programme in terms of running more
methodology and language sessions in all municipalities,
and possibly expanding the range of workshops in response
to teacher feedback. For example, we would recommend
considering workshops that focus on assessment for
learning (i.e. learning oriented assessment) in order to build
teacher confidence and their ability to engage in a range of
assessment activities, such as planning and collaborating
on test production, and making use of assessment data
to drive forward the learning. We would also recommend
the investigation of further course planning, particularly
looking at how course planning could be used to help
teachers integrate use of the methodologies demonstrated
in thesessions.
Classroom observations could be incorporated in future
phases as an additional professional development
opportunity which may improve teaching and collaboration
within and across schools. Finally, as the initiative continues,
introducing English teaching qualifications may be beneficial
such as Teaching Knowledge Test (TKT) or qualifications
targeting English teachers of primary, Certificate in English
Language Teaching Primary (CELT-P), and secondary,
Certificate in English Language Teaching Secondary (CELT-S),
which could be seen as added value for those teachers who
have demonstrated high achievement and engagement.
CAMBRIDGE ENGLISH : RESEARCH NOTES
Limitations
The pupils who took the CEPT test ranged in age from 9
to 18. We would recommend testing the English skills of a
cohort of pupils from one age group, or one school year, so
that meaningful conclusions can be drawn.
The post hoc nature of this investigation, looking
retrospectively at changes, and reliance on a teacher and
pupil willingness to participate, means that the perceptions
may not reflect actual changes, since no baseline data
was available. Findings, therefore, should be interpreted
cautiously; however, these findings can be used to monitor
programme effectiveness as it progresses.
References
Borgers, N, Leeuw, E D and Hox, J (2000) Children as respondents
in survey research: Cognitive development and response quality 1,
Bulletin de Mthodologie Sociologique 66, 6075.
Improving student learning through upskilling
teachers: The case of Lebanon
COREEN DOCHERTY CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
TANIA BARAKAT ALLC INTERNATIONAL HOUSE BEIRUT, LEBANON
ELAINE KNIVETON ALLC INTERNATIONAL HOUSE BEIRUT, LEBANON
LAMA MIKATI ALLC INTERNATIONAL HOUSE BEIRUT, LEBANON
HANAN KHALIFA CAMBRIDGE ENGLISH LANGUAGE ASSESSMENT
Project background
The Developing Rehabilitation Assistance to Schools and
Teachers Improvement (D-RASATI) programme was a
5-year USAID-funded project to support the Lebanese
Ministry of Education and Higher Education (MEHE) in
improving the performance of the public school system.
One of the objectives of the programme was to improve
English language levels of public school teachers of English
and those who teach through the medium of English.
The D-RASATI project consisted of two phases. In the
first phase, D-RASATI 1, Cambridge English Language
Assessment in partnership with ALLC International
House Beirut (ALLC-IH) assessed over 4,000 primary
and secondary Lebanese teachers in reading, listening,
speaking and writing skills, using BULATS (Business
Language Testing Service), a multi-level English exam aligned
to the Common European Framework of Reference for
Languages (CEFR, Council of Europe 2001) and measuring
from CEFR Level A1 to C2. This benchmarking activity
was designed to support the MEHE, and the teachers
UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
ISSUE 65 / FEBRUARY 2017|75
Carless, D (2009) Trust, distrust and their impact on assessment
reform, Assessment and Evaluation in Higher Education 34 (1), 7989.
Council of Europe (2001) Common European Framework of
Reference for Languages: Learning, Teaching, Assessment, Cambridge:
Cambridge University Press.
Creswell, J W (2009) Research Design: Qualitative, Quantitative and
Mixed Methods Approaches (3rd Edition), Thousand Oaks: Sage.
Csizr, K and Drnyei, Z (2005) The internal structure of language
learning motivation and its relationship with language choice and
learning effort,The Modern Language Journal89 (1), 1936.
Jones, N (2013) The European Survey on Language Competences
and its significance for Cambridge English Language Assessment,
Research Notes 52, 26.
in-service professional development, by providing the
Ministry with a national profile of English language levels
of teachers in order to target training for those who teach
English as a first or second language or use English as the
medium of instruction for science and mathematics. The
benchmarking activity identified that approximately 95%
of teachers needed to improve their English language levels
in order to achieve the language targets recommended by
D-RASATI of CEFR Level C1 for English language teachers
and B2 for teachers who teach content through the medium
of English (American University of Beirut 2012).
In the second phase, D-RASATI 2, ALLC-IH was
commissioned to provide approximately 2,500 teachers
with English language courses between January 2014 and
February 2016 as part of the English Language Training for
Public School Teachers Activity. These courses ranged from
basic to advanced and targeted all four language skills:
reading, writing, speaking and listening. A critical aspect
of the D-RASATI 2 project and this activity was to ensure
that teachers throughout the country had easy access to a
language course appropriate for their needs. An additional
76|ISSUE 65 / FEBRUARY 2017
feature of these courses was that those teachers who
were enrolled in the B2 level course were eligible to sit
the Cambridge English: First exam, providing them with
the opportunity to attain international certification and
indicatethey have achieved operational proficiency in English.
In order to determine the extent to which these language
courses met the needs of teachers and contributed to the
aim of the D-RASATI 2 programme to improve the English
language standards of Lebanese teachers, Cambridge
English, in collaboration with D-RASATI 2, undertook an
evaluation of the project, which we report in this article.
D-RASATI 2 English Language
Training
Course offerings
D-RASATI 2 offered public school teachers seven
courses covering CEFR Levels A1 to C1 in 17 locations
across Lebanon. The courses were designed based on a
communicative approach to language instruction, and
they were learner centred with learners expected to
fully participate and be 100% involved in their language
development. An inductive approach was used, requiring
learners to work out meaning for themselves rather than
having language explained to them. A major focus was on
communication, thus building the teachers confidence and
ability to run their own classes in English. In addition, the
teacher trainers provided a sound model of communicative
methodology including a range of different lesson shapes
such as text-based presentations, testteachtest and task-
based learning. The teachers could then take these models
and use them in their own classrooms.
Each course was scheduled for a 3-hour period, two days per
week, except for the courses which ran over summer 2015
which were 3 hours per day, four days per week. Enrolment
varied by location based on the number of public schools in
each region which use English as a language of instruction as
opposed to French, but approximately 2,500 teachers were
enrolled in courses over the duration of the project,with
around 25% of teachers taking more than one course.
Teachers were placed into courses based on their BULATS
score and availability of classes in their region. That is, a
class would be set up in a region if there were at least seven
teachers at the same language level.
English language trainers
A total of 49 trainers were recruited to deliver the English
language courses across all sites. All trainers were
university graduates and the majority held the Cambridge
English CELTA (Certificate in Teaching English to Speakers of
Other Languages). This qualification ensures that trainers
have an understanding of the principles of effective
communicative language teaching, including a range of
practical skills for teaching English to adult learners. Those
who did not hold the CELTA had considerable English
language teaching experience and underwent training
UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
CAMBRIDGE ENGLISH : RESEARCH NOTES
with the ALLC-IH Academic Management Team before
they started the courses and throughout the duration of
theproject.
Cambridge English: First
A key indicator of this project was for 245 teachers to be
prepared to sit the Cambridge English: First (also known
as FCE) exam and be given the opportunity to achieve a
B2 level certificate in English. The Cambridge English: First
exam is a standard high-volume global Cambridge English
exam, calibrated at B2 level of the CEFR, but reports on
performance from B1 to C1. It tests learners in all four skills:
reading, writing, listening and speaking. The Cambridge
English: First exam was selected for this project as it
represents the first target proficiency set for teachers who
use English as an instructional language. Teachers who
were enrolled in the Pre-FCE course were selected to take
the Cambridge English: First exam based on their overall
progress in their language course, their performance in a
series of mock tests and recommendation by their English
language trainer.
Evaluation aims
This evaluation study covers the period between January
2014 and August 2015 and aims to answer the following
research question: What are stakeholders (decision
makers, teachers, English language trainers) attitudes
to and perceptions of the language courses offered by
D-RASATI 2 as part of the English Language Training for
Public School Teachers Activity and the use of internationally
recognised assessment tools?
This study was designed to provide USAID and MEHE
with a comprehensive profile of the Lebanese teachers
who participated in the courses as part of this activity. This
profile can be used to inform future developments aimed at
supporting MEHEs long-term objective of raising standards
in English language teaching and learning. It can facilitate
data-driven decisions and recommendations to sustain
conditions for success, and help to identify any areas
forimprovement.
Methodology
Research design
A mixed methods approach was used due to its value in
providing a richer understanding of a situation through the
collection and integration of complementary qualitative
and quantitative data (Creswell and Plano Clark 2011).
A convergent parallel design was chosen, which is
characterised by the collection of the two data strands in
parallel (see Figure 1).
Areas under investigation
The key areas listed in Table 1 formed part of the conceptual
framework for the research, and fed into the design of the
parallel questions for the questionnaires and interviews.
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|77

Figure 1: Example of convergent parallel design procedural

diagram (Adapted from Creswell and Plano Clark 2011:69)
their trainers. Online and paper-based versions were made
available to teachers at the end of each English language
Quantitative data collection course between January 2014 and August 2015. English
& analysis
Procedures: language trainers completed the online version of the
Questionnaires
Test data questionnaire in July 2015.
Analysis:
Descriptive analysis The analysis of the questionnaires consisted of two steps.
Group comparisons
Products: Merge data and interpret First, frequencies were calculated for each statement
Test scores results
Frequency % Procedures:
using IBM SPSS Statistics 20 and then response patterns
Mode Consider how merged of different groups within and across questionnaires were
results produce a better
understanding of the compared by performing a chi-square test. Variables
Quantitative data collection situation
& analysis Products:
such as gender, subject specialism, grade taught and
Procedures: Discussion English course attended were used to define groups within
Semi-structured interviews
Open-ended comments in the data and compare responses. For the chi-square
questionnaires
Analysis: analysis, the critical value to determine whether there
Thematic analysis were any statistically significant responses was r <.005;
Products:
Major themes a standardised residual of +2.0 or 2.0 was used to
determine whether any particular response was showing a
meaningful difference between groups. Cambridge English:
Quantitative data: Questionnaires and test data
First exam data was also collected and analysed by CEFR
The quantitative data consisted of:
level and by skill for the December 2014, June 2015 and
responses to questionnaires completed by teachers and August 2015 exam sessions.
the English language trainers
Qualitative data: Interviews and questionnaire
candidate test scores.
comments
Attitudinal questionnaires were designed in order to Semi-structured interviews were conducted with key
elicit participant perceptions of the D-RASATI 2 project focal persons in order to identify the intended impacts
in general and the English language courses in particular. and purposes of the initiative at the micro (learning and
Two questionnaires were developed, one for the teachers teaching) and macro levels (schools and society). The
enrolled in the language courses and the other for the interviewees were asked to consider the programme goals
English language trainers. Validated statements were and the extent to which they had been achieved, reasons
selected from the Cambridge English Questionnaire Item for why this was the case, any unintended impacts of the
Bank. Additional questions, specific to this context and project, and lessons learned. The one-to-one interviews
project, were developed in collaboration with ALLC-IH. were conducted in English with a representative of
The statements included a variety of response options D-RASATI and a key focal person from the MEHE on
with the most common being Likert scale items consisting thisproject.
of a 4-point scale requiring a single response. The most
The teacher and English language trainer questionnaires
typical response options were: Strongly agree, Agree,
included open-ended fields to allow respondents to include
Disagree and Strongly disagree. A key feature of the
information that was not captured in the selected response
questionnaires was that they contained parallel statements
statements. There was an open-ended field at the end of
in order to compare the responses of teachers to those of
each section (i.e. on course scheduling; effectiveness of

Table 1: Key investigation areas

Area of investigation Sub-areas Data type Instrument

Attitude towards English
Course implementation
Lesson effectiveness and
teacher engagement
Impact of courses
Importance of English language learning Qualitative Interviews with decision makers
Confidence in learning English Open fields in questionnaires
Attitude towards the D-RASATI 2 project
Quantitative Questionnaire to teachers and
English language trainers
Course placement, scheduling and delivery Qualitative Interviews with decision makers
Open fields in questionnaires
Quantitative Questionnaire to teachers and
English language trainers
Teacher engagement Qualitative Interviews with decision makers
Effectiveness of ELTs teaching practices Open fields in questionnaires
Effectiveness of assessment
Quantitative Questionnaire to teachers and
English language trainers
English language improvement Qualitative Interviews with decision makers
Teacher confidence Open fields in questionnaires
Teacher adoption of new pedagogical practices
Quantitative Questionnaire to teachers and
English language trainers
Exam data

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
78|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

classroom activities; assessment practices; impact of the in more than one grade, English is the most commonly
course). The qualitative data was analysed using theme taught subject (36%) and more teachers work in the
analysis, which involved the identification of patterns in the primary educational stage than the others (37%). The vast
data sets and then determining the themes based on these majority of respondents work in villages with fewer than
patterns (Braun and Clarke 2006). 5,000 people (41%). Further analysis of the data revealed
significant differences in teacher profiles related to gender
(r < .001). Male respondents indicated they were educated
Study participants to a higher level than female respondents, and that they are
also less likely to teach English and to teach at primary level
The number of participants between January 2014 and
than their female counterparts.
August 2015 and the response rates for the questionnaires
can be found in Table 2. It is important to note that the The background information collected via the
figures for teachers do not represent unique individuals questionnaires indicates that the course participants were
because, in some cases, teachers took more than one made up of experienced teachers, from all educational
course or repeated the same course thus filling out the stages and representing the three main subject areas
questionnaire more than once. targeted for training: English, mathematics and science.
However, gender-based differences were identified
Table 2: Questionnaire returns in the teacher profiles. English language and primary
school teachers tended to be female in the dataset and
Stakeholder Questionnaire Total number of Response
returns participants rate mathematics and science teachers tended to be male.
Secondary school teachers also were typically male. The
Teachers 1,666* 2,330 72%
male teachers who responded to this survey tended to be
English 44 49 90%
language educated at a higher level than the female teachers. These
trainers differences suggest that the English language teaching
*Duplicate responses and responses with fewer than 12 questions answered population may benefit from increased recruitment of
were removed from the data. In some cases respondents did not answer male teachers, as research suggests that language learning
all questions, and calculations in the section Findings and discussion, are outcomes of male students can benefit from having
based on total responses for individual questions. male role models (Loulidi 1990). Similarly, increasing the
presence of female teachers in mathematics and science
education, particularly those who already are proficient in
Participant profiles English, may also benefit learners by providing female role
As can be seen in Table 3, the majority of the teachers and models for those subjects.
English language trainers who participated in this project
Another important aspect of the programme is to provide
were female (76% and 66% respectively).
equal access to quality education for all teachers regardless
of location. The majority of questionnaire respondents
Table 3: Teacher and English Language Trainer (ELTs)
demographic profile*
worked in villages or small towns, which suggests that the
project organisers were effective in targeting a teaching
Gender Teachers ELTs population which may not have the same access to teacher
Male 24% 34% training opportunities because of their location.
Female 76% 66%

Findings and discussion

Age Teachers ELTs
Under 30 6% 34%
3140 44% 39% This section reports on the findings from all data strands by
4150 32% 25% construct. First, attitudes towards learning English and the
importance of it to the participating teachers is presented,
51 and over 17% 2%
followed by participant perceptions of the English language
*Percentages may not sum to 100 due to rounding. courses, teachers level of engagement in the courses and
the perceived effect of the courses on teachers language
development and their pedagogical practices.1
The teachers were also asked in the questionnaire about
their teaching experience, the subjects and grades they Part of the analyses included group comparisons with
teach, as well as the location of their school (see Table 4). gender, subject specialism, grade taught and English course
These variables were used to investigate whether particular attended (only the level of significance is reported below).
characteristics affected perceptions of the project. As It should also be noted that several of the variables of
Table 4 shows, over half of the teachers who responded interest were confounded in the dataset as the majority
have more than 15 years of teaching experience (55%). of female respondents were English teachers in primary
Although most teachers teach more than one subject and school whereas the majority of male respondents were

Please note that this article contains unedited quotes and comments and therefore linguistic errors may occur.
1

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|79

Table 4: Teaching profile and teacher experience*

How many years have you been teaching?** Male teachers Female teachers Total

Less than 1 year <1% <1% <1%

13 years <1% <1% <1%
410 years 14% 19% 18%
1115 years 29% 25% 26%
More than 15 years 55% 55% 55%

Location of the school you teach in Male teachers Female teachers Total

Rural area (fewer than 1,000 people) 5% 6% 6%

Village (1,001 to 5,000 people) 30% 45% 41%
Small town (5,001 to 15,000 people) 28% 20% 22%
Town (15,001 to 100,000 people) 20% 10% 13%
Small city (100,001 to 500,000 people) 9% 10% 9%
Large city (more than 500,001 people) 8% 9% 9%

Highest teaching qualification Male teachers Female teachers Total

Certificate/diploma 18% 26% 24%

Bachelors degree 24% 38% 34%
Postgraduate certificate/diploma (including teaching 29% 22% 25%
certificate)
Masters degree 28% 13% 17%
PhD 1% <1% 1%

What do you teach? Male teachers Female teachers Total

English only 10% 44% 36%

Mathematics only 52% 16% 25%
Science only 31% 18% 21%
Mathematics and science 4% 5% 5%
English + mathematics and/or science 2% 13% 10%
Other 1% 3% 2%

Which grades do you normally teach? Male teachers Female teachers Total

Primary Cycle 1 and 2 (Grades 16) 8% 47% 37%

Intermediate level Cycle 3 (Grades 79) 21% 19% 19%
Secondary level Cycle 4 (Grades 1012) 47% 15% 23%
Primary and intermediate (Cycle 1 and/or 2 and Cycle 3) 4% 11% 9%
Intermediate and secondary (Cycle 3 and 4) 19% 8% 10%
All (Cycle 1 and/or 2, Cycle 3 and 4) 1% 1% 1%

*Percentages may not sum to 100 due to rounding.

**For all questions, more than one response was possible but data was recoded into the most common combinations.

secondary school teachers of either maths or science
(see Table 4). Therefore, any significant findings must be
interpreted cautiously as it is unclear whether one variable
or a combination of variables is contributing to these
differences. Comments left by teachers in the questionnaire
were used to help identify which variable was likely
contributing to differences. Unless otherwise stated, the
use of agree includes responses to strongly agree, and
disagree includes strongly disagree.
Attitude towards English
Attitudes towards a language can influence learner
behaviour in terms of effort expended on learning and
learner success (Csizr and Drnyei 2005, Drnyei
2003, Horwitz 2001). Learner self-efficacy and teacher
beliefs in learners capacity to learn can also influence or
reinforce learner attitudes towards a language (Bandura
1977, Mills, Pajares and Herron 2006, Multon, Brown and
Lent 1991). Therefore, the value placed on learning English
by the teachers and attitudes towards learning English
is an important construct to evaluate. We look in turn at
teachers perceptions of the need to learn English in general
and for their work, their confidence in learning English and
their attitude towards the D-RASATI 2 project.
The importance of studying English
The value placed on an activity such as language learning
is most strongly associated with the willingness to engage

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
80|ISSUE 65 / FEBRUARY 2017
in an activity and persist with it even when it becomes
difficult (Csizr and Drnyei 2005, Wigfield and Eccles
2000). The vast majority of teachers strongly agreed that
it is important to learn English (see Figure 2), with only 3%
disagreeing with this statement.
Figure 2: Teachers: Learning English is important to me
2% 1%
Strongly agree
42% Agree
55% Disagree
Strongly disagree
In the questionnaires, several teachers left comments about
the value of learning English for themselves and for their
students, for example: Since the main common language in
the world is English, and nowadays we are interacting with
multinationalities so it is a must that we should improve
English language.
As a key aspect of the programme is to improve English
language levels of teachers in order to improve student
learning outcomes, we were interested in determining the
extent to which teachers felt the need to improve their
English for their job. Over 90% of respondents agreed
with this statement, with responses evenly split between
strongly agree and agree. Teachers comments in the
questionnaire reinforce this viewpoint:
Teaching in English is necessary especially for those who
teach subjects in English or the English language since
students are going to have their certificate in the official
exams in English in these subjects not in Arabic.
Teaching in English is a must in the official schools.
Students should use the English language not only in the
English periods.
Further analysis of responses to both statements above
by subject specialism and gender resulted in statistically
significant differences. English teachers and females
were more likely to strongly agree that learning English is
important to them and that it is important to their job, than
teachers of maths and science, and males (r < .001) (see
Figure 3).
Although these variables are confounded in the data, it
is not surprising that English teachers are more likely to
recognise the importance of the language for their job
than teachers of other subjects. However, some of the
comments left by mathematics teachers suggest that they
do not recognise the role of language in maths education,
which may be a cause for concern:
UCLES 2017
CAMBRIDGE ENGLISH : RESEARCH NOTES
Figure 3: The importance of English learning by gender and
subject specialism (strongly agree)
80%
70%
60%
50%
40%
30%
20%
10%
0%
Male Female Maths Science English
Gender Subject
Learning English is important to me
It is important for my job I improve my English
As a math teacher, I dont need all these English hours.
For Mathematics, English is not so important because
there are few words to use.
I dont need high level in English language because Im
teaching mathematics and chemistry that dont need so
much English skills.
A common misconception is that maths is the domain
of numbers and not language but language is crucial
in the development of new knowledge, whether that
is mathematical knowledge or other knowledge,
because language is the tool learners use to think
about a subject and talk about their thinking (Barwell
no date). Without strong English language skills,
learners learning mathematics through the medium
of English will be limited to the lower-level cognitive
processes of memorising rules and processes rather
than engaging in higher-level processing such as
mathematical thinking which encourages learners
to pose and solve mathematical problems (Devlin
2012). It should be noted that several of the maths
teachersdidmakethisexactpointbutitmaybethat they
are in the minority:
Mathematics contains complex problems especially in
probability that need to know English very well. So I insist
in teaching my students in English.
Its essential to learn English well, especially for problem
solving in math.
It is important for the MEHE to combat the view that
language does not play a role in maths and science
instruction. The teachers who already recognise the
importance of language in these subject areas may be a
useful resource in changing the minds of their colleagues.
Confidence in English language learning
Teachers confidence in their ability to learn English was
also investigated in the questionnaires as this can influence
their level of engagement and learning outcomes (Horwitz
2001, Mills et al 2006, Multon et al 1991). Figure 4 shows
that teachers agreed that they are confident in their
capacity to learn English (96%); however, the ELTs were
more likely to strongly agree (68%) than the teachers
(40%) (r < .001).
IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|81

Figure 4: Confidence in teachers capacity to learn English

according to teachers and ELTs
Comments left by ELTs support the view that teachers were
pleased with the project:
80%
Teachers
The teachers have shown willingness and enthusiasm to
70% ELT
learn.
60%
My teachers were eager to learn and improve their English
50%
skills.
40%
The majority of the teachers I taught seemed to be
30% pleased about learning.
20%
However, in contrast to these positive responses, ELTs did
10% also note that some participants had conflicting attitudes
0% about learning English that arose from the fact that the
Strongly disagree Disagree Agree Strongly agree courses were mandatory. The following quote from an ELT
is an example of this:
Again, significant differences were found when comparing
... after they started the training course they became
responses by gender and subject specialism, with females
more motivated. But the major problem is that they felt
and English teachers more likely to strongly agree that obliged to take the training.
they are confident in their English language learning
ability compared to males or teachers of maths or science During the interviews with key focal persons, the issue
(r<.001) (see Figure 5). of the mandatory nature of the courses and teachers
reaction to this was discussed. It was noted that although
Figure 5: Confidence in ability to learn English by gender and
subject specialism (strongly agree)
there were no serious penalties for not attending or not
doing well on the language courses, some teachers were
60% concerned at the beginning about potentially negative
I am confident that I have the
50% capacity to learn English consequences. However, these views seemed to abate as
40% the courses progressed. In some cases, there was very little
time between when teachers were invited to attend the
30%
English courses and the start of them, which may not have
20%
given teachers enough time to digest and come to terms
10% with the MEHE request.
0%
Male Female Math Science English
General notes on teacher attitudes
Gender Subject Teachers generally reported positive attitudes towards
English learning and recognised that English is important
Attitude towards the D-RASATI 2 project to their job and belief in their own capacity to learn English,
Both teachers (90%) and ELTs (91%) agreed that course which was also evidenced by the ELTs. However, the
participants were pleased to be able to study English as data suggests that teachers of maths and science, who
part of this project (see Figure 6). However, maths and also tended to be male, were less positive about learning
science teachers and males were less likely to strongly English and did not see the relevance of English to maths
agree with this statement (r < .001), which is likely instruction in particular. There is a common misconception
linked to the point raised above; the male teachers are that maths instruction does not require language skills
predominantly maths teachers, who do not necessarily see because of the focus on numbers but research suggests
the relevance of English for their work. that language ability does play an important part in
learning maths (Linneweber-Lammerskitten 2012).
Figure 6: Teachers are pleased to be able to study English as part
of the D-RASATI 2 project
Indeed, one of the key focal persons pointed out that the
underperformance of Lebanese learners on international
80% maths benchmarking tests was attributed to language
Teachers
70% Teacher trainers ability rather than maths ability. Therefore, maths teachers
who are teaching through the medium of English may
60%
need further information on the importance of language in
50% learning maths in order to change their attitudes towards
40% English language learning.

30% Although there was general consensus that teachers were

pleased to be studying English as part of the D-RASATI 2
20%
project, concerns were raised about the mandatory nature of
10% the courses. Some of the ambivalence which was expressed
0% towards the courses may have arisen from a lack of timely
Strongly disagree Disagree Agree Strongly agree information about them which, in some cases, involved

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
82|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Figure 7: Teacher attitudes to course scheduling

teachers being invited to attend with very little notice.
In addition, teachers may have had fears about possible 80%
Strongly disagree Disagree
negative consequences if they did not do well. As with 70% Agree Strongly agree
any large-scale reform initiative, there will always be some 60%
unease generated because of uncertainty or feelings that 50%
control over their worklife balance is being lost. However,
40%
the success of such an initiative will depend greatly on
30%
teachers willingness and ability to implement or mediate
change (Fullan 2001). It is important, therefore, that teachers 20%

understand why they are being asked to improve their 10%

English language skills (particularly for maths and science 0%

teachers), what, if any, consequences there are if they are The English classes were The English classes were The length of each class
offered on convenient offered at convenient period (i.e. 3 hours) was
unable to do so, and the benefits of making these changes. days of the week times of the day appropriate for learning to
take place

Course implementation
The session hours should be part of my employment
A key aspect of the project was to provide training to
hours (not extra hours).
teachers across Lebanon ensuring equal access to quality
English language instruction. D-RASATI 2, thus, had the Trainers also noted that the scheduling of the courses may
challenge of meeting the needs of over 2,500 teachers have affected teachers ability to fully engage in the content.
located throughout the country. In response to this, They commented that teachers:
D-RASATI 2 offered 3-hour courses held twice a week
would like to study but are often overwhelmed by work
and during the summer these courses were four days per
and personal responsibilities
week in 17 locations. A major logistical challenge was the
scheduling of these courses. Because the courses were found that their numerous responsibilities prevented
mandatory, it was important to offer lessons at convenient them from taking advantage of the course to the full.
times and locations which were accessible for most
The key focal persons were aware of this issue and noted
teachers. Therefore, teachers were asked about their views
that D-RASATI 2, under the guidance of MEHE, changed
on the course placement, scheduling and attendance.
course times in some cases so that only 50% of the course
was outside school hours. Although teachers seemed to
Course placement
indicate a preference for morning lessons, it may have
As mentioned previously, over 4,000 permanent and
been difficult for school principals to find replacements to
contractual public sector teachers were assessed using
cover lessons while the teachers were taking the language
BULATS in order to determine training needs for teachers
courses. As the majority of teachers in this project were
teaching English as a first or second language or using
female, course scheduling is an important consideration
English as the medium of instruction for science and maths.
because females still tend to have primary responsibility for
The test data was then used to place teachers in the English
childcare in Lebanon and there are limited public facilities
language courses. Teachers and ELTs were asked about
or subsidies to assist in finding alternative childcare (World
the extent to which they felt they were placed into the
Economic Forum 2013).
appropriate level and 87% of teachers and 93% of ELTs felt
that placement was accurate.
Course attendance and homework
Despite the fact that teachers commented that they were
Course scheduling
unhappy with the courses being scheduled during non-work
As can be seen from Figure 7, the majority of teachers
hours, the majority of teachers agreed that they attended
generally agreed that the language courses were offered on
their courses regularly, arrived on time and completed their
convenient days of the week, times of the day and that the
homework (see Figure 8).
length of each class period was appropriate.
Figure 8: Attendance and homework completion according to
Course scheduling, however, elicited the largest number of teachers
negative comments from teachers, particularly the question
concerning the timing of the course with 35% of teachers 60%

disagreeing that it was convenient. The main complaints 50%

were related to issues of childcare, having to study after a 40%
long day of teaching and courses taking place during non-
30%
work hours. The following comments are reflective of the
vast majority of comments left: 20%

10%
Because we have teaching hours in the morning and we
come tired to class. Also, we have other responsibilities 0%
in life. I attended regularly I arrived on time I completed homework
tasks set
The government didnt reduce our teaching hours.
Strongly disagree Disagree Agree Strongly agree

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
CAMBRIDGE ENGLISH : RESEARCH NOTES
However, some teachers did comment that it was a
challenge to complete homework and practise what they
learned, which again was related to course scheduling
taking place partially during non-work hours:
... to improve my language skills, I needed time to review
the tasks and the information given during the course, but
unfortunately I did not have time to do that, which stood
as an obstacle.
The ministry of education should give less teaching hours
for all the teachers who are attending these courses. We
enjoyed the course a lot but there was not enough time to
study and practice.
Male teachers and science teachers were more likely
to disagree that they attended regularly (r < .005) and
arrived on time (r < .001) than female teachers and English
teachers. This is not particularly surprising considering
they had previously indicated that they did not understand
the relevance of the course to their work and were more
likely to indicate that they were less pleased about studying
English as part of this project.
Course objectives and learning materials
Teachers agreed that they understood the course
objectives, the coursebook was useful and the course
content was interesting (see Figure 9).
Figure 9: Clarity of course objectives and perceptions of learning
materials
60%
50%
40%
30%
20%
10%
0%
The course objectives The coursebook was The course content
were clear useful was interesting and
met my needs
Strongly disagree Disagree Agree Strongly agree
Male teachers and those who teach maths and science
were more likely to disagree with these statements (r <
.001). Comments left in the questionnaire indicate that this
may be related to their interest in receiving subject-specific
instruction on teaching through the medium of English
rather than lessons which focus on language development:
I think for me, as a math teacher, I should take some
English course about mathematics and about teaching
Mathmatic.
The grammar and vocabulary I studied in this course do
not help me in the teaching science in fact, they are not
related to the topics, we teach in class.
General notes on attitudes towards the course
Teachers generally agreed that they were placed into the
appropriate level and the course content and learning
UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
ISSUE 65 / FEBRUARY 2017|83
materials were appropriate. They were less positive
about the scheduling of the courses, in particular the
timing of the courses during the day. Many teachers left
comments indicating that having English classes outside
of work hours disrupted their personal lives; in particular
it caused problems for childcare and conflicted with other
work obligations. Some teachers and ELTs suggested
that teachers were not able to fully engage in the lessons
because they were tired from working all day and had little
time to do any self-study at home.
With a large female teaching population, the issue of course
scheduling is an important consideration as this segment
of the population often has increased home responsibilities.
Any future courses should ensure that the target population
is consulted as far in advance as possible so that there
are opportunities to address any concerns raised and
there is enough time for participants to make alternative
arrangements for childcare. Also, giving teachers some
element of choice might reduce negative perceptions of the
course caused by its mandatory nature.
Lesson effectiveness and teacher engagement
This section investigates the extent to which teachers
were engaged in their lessons and their perceptions of
the effectiveness of their lessons. We will start by looking
at indicators related to engagement such as attendance,
completion of homework and participation in lessons. Then,
we will report on teachers perceptions of the quality of
lessons and effectiveness of their ELTs.
Engagement in lessons
In order to determine teachers level of engagement, they
were asked about how involved they were in their learning,
the extent to which they took advantage of learning
opportunities, enjoyed their lessons and found the lesson
activities useful. Figure 10 provides an overview of their
responses. Although teachers tended to agree with the
statements in Figure 10, male teachers and maths teachers
were more likely to disagree with them than female
teachers and English teachers (r < .001).
Figure 10: Teachers level of engagement in lessons
70%
60%
50%
40%
30%
20%
10%
0%
I felt involved in my I used the I enjoyed the The tasks/activities
learning opportunities I was English language used in class were
given to practise training course I useful for improving
the language attended my English
Strongly disagree Disagree Agree Strongly agree
The profile of teachers within the courses was raised by
a small number of participants in the questionnaire open
84|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

fields. Some teachers felt that having a mixed group in features such as type of homework or assessments given,
terms of subject specialism within a course may have the grading of assignments or the focus of the lesson. They
limited learning or demotivated some teachers, as the were not easily grouped into general categories.
following comments indicate:
Course delivery: Interactional patterns
It would be useful whenever teachers are classified with
The language courses were based on a communicative
regard to their certificates and diplomas, as well as the
cycles they are teaching.
approach to language learning because it is a widely
accepted premise both in the theoretical and practical
Some activities were very easy and didnt add anything
language learning domains that exposure to a foreign
to me. It was targeting teachers of other subjects like
Math...
language within the learning environment plays a positive
role in learning outcomes. The European Survey on Language
It could be improved if English teachers were placed alone
Competences (ESLC) found that greater use of English, by
in the training session. English teachers got bored waiting
for the other Science teachers to catch up.
both students and teachers, in the classroom was positively
related to language ability (Jones 2013). Therefore, a number
Comments left by trainers, however, both supported and of statements were included in the questionnaires to
refuted this point of view making it difficult to draw any investigate the extent to which learners had opportunities to
concrete conclusions. In future projects of this nature, use English with others and were willing to take advantage
consideration of the participant profile within classes may of these opportunities. Figure 12 shows teacher (T) and ELT
need to be explored further. responses to statements about the frequency of different
interactional patterns in the classroom and teachers
Course delivery: Teaching practices willingness to speak to others in English.
Teachers were given a series of statements about the
Figure 12: How often did the following happen during English
ELTs teaching practices. These statements were designed lessons?
to evaluate the effectiveness of the course delivery and
100%
the quality of teaching. Figure 11 provides an overview of 90%
teacher responses. 80%
70%
Figure 11: Teachers perception of course delivery 60%
50%
40%
Strongly disagree Disagree Agree Strongly agree 30%
THE ELT... 20%
80%
10%
70% 0%
T ELT T ELT T ELT T ELT
60%
50% I worked I worked I worked I spoke
individually in pairs in groups to classmates
40%
in English
30%
Very often Sometimes Not very often Never
20%
10%
0% As can be seen in Figure 12, teachers reported working in
pairs more often than their ELTs whereas ELTs reported
ar ell

el d

w in

nt t

an d

on o

lo ng
de or

ss t
sw e

st di
ep w

bl ons

do sti
as ag
ed

ay

le y
s

gy
tu pp

er f I

e log
pr as

cl an

ho re
r s ra

nd e i

that teachers spoke to others in English more often than

oy ss

th o
w

et e
e
th m

tu m

e n
nj le

he od

m int

h
a
d

no ed

nc ec
s/ o

an
e

hi a g
e

they reported for themselves (r < .001). ELTs indicated

an nt

lp

ha d t

ed
he
e

ith d

en use
e
es

w ishe

us

pr

that teachers primarily worked individually or in groups.

bl

ta
es

Comments left by teachers were split between those who

felt that they had opportunities to speak both to colleagues

There was general consensus that course delivery was
effective and teachers agreed that their ELTs were skilled
teachers. Teacher comments about their ELTs reinforce this
perspective:
The trainer was talented as he used a diversity of teaching
techniques and he was very patient with us.
The preparation for the workshop was perfect by the
trainer, and the explanation was clearly done.
[The course was] enriching, motivating, presented with
high professionalism.
You made me love English.
Negative comments about ELTs tended to be very disparate
in nature in that they focused on individual course-specific
and native speakers (i.e. the ELTs) and others who felt
that there needed to be more opportunities to speak
withothers:
I was encouraged to speak English since my teacher is
foreigner as well as my class mates liked to communicate
in English.
I used English communicate with my friends and have the
opportunity to listen to a native speaker.
I think the main problem in learning English is interaction
and speaking English. I think this course or maybe this
level need more time to improve our speaking.
Turning now to whether teachers enjoyed working with
others in their lessons, it is clear from Figure 13 that they
did. One of the key focal persons pointed out that an

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
CAMBRIDGE ENGLISH : RESEARCH NOTES ISSUE 65 / FEBRUARY 2017|85

unintended positive outcome of this project was the fact
that teachers from different schools who may not normally
come in contact with each other were in the same class,
which allowed them to share their teaching experiences.
This interaction can create informal opportunities to
learn new ideas or methods of teaching as well as build a
supportive collegial network.
Figure 13: Teachers: I enjoyed working with other people in my
class
1%
The majority of teachers agreed that the graded
assignments they were given were useful (95%) and that
they understood the criteria used to assess them (96%)
(see Figure 15). There were mixed views on whether they
were motivated to improve by the prospect of receiving
a Cambridge English certificate, with English language
teachers and those at higher CEFR levels being more
interested in receiving a certificate:
It would be so sorrowful if we will not have the
opportunity to sit for the CAE exam.

5% Figure 15: Impact of assessment

90%
80%
70%
Strongly agree
60%
Agree
40% 54% 50%
Disagree
40%
Strongly disagree
30%
20%
10%
0%
The graded assignments I understood the grading I am motivated to improve
were useful for improving criteria used to evaluate my English by the prospect
my English my performance of receiving a Cambridge
English exam certificate
Course delivery: Assessment
Not sure Strongly disagree Disagree Agree Strongly agree
Assessment has multiple functions in education, including
the measurement of achievement and the provision of
feedback to learners on their strengths and weaknesses. General notes on lesson effectiveness and teacher
When assessment practices are linked closely with engagement
learning goals, it has the ability to both motivate learners Teachers appeared to be engaged in their lessons by
and improve learning (Carless 2007). In general, attending regularly, completing homework tasks and
teachers agreed that the assessment activities used were participating in lessons; however, they did indicate that had
appropriate, at the right level and in line with lesson aims the courses been during work hours, they may have been
(see Figure 14). However, they agreed less strongly with able to take more advantage of the lessons because they
these statements than their ELTs (r < .001). This is perhaps would have had more time to study. Teachers reported
not surprising because the ELTs were likely involved in the that the lesson activities were useful and that the ELTs
development of some of the assessment tools used within utilised effective teaching practices. Teachers were given
their courses. In the questionnaires, some teachers stated opportunities to use English in the class with others and
that they were not happy with the grades they received or seemed to make use of these opportunities. An important
the number of tests given: outcome of the communicative approach taken in lessons
is that teachers were given opportunities to interact with
The teacher was very stingy in putting marks.
teachers from other areas, which allowed them to share
I think there were too much Exams more then needed. teaching experiences and understand how teaching is
structured in other schools.
Figure 14: Appropriacy of assessment
The tests/exams
Teachers agreed that the assessment practices were
100% effective in that they were useful in helping them improve
90% their English. Teachers of English and teachers with a higher
80% level of English were motivated by the prospect of receiving
70% a Cambridge English certificate.
60%
50%
40%
Impact of courses
30%
A main aim of the D-RASATI 2 project is to improve
20% English language levels of both teachers and learners. This
10% section will focus on the extent to which teachers felt their
0% language levels improved and the effect of the course on
T ELT T ELT T ELT
used during the course used during the course were in line with the their pedagogical practice.
assessed an appropriate were at the right level focus of lessons
range of knowledge (e.g. not too easy
and skills or too hard)

Strongly agree Agree Disagree Strongly disagree

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
86|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Figure 17: Impact of course on teaching practices and confidence

Skill development in English
Teachers and ELTs were asked whether they thought 80%

teachers English had improved during the course. Figure 70%

16 provides an overview of teacher and ELT responses. 60%
As can be seen, both teachers and ELTs reported that the 50%
vocabulary and grammar of teachers improved the most
40%
while their listening ability improved the least compared to
30%
the other skills and systems. Listening was the skill which
was identified as the weakest for those teachers who took 20%

the Cambridge English: First exam, while speaking was the 10%
strongest. In the comments left by teachers, they indicated 0%
that listening was an area in need of further development, The training has had a Before taking this English After taking this English
positive effect on my language course, I was language course, I feel
and some teachers mentioned problems with the acoustics teaching at school confident teaching more confident about
in English teaching in English
of the room and difficulty with the different accents on
recordings, which teachers felt limited their ability to Strongly agree Disagree Agree Strongly disagree

improve their listening. The following comments reflect

chance to observe professional teachers so that we can
these views:
gain a lot of ways of teaching.
We need more listening exercise and to listen to English
Teachers also agreed that they feel more confident
accent.
teaching in English as a result of taking this course (93%).
The course is a great aid concerning reading, writing,
Finally, 72%2 of teachers reported that they would like to
grammar and vocabulary where my skills have improved a
lot in all these domains. But in listening I still need much
continue taking more English language training courses.
more practice. This is a positive outcome of the project and an indication
that despite the scheduling issues and some views that the
Figure 16: Perceived improvement as a result of taking the course lessons were not relevant for particular teaching domains, a
(teachers and ELTs) large number of teachers would like to continue studying.
100%
90% General notes on the impact of courses
80% The language courses appear to have had a positive impact
70% on both teachers language skills and their pedagogical
60% practices. Although slightly fewer teachers achieved the
50%
Cambridge English: First certificate than targeted, teachers
40%
reported feeling the courses helped them improve their
30%
English, in particular their vocabulary and grammar.
20%
Teachers felt the courses had the least effect on their
10%
listening skills and the exam data suggests this is an area in
0%
T ELT T ELT T ELT T ELT T ELT T ELT need of increased focus in any future courses.
Vocabulary Grammar Reading Writing Speaking Listening
One clear impact of the initiative is that the pedagogical
A lot A moderate amount Only a little bit Not at all Not sure
practices of the ELTs were adopted by many of the
teachers, thus potentially improving the language learning
Impact of course on teaching practices experiences of their pupils. Teachers report that their
A significant impact of the language training courses was confidence in teaching English has improved and that they
the extent to which they positively influenced teachers would like to continue studying English, which are also
own pedagogical practices in their classrooms. As Figure positive outcomes.
17 shows, the teachers agreed that the courses have had
a positive effect on their teaching (90%) with English
teachers more likely to strongly agree (30%) than all other Lessons learned
groups (20% for maths and 21% for science) (r < .005).
The D-RASATI 2 project was an ambitious large-scale
Teachers themselves commented about the positive project aimed at improving teaching and learning in
influence their ELTs have had on their own teaching as the Lebanon. As with any large-scale reform initiative, it
following comments illustrate: can take time for all stakeholder groups to become
enthusiastic about engaging in change and for any impact
It [the course] was usefull and it improved my Eng
on learner outcomes to become visible. The following
language and provided us with some new technique to
apply with our students.
section summarises the successes of this project and
recommendations for the future.
This course has helped me a lot to apply new methods
of teaching in my class. What I would like is to get the

This question was added to the questionnaire during the last cycle of classes so it was only responded to by 435 teachers.
2

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
CAMBRIDGE ENGLISH : RESEARCH NOTES
Success stories
Over 2,000 teachers were given English language
lessons over half of all those identified as in need of
English language upskilling.
Many teachers were pleased to be involved in this
project and would like to continue studying English.
Teachers reported that the courses were well d esigned
and the ELTs were effective and skilled teachers.
Teachers in rural or remote locations throughout the
country were given access to English lessons, which was
an important aim of this project.
Teachers reported adopting the teaching practices
of their ELTs, which were based on a communicative
methodology, which may improve their own
instructional quality.
Teachers reported feeling more confident teaching in
English as a result of these language courses.
The courses provided an opportunity for teachers from
different schools to work together and learn from each
other.
Recommendations
Communicate with stakeholders as early as possible
about an initiative and its benefits in order to allow
all those involved time to understand the purpose of
the project and make any necessary preparations in
advance.
Consider offering some alternative course delivery
methods (e.g. blended learning, online delivery) which
may reduce issues around scheduling courses outside
work hours; however, this may reduce the contact
between teachers which was a positive feature of this
project.
Prioritise listening in future courses aimed at a similar
cohort as this was identified as a weaker skill.
Actively combat misconceptions about the role of
language in maths and science education to ensure that
these teachers recognise the usefulness of language
courses.
Consider adding a methodological component to future
courses particularly for maths and science teachers,
which may strengthen the link between these courses
and teachers own classes.
Overall, for English teachers, the courses were relevant
for both their lesson content and teaching methods. The
comments left by maths and science teachers suggest
that they did not necessarily see the relevance of either
the content or methods but they did seem to indicate that
they would like to take courses which focused on teaching
maths or sciences through the medium of English. Perhaps
by explicitly adding a methodological component to the
courses, both groups may benefit even more; for example,
UCLES 2017
ISSUE 65 / FEBRUARY 2017|87
two lessons a week on English language learning and one
lesson a week on methods to teach in English. It is clear
that greater promotion and understanding of methodology
amongst teachers is required.
References
American University of Beirut (2012) D-RASATI Project English
Language Course, Request for Proposal.
Bandura, A (1977) Self-efficacy: Toward a unifying theory of
behavioral change, Psychological Review 84 (2), 191215.
Barwell, R (no date) The role of language in mathematics, National
Association for Language Development in the Curriculum, available
online: www.naldic.org.uk/Resources/NALDIC/docs/resources/
documents/The%20Role%20of%20language%20in%20
mathematics.pdf
Braun, V and Clarke, V (2006) Using thematic analysis
inpsychology, Qualitative Research in Psychology 3 (2), 77101.
Carless, D (2007) Learning-oriented assessment: Conceptual basis
and practical implications, Innovations in Education and Teaching
International 44 (1), 5766.
Council of Europe (2001) Common European Framework of
Reference for Languages: Learning, Teaching, Assessment, Cambridge:
Cambridge University Press.
Creswell, J W and Plano Clark, V L (2011) Designing and Conducting
Mixed Methods Research (2nd edition), Thousand Oaks: Sage.
Csizr, K and Drnyei, Z (2005) The internal structure of language
learning motivation and its relationship with language choice and
learning effort, The Modern Language Journal 89 (1), 1936.
Devlin, K (2012) Introduction to Mathematical Thinking, Petaluma:
Keith Devlin.
Drnyei, Z (2003) Attitudes, orientations, and motivations in
language learning: Advances in theory, research, and applications,
Language Learning 53 (1), 332.
Fullan, M (2001) The New Meaning of Educational Change (3rd
edition), New York: Teachers College, Columbia University.
Horwitz, E (2001) Language anxiety and achievement, Annual
Review of Applied Linguistics 21, 112126.
Jones, N (2013) The European Survey on Language Competences
and its significance for Cambridge English Language Assessment,
Research Notes 52, 27.
Linneweber-Lammerskitten, H (2012) Items for a description of
linguistic competence in the language of schooling necessary for
learning/teaching mathematics (at the end of compulsory education).
An approach with reference points, Language Policy Unit, Directorate
of Education, DGII, Council of Europe.
Loulidi, R (1990) Is language learning really a female business?,
Language Learning Journal 1, 4043.
Mills, N, Pajares, F and Herron, C (2006) A re-evaluation of the
role of anxiety: Self-efficacy, anxiety and their relation to reading
and listening proficiency, Foreign Language Annals 39, 276295.
IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
88|ISSUE 65 / FEBRUARY 2017 CAMBRIDGE ENGLISH : RESEARCH NOTES

Multon, K D, Brown, S D and Lent, R W (1991) Relation of World Economic Forum (2013) The global gender gap report,
self-efficacy beliefs to academic outcomes: A meta-analytic available online: www3.weforum.org/docs/WEF_GenderGap_
investigation, Journal of Counseling Psychology 38, 3038. Report_2013.pdf

Wigfield, A and Eccles, J S (2000) Expectancy-value theory of

achievement motivation, Contemporary Educational Psychology 25,
6881.

UCLES 2017 IMPROVING STUDENT LEARNING THROUGH UPSKILLING TEACHERS: THE CASE OF LEBANON
Studies in Language Testing
An indispensable
resource for anyone
interested in new
developments
and research in
language testing

To find out more about our

full list of publications:
www.cambridge.org/elt/silt
www.cambridgeenglish.org/silt
 To subscribe to Research Notes and download previous issues, please visit:
www.cambridgeenglish.org/research-notes

Contents
Editorial2

Understanding language learning in Malta 3

Nahal Khabbazbashi, Coreen Docherty and Christine Walker

An investigation of Portuguese students attitudes to assessment and Cambridge English exams 24

Jane Lloyd

Assessing the English language progress of students in a trilingual education framework in the
Basque Country 39
Belinda Cerda, Andrew Blackhurst and Christine Walker

Which factors affect English language attainment? A study of school students in Chile 51
Agnieszka Walczak, Coreen Docherty and Graeme Harrison

The impact of the Antioquia English in the Park initiative on language teaching and practice 65
Jane Lloyd and Coreen Docherty

Improving student learning through upskilling teachers: The case of Lebanon75

Coreen Docherty, Tania Barakat, Elaine Kniveton and Lama Mikati

For further information visit the website:

www.cambridgeenglish.org

Cambridge English
Language Assessment
1 Hills Road
Cambridge
CB1 2EU
United Kingdom

www.cambridgeenglish.org/helpdesk

UCLES 2017 CE/4561/7Y02

*3606283952*

All details are correct at the time of going to print in February 2017.
Position artboard at base of page - size
alwaysatUCLES
100%, no smaller.
2017 thisBSIpublication
logo oval may not be reproduced without the written permission of the copyright holder
9mm height min. Baseline is 12mm from
base of page, and 99mm wide for DLs