Full

PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI
THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

WHEN ENGLISH RINGS THE BELL: A SURVEY RESEARCH
A THESIS
A Thesis Presented to
The Graduate Program in English Language Studies
in Partial Fulfillment of the Requirements
for the Degree of Magister Humaniora (M.Hum.)
in English Language Studies
by
Teguh Ariebowo
116332027
THE GRADUATE PROGRAM OF ENGLISH LANGUAGE STUDIES
SANATA DHARMA UNIVERSITY
YOGYAKARTA
February, 2017
i
ATHESIS - '
THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

WHEN ENGLISH
--- RINGS
-- THE BELL: A SURVEY RESEARCH
-----~
Presented hy, -
Member
-~~
_; =1
-=- -::: l
~- - =-1
-- -j
=~~
- :_:
------
"
-------- -
·-~
v
ACKNOWLEDGEMENTS
In the name of Allah The Most Gracious and The Most Merciful. Praise and
thanks are due to Allah, who has given me strength to complete this thesis. I also
wish to express my gratitude to the following for their contribution throughout the
completion of this study.
I owe an enormous debt of gratitude to F.X. Mukarto, Ph.D., my supervisor,

for all his patience, continuous supervision, invaluable guidance, encouragement,
help, and feedback all the way through the processes of writing this thesis. My
gratitude also goes to Dr. J. Bismoko and Dr. B.B. Dwijatmoko for the inspiration,
feedback, and suggestion so that I am able to revise this work and make it better. I do
not forget to say thanks to all the lecturers and staffs in the Graduate Program of
English Studies, Sanata Dharma University, Yogyakarta for the knowledge and skills
that I have got throughout the undertaking of my graduate study.
I am also sincerely grateful for having wonderful experience with all English
teachers who enthusiastically participated as respondents of this research. Without
their contribution and participation, this research would have not be completed. Those
supportive and cooperative teachers are from SMP N 1 Sleman, SMP N 1 Galur, and
SMP N 1 Gunung Kidul.
I should also thank to my first English teacher in an English course, Ms. Erni
Dhukun and Ms. Pipin for introducing English to me in an interesting and fun way.
The gratitude is also addressed to Ibu Henny Herawati who gave support and
involved me in fruitful discussion about teaching which was able to enlighten me in
finishing this work.
Finally my special thanks are due to my beloved parents, my lovely wife and
my siblings. Thank you for giving me unstoppable support to accomplish this
mission. Last but not least, great thanks also go to people who I impossibly mention
those people one by one. Thank you for invaluable source of support and inspiration
for my over years studying here.
vi
TABLE OF CONTENT
STATEMENT OF THE ORIGINALITY .................................................................... iv

LEMBAR PERNYATAAN PERSETUJUAN PUBLIKASI KARYA ILMIAH
UNTUK KEPENTINGAN AKADEMIS ..................................................................... v
ACKNOWLEDGEMENTS ......................................................................................... vi
TABLE OF CONTENT .............................................................................................. vii
LIST OF FIGURES ..................................................................................................... ix
LIST OF TABLES ........................................................................................................ x
LIST OF ABBREVIATIONS ...................................................................................... xi
LIST OF APPENDICES ............................................................................................. xii
ABSTRACT ...............................................................................................................xiii
ABSTRAK ................................................................................................................. xiv
CHAPTER I INTRODUCTION ................................................................................... 1
A. Background ........................................................................................................ 1
B. Problem Identification and Limitation ............................................................... 4
C. Problem Formulation ......................................................................................... 5
D. Purpose of Study ................................................................................................ 5
E. Benefits of the Study .......................................................................................... 5
CHAPTER II REVIEW OF LITERATURE................................................................. 7
A. Theoretical Review ............................................................................................ 7
1. Textbooks ....................................................................................................... 7
2. Evaluation..................................................................................................... 16
3. Review on Indonesia National Curriculum 2013 ......................................... 23
4. Review of Previous Studies.......................................................................... 26
B. Theoretical Framework .................................................................................... 28
CHAPTER III RESEARCH METHOD ..................................................................... 30
A. Research Method ................................................................................................ 30
B. Research Respondents ........................................................................................ 31
C. Research Instrument ........................................................................................... 32
vii
D. Data Gathering Techniques ................................................................................ 32

E. Data Analysis ...................................................................................................... 33
CHAPTER IV RESEARCH FINDINGS AND DICUSSION.................................... 35
A. General Findings ................................................................................................ 35
B. Detail Findings.................................................................................................... 37
1. The Aim of the Textbook ............................................................................. 37
2. The Approach of the Textbook .................................................................... 39
3. Design and Organization of the Textbook ................................................... 46
4. Content of the Textbook ............................................................................... 51
5. Language Skill Presented in the Textbook ................................................... 54
6. Topic Presented in the textbook ................................................................... 56
C. Discussion ........................................................................................................... 59
1. The Aim of the Textbook ............................................................................. 59
2. The Approach of the Textbook .................................................................... 61
3. The Design and Organization of the Textbook ............................................ 62
4. Language Content of the Textbook .............................................................. 66
5. Language Skill Presented in the Textbook ................................................... 66
6. The Topic Presented in the Textbook........................................................... 67
CHAPTER V CONCLUSION .................................................................................... 70
A. CONCLUSION ................................................................................................ 70
B. Suggestions ...................................................................................................... 73
Bibliography................................................................................................................ 75
Appendices .................................................................................................................. 78
viii
LIST OF FIGURES
Figure 2.1 The Theoretical Framework ..................................................................... 29

Figure 4.1 Sample of activity found in textbook: Activity 5 ..................................... 41
Figure 4.4 Aim of Chapter 1 ...................................................................................... 48
Figure 4.5 Sample of activity found in textbook ....................................................... 53
Figure 4.6 Sample of a simple lyric and a very simple dialog ................................... 56
Figure 4.7 The new version of Bloom's taxonomy .................................................... 63
ix
LIST OF TABLES
Table 2.1 Criteria of an Effective Textbook .............................................................. 15

Table 2.2 Checklist Suggested by Rea-Dickins and Germain ................................... 22
Table 4.1 Overall Survey Result ................................................................................ 36
Table 4.2 Survey Result on the Aims of the Textbook ............................................. 38
Table 4.3 Survey Result on the Approach of the Textbook ....................................... 40
Table 4.4 Survey Result on the Design and Organization of the Textbook............... 48
Table 4.5 Complexity Level of the Activity .............................................................. 51
Table 4.6 Complexity Level and Number of Activity ............................................... 52
Table 4.7 Survey Result on the Content of the Textbook .......................................... 53
Table 4.8 Survey Result on the Skills Presented in the Textbook ............................. 56
Table 4.9 Survey Result on the Topic Presented in the Textbook ............................. 58
x
LIST OF ABBREVIATIONS
C13 Curriculum 2013

TEFL Teaching English as Foreign Language
EFL English as a Foreign Language
WERB When English Rings the Bell
xi
LIST OF APPENDICES
Appendix 1: Questionnaire Draft 1 79

Appendix 2: Questionnaire Draft 2 90
Appendix 3: The Blueprint of the Questionnaire 98
Appendix 4: The Respondents’ on Statement 106
Appendix 5: The Respondents’ Answer on Open Question Questionnaire 110
Appendix 6: Translated transcript of the interview 115
xii
ABSTRACT
Teguh Ariebowo. 2016. The Evaluation of Seven Grade English Textbook “When
English Rings the Bell”: A Survey Research. Yogyakarta: English Language
Studies. Graduate Program Sanata Dharma University.
As the process of curriculum development, Indonesia developed a new national

curriculum in 2013 which is called Curriculum 2013 (C13). This new curriculum raised a
great discussion and debate among people of Indonesia. The textbooks of any subject which
were made based on the C13 then became problematic since its basic competence has
become tremendous concern of textbooks especially the English textbook. Therefore, the
researcher focuses on the 7th grade textbook of secondary school to be evaluated. In this
evaluation study, the researcher try to find out to what extend textbook meets the criteria.
This study belongs to a descriptive quantitative research. This research took the data
from the survey among the teachers of English using this textbook in their classroom. To be
able to conduct the survey, the researcher developed a questionnaire distributed to 17
teachers as the respondents in the Special Province of Yogyakarta. When this research was
conducted, there were only few schools appointed by the government to implement the C13.
The questionnaire administered the Likert using scale 1 – 4. The statements were developed
from some criteria of effective textbook based on the experts for instance Brian Tomlinson
(2008), David Nunan and Cunningsworth (1995). The criteria include a) the aim, b) the
approach, c) the design and organization, d) the language content, e) the skill and f) topic
presented in the textbook.
The findings showed that some criteria were fulfilled in the textbook but some other
criteria were not. According to the survey result, the aims of textbook met perfectly and
suited with the aim of the C13. The approach implemented in the textbook met the criteria
quite well although some respondents had no idea of what approach they should adopt in this
curriculum. A different finding happened in the design and organization of the textbook. In
some aspect of this criterion, the respondents saw that textbook did not meet the criteria very
well. Different attitude found in responding language content of the textbook. All
respondents did not appreciate the language content. Then respondents saw that the topic
presented in the textbook met the criteria so much well.
In conclusion, the researcher sees that the textbook has some strengths and
weaknesses. The presented topic, for instance, is one of the strengths since it is suitable with
the learners’ level and social condition in Indonesia. The illustration found in the textbook
represents the gender, ethnic, and religion diversity in Indonesia. However, the textbook is
like other books which always have pitfalls. One of the most obvious pitfalls in textbook is
the language content since this book does not have grammar reference and its level of
difficulty way too simple for 7th grade students. Finally, the researcher suggests material
developers to revise this textbook and to English teachers to always conduct evaluation
toward books or new materials before utilizing them in classroom. Hence teachers are able to
choose the most suitable textbook for their learners.
Keywords: Effective textbook, Textbook evaluation, Material Evaluation.
xiii
ABSTRAK
Teguh Ariebowo. 2016. The Evaluation of Seven Grade English Textbook “When
English Rings the Bell”: A Survey Research. Yogyakarta: English Language
Studies. Graduate Program Sanata Dharma University.
Sebagai bentuk proses perkembangan kurikulum, Indonesia mengembangkan

sebuah kurikulum nasional yang baru pada tahun 2013 yang disebut Kurikulum 2013
(K13). Kurikulum baru ini memunculkan diskusi dan perdebatan besar di kalangan
masyarakat Indonesia. Buku-buku teks pelajaran yang disusun berdasarkan K13
kemudian menjadi masalah disebabkan kompetensi dasar yang menjadi kekhawatiran
besar dari buku-buku teks pelajaran khususnya Bahasa Inggris. Dengan demikian,
peneliti berfokus pada buku teks pelajaran Bahasa Inggris untuk siswa kelas 7 SMP
untuk dievaluasi. Dalam kajian evaluasi ini, peneliti mencoba untuk mencari tahu
seberapa jauh buku teks bahasa Inggris tersebut memenuhi kriteria.
Kajian ini digolongkan dalam sebuah penelitian kuantitatif diskriptif.

Penelitian ini mengambil data dari survei yang dilakukan di kalangan guru-guru
Bahasa Inggris yang menggunakan buku teks pelajaran ini di kelas. Untuk
melaksanakan survei ini, peneliti mengembangkan sebuah kuisioner atau angket yang
dibagikan kepada 17 guru di DIY sebagai respondennya. Ketika penelitian ini
dilaksanakan, hanya terdapat beberapa sekolah yang ditunjuk oleh pemerintah untuk
mengimplementasikan K13. Angket tersebut menggunakan skala Likert dari 1 – 4.
Pernyataan-pernyataan dalam angket dikembangkan dari kriteria buku teks yang
efektif yang berdasarkan dari beberapa ahli seperti Brian Tomlinson (2008), David
Nunan (1989) and Cunningsworth (1995). Kriteria tersebut meliputi a) tujuan, b)
pendekatan, c) desain dan organisasi, d) bahasa isi, e) keterampilan dan f) topik yang
disajikan di dalam buku teks.
Hasil dari survei menunjukkan bahwa buku teks tersebut memenuhi beberapa
kritera dan juga beberapa hal lain tidak memenuhi kriteria. Berdasarkan hasil survei,
tujuan buku teks berkesinambungan dengan tujuan K13. Pendekatan yang
diimplementasikan di dalam buku teks cukup memenuhi kriteria walaupun beberapa
guru yang mengisi angket tidak mengerti dan memahami pendekatan apa yang
seharusnya mereka gunakan di dalam kurikulum ini. Berbeda dengan hasil jajak
pendapat mengenai desain dan organisasi buku teks bahwa di beberapa kriteria buku
ini tidak dapat memenuhinya dengan baik. Sangat jauh berbeda dengan hasil jajak
pendapat mengenai bahasa isi dari buku teks tersebut di mana semua guru yang
menjadi responden bahwa mereka bersikap kurang setuju dengan bahasa isi dengan
xiv
buku teks tersebut. Kemudian, di bagian topik, para guru berpendapat bahwa topik
yang disajikan sudah memenuhi kriteria buku dengan baik.
Pada kesimpulannya, peneliti melihat bahwa buku teks terebut memunyai

kelebihan dan kekurangan. Topik yang disajikan, misalnya, merupakan kekuatan dari
buku teks ini mengingat bahwa topik yang disajikan cocok dengan tingkatan
pembelajar dan kondisi sosial masyarakat Indonesia. Ilustrasi yang ditampilkan dalam
buku teks ini memperihatkan keanekaragaman gender, etnis dan agama di Indonesia.
Namun demikian, buku teks ini tidak luput dari kekurangan seperti halnya pada buku
teks lainnya. Salah satu kekurangan buku teks ini yang sangat terlihat jelas adalah
tidak adanya referensi tata bahasa yang bisa dijadikan rujukan oleh pemelajar dan
tingkat kesulitan bahasa yang digunakan dalam buku ini terlalu mudah untuk
pemelajar tingkat 7 SMP. Pada akhirnya, peneliti menyarankan kepada pengembang
materi untuk merevisi buku teks ini. Selain itu paling penting adalah menyarankan
kepada setiap guru Bahasa Inggris untuk melakukan evaluasi terhadap buku-buku
setiap sebelum menggunakan buku-buku tersebut di dalam kelas. Dengan demikian,
para guru dapat memilih buku teks yang cocok bagi para pemelajar di kelas mereka.
Kata kunci: Evaluasi materi pembelajaran, evaluasi bahan ajar, evaluasi.
xv
CHAPTER I
INTRODUCTION
This chapter comprises the elaboration of the background of the study, the
problem identification and limitation, the problem formulation, the purposes of the
research, and the research benefits.
A. Background
It has been declared by experts that material evaluation is significant in
education field. First, Rod Ellis (1998) underscored that it is mandatory to evaluate
every single book to teach a language. Second, Ellis (1998) illuminated that the
evaluation of a textbook is important because it aims to see how far it has worked and
is a partial procedure of a macro evaluation of curriculum development. Third,
Sheldon (1988) said that a textbook evaluation needs to be administered since a
textbook which is imposed on parties, both learner and teacher, by a higher authority
(in this case the Indonesia Ministry of Education) and no room for change and
modification will result a discontent. From what experts say, it can be concluded that
an evaluation process of a textbook is fundamental because it affects effectiveness of
teaching and learning process. If it happens that textbook is discontent for students
and teachers and it is kept to be used, there will be a hole in the teaching and learning
process. So, textbook evaluation is needed to be a patch on the hole through inputs
and feedback resulted from the process for its revision.
Another reason why this research is about material evaluation is because
textbook is an important resource for (1) presenting language components, (2)
1
activities, (3) stimulating ideas, (4) self-directed learning, and in some contexts, (5)
used as a syllabus and a support for less experienced teachers (Cunningsworth, 1995).
Agreeing to Cunningsworth (1995), Tomlinson (2008) stated that textbook evaluation
is a crucial process that can have a direct effect on the teaching procedure. Textbook
evaluation can be carried out to help a publisher or an institution to make decisions,
to help ourselves in developing and selecting textbook, to find out if the book being
used meets the needs of the students and to try carry out a research project (2003).
Without having good materials to teach, the program will not run very well.
Therefore, the writer sees evaluating material is absolutely important and useful for
teachers, stakeholders and the program.
There are two other reasons why teachers, experts and material developers
should do a material evaluation besides determining whether the material “works” or
not. Empirical evaluation of teaching materials can be found in accounts of trialing of
new materials (Bernard and Randall, 1995). In this case, the evaluation of teaching
materials is taken when there is a new material used in the teaching and learning
process. Hence, the main purpose of the teaching materials evaluation is to determine
whether the materials are suitable for the teaching and learning process. Another case
is when the teaching materials are evaluated to find out whether it is worthwhile
using the materials again. In such case, the evaluation is taken when teachers, experts
and practitioners plan to evaluate the existing materials. Since it has been explained
above that in Indonesia there is a new teaching material for the C13, this evaluation
research is conducted to see the teachers’ perspective toward textbook. Last but not
least, according to the need analysis survey in TEFL/TESL teachers and researchers
considered textbook evaluation to be one of the most important components of a
teacher training course (Hutchinson & Waters, 1991).
The Indonesia Minister of Education decided to load the new in 2013. Hence
in 2013, Indonesia government imposed the new curriculum called Curriculum 2013
or C13. The C13 was implemented for some rationales mentioned by the Minister of
Education (2012). Those are current global challenges, required competencies, and
current negative phenomena especially among young people and discouraging
perceptions among Indonesians regarding education (Hamied, 2014). However,
Karnodi (2014) in Hamied (2014) stated that the first implementation of C13 in 2013
seemed to raise different reactions among teachers in Indonesia. Some with defiance,
as many of them still argue that C13 is a type of curriculum developed in a top-down
and centralistic fashion. Therefore, the researcher points that a textbook evaluation of
C13 is in the urge to be conducted.
In conclusion, this present study is evaluating the new English textbook used
in seventh grade of junior high school in Indonesia mainly concerning that material
evaluation is very important today to determine to what extent a material has worked
and broader is to facilitate the whole process of curriculum development (Ellis,
1998). When we talk about evaluating teaching material, or textbook, it means that
we talk about textbook as a whole package. However, this present study limits the
dimension of evaluation. The limitation of the evaluation will be explained in the next
subtopic.
B. Problem Identification and Limitation

As mentioned earlier that the C13 raised different reactions among teachers
toward its administrative matters to pedagogical matters. Administrative matters
include dissemination of the curriculum, budget, distribution of textbooks and teacher
training (Weir & Roberts, 1994). Some questions in the evaluation may be raised
such as on how the dissemination of the curriculum will be arranged, how much
money that the authority should be put in the budget for the whole program, and
many other critical questions towards the program. Classroom teacher behaviors,
teaching method or the learning process, classroom learner activities and the teaching
materials take part in the discussion of pedagogical matters. This explains what Ellis
(1998) suggests in his journal that program evaluation covers administrative matters
and curricular matters where materials evaluation, teacher evaluation and learner
evaluation employ in it. These evaluations later will be called as macro-evaluation.
More specific than macro-evaluation is the micro-evaluation. This will involve more
specific topic for each aspect mentioned above. For instance, in material evaluation,
task evaluation can be done in micro-evaluation.
Too many aspects must be evaluated in macro-evaluation. As a matter of fact,
there are only small numbers of a complete holistic evaluation that involves various
kinds of information relating to one or both administrative matters (i.e. the logistical
and financial underpinnings of the program) and curriculum matters which later can
be broken down into materials, teachers, and learners. Hence, this study evaluates a
more specific subject. A micro-evaluation will be undergone to examine the English
textbook of junior high school in Indonesia. In this case, this research will focus only
on textbook published by the Indonesia Ministry of Education entitled When English
Ring the Bell (WERB) for seven grade level. Then, to evaluate the textbook, the
researcher determined the criteria that a textbook should meet. The textbook criteria
will be elaborated in detailed in chapter 2.
C. Problem Formulation
To evaluate the tasks in the English textbook of 7th grade of junior high school
using 2013 curriculum WERB research questions are raised as follows:
1. To what extent does textbook meet the criteria?
2. What are the strengths and weaknesses of textbook according to the survey?
D. Purpose of Study
The purposes of this study are, first, to find out to what extent the textbook
meets the criteria and to find the strengths of textbook and the weaknesses of
textbook according the users. Second, the researcher then can make any justification
based on the empirical data and the theory whether the textbook is suitable to be used
as the main material, needs additional material, is suitable as the additional material
or needs to be revised. Then the researcher then can make suggestion to teachers who
use this book, to material developers and to future researchers related to textbook
evaluation.
E. Benefits of the Study

This evaluation research on English textbook of 7th grade of junior high
school is absolutely beneficial for material developers (in this case the authority),
teachers, learners, and stakeholders.

For materials developers, this research evaluation is beneficial as reference in
developing textbook to design better and more effective textbook. The material
developer in this case the authority may refer to what has been discussed in this
evaluation research.
For teachers, this research evaluation will give detailed explanation on how
the textbook meets the criteria. Consequently, teachers may respond towards the
findings. Then teachers can take any anticipated action to take a positive response to
do in relation to the findings.
This research will also be beneficial for stakeholders since they can use result
as a reference to develop the whole English program in junior high school. Besides,
this particular evaluation research will be valuable for other evaluators especially
program evaluators to help them evaluate English program for junior high school.
CHAPTER II
REVIEW OF LITERATURE
This chapter is mainly presenting the literature reviews related to the topic of
the research. First, theoretical review is elaborated briefly to build the foundation of
this research. Second, related studies are presented and explained in this chapter to
point out what have been discussed so far by other researchers and evaluators. Last, a
framework is presented in the end of this chapter to give a clear construct of this
research.
A. Theoretical Review
This section reviews relevant constructs and critically analyzes their relations
in building up theoretical framework of the research. Based on the research problems,
it is recognized into three major constructs, namely English textbook, Evaluation in
English Language Teaching, and Indonesian curriculum of 2013.
1. Textbooks
Commonly, textbooks are a book about a particular subject that is used in the
study of that subject especially in a school. Those are commonly used by the learners
as well as the teachers in a school that provides structured syllabus of the course
program (Ur, 1991). Usually, the books are accompanied by teacher’s book as the
manual in teaching the subject and workbook to support learners in practicing the
subject.
7
a. Definition of Textbooks
Before going further talking about course books evaluation, this section
initially talks about the subject of the study which is a textbook. As a matter of fact
textbooks are taken for granted in some places while in some other places textbooks
are not used at all. In such case, textbooks are only used as the supplementary
material. However, some teachers select textbooks and use them in their classes (Ur,
1991). In other words, in some situation, textbook is used selectively and it is used to
as supplementary materials in the English course program. Therefore, it is important
to clarify the definition on what it is called as a textbook.
It seems that Ur’s (1991) definition about textbook is the easiest to
understand. She gives a clear position of the term textbook when the discussion
comes to the necessity of using a textbook. It is explained that textbook is book of
which the teacher and each student has a copy, and which later on in the learning
process it is followed systematically as the basis of a language course.
Textbooks now become a must-have element in EFL/ESL teaching. Those are
widely accepted to be the heart of any English language situation (Sheldon, 1988;
Hutchinson and Torres, 1994). It is also argued that textbooks have become the
common element in the teaching of English as the second language or foreign
language (Tsiplakides, 2011). This happens because it seems that it is too difficult for
teachers to create their own teaching materials (Sheldon, 1988). Another reason by
Sheldon (1988) to justify the argument above is that textbooks lessen the preparation
time by providing ready-made teaching texts and learning tasks.

b. Roles of Textbooks
In our discussion about textbooks, a role play of a textbook is inseparable
from the discussion. Ur (1991) says that a textbook is a framework for the teachers
and learners making them know where they are going and what is coming next.
Therefore, there is a sense of structure and progress. A textbook can be also a
syllabus of an EFL/ESL course since a textbook should provide a syllabus in it.
Besides providing a syllabus, a textbook even provides texts and learning tasks.
Agreeing to Ur’s opinion (1991), a textbook is a source for learner practice and
communicative interaction (Cunningsworth, 1995). A textbook is a convenience in
teaching and learning EFL/ESL since in a textbook the components stick together in
order and structured (Ur, 1991). According to Ur (1991), a textbook also roles as a
guidance for inexperience teacher or for teachers who occasionally unsure of their
knowledge of the language. In other meaning, it can become a support for less
experienced teachers (Cunningsworth, 1995). Finally, a textbook may become a
reference and source where learners can use textbook to learn new material, review
and monitor progress with some degree of autonomy (Ur, 1991; Cunningsworth,
1995).
In conclusion, textbook plays a very important role in teaching and learning
process since from the above explanation of the experts; textbook is inseparable from
the teacher and student. Three important roles of a textbook in English course
program dominate in teaching and learning English in EFL context. They are as a
guide book for teachers and learners, a reference book for teachers and learners and a
ready-made syllabus for teachers.

10
c. Advantages and Disadvantages of Textbooks

In regard idea or concept, the discussion definitely dragged to search the
advantages and disadvantages of it. In the same time, advantages and disadvantages
of textbook will be described based on some experts of language materials
developers.
Richard (2001) states in his book entitled Curriculum Development in
Language Teaching that the advantages and disadvantages of the use of textbook
depend on how it is used and the context for its use. He concludes that textbooks
provide structure and a syllabus for a program, helps standardize instruction,
maintains quality, provides a variety of earning resources, and is efficient. Besides,
textbooks also provide effective language models and input, can train teachers and are
visually appealing.
However, textbooks may still potentially have negative effects (Richards,
2001) such as textbook may contain inauthentic language, distort content, not reflect
student’s, and can deskill teachers. Lastly, textbook is usually expensive.
Furthermore, textbooks fail to help the learners to make full use of the language
experience available to them outside the classroom (Tomlinson, 2008).
In conclusion, textbooks have important roles in teaching and learning
EFL/ESL. The relationship between teacher and textbooks is an important
consideration and is at best when it becomes partnerships when both share common
goals to which each party contributes to the teaching and learning process
(Cunningsworth, 1995).
11
d. Textbook Criteria
It is such an uneasy task to judge whether a textbook or teaching material, in
this case is ELT material, is good or bad (Tomlinson, 2008) since every textbook or
ELT material has its own strengths and weaknesses and there is no perfect textbook
exists. Instead of using the word good and bad, Cunningsworth (1995) used the word
effective and ineffective. Consequently, many experts and researchers have been
doing evaluation on textbook as part of the curriculum development process to find
the best possible one that will fit and be appropriate to a particular learner group.
Hence in this section of the study, the researcher is explaining what aspects of
textbook that are usually evaluated by the experts and researchers as well as teachers
to seek the best and most appropriate textbook for a particular group.
Cunningsworth (1995) suggested that it is significant to ensure that careful
selection is made and that materials selected closely reflect the need of the learners,
the aim, methods and values of the teaching program. To carefully select the
materials or textbooks, a predictive evaluation can be administered (Ellis, 1997) and
some criteria of a textbook should be considered (Allwright, 1981, Cunningsoworth,
1995, Richard, 2001 and Sheldon 1988).
Goal of a textbook seems to be an important criterion to be evaluated. In many
evaluation checklists from the experts such as Allwright (1981), Sheldon (1988),
Cunningsworth 1988 and Richards (2001) evaluative question regarding the goal of a
textbook is always put at the beginning of the checklist. As also suggested by Nunan
(1989), in developing teaching materials the objective, aim or goal of a textbook
should be generated in the first hand. In addition, the aims of a textbook should
12
correspond with the aim of the syllabus and curriculum (Cunningsworth, 1995;
Nunan, 1989 and McDonough and Shaw 1993:64). However, conflict of goals cannot
be avoided since teachers, learners, materials developers and sponsors involve in
teaching and learning management (Allwright, 1981). There is always a conflict of
teachers, leaners, material developers and sponsor goals found in textbook. Shortly,
Allwright (1981) argued that a textbook may contribute to some ways but it cannot
determine goals.
The approach of a textbook is also considered to be evaluated in a textbook
evaluation process. Cunningsworth (1995) and (Brown, 2001) suggested some
questions to evaluate this criterion whether a textbook is flexible to any teaching and
learning situation and style. In a slight different opinion to this approach issue by
Allwright (1981) was that in materials evaluation assessors or teachers should be able
to measure to what extent the management of the activities are maximally profitable.
However, he pinned important clause that textbook can and do contribute to the
teaching and learning process, including the teaching approach, but cannot possibly
cope with many important decision made by the teachers regarding their various
situation (Allwrigth, 1981). In conclusion to the discussion about the approach, the
researcher agrees to Allwright’s opinion that detailed local decision is clearly beyond
the scope of publication (1981). The materials or textbook then can only impose
suggestion to the teachers.
Human being loves everything which is beautiful. It is also not too much to
say that aesthetically positive experience through the use of attractive illustration and
its design can facilitate learners in language acquisition and development (Tomlinson,
13
2008) since it can arouse the learners’ interest (Rowntree 1997,92). A good material
must provide a pleasurable learning experience which can be achieved by practicing
learning items in an interesting and novel way (Richards, 2001). This can be done by
using attractive illustration and design (Tomlinson, 2008). Besides the design and
illustration, a good textbook should help the learners feel at ease (Rowntree 1997).
Therefore the sequence and the grading of textbook should be able to make the
learners at ease and suitable for the learners (Cunningsworth, 1995). In the same time,
Cunningsworth (1995) also believes that a textbook should allow the teachers to
undergo various teaching styles. In a simple way, a textbook should be flexible for
the teachers.
The next criterion of a textbook to be reviewed is the language content. In this
aspect of textbook, three main elements are described. Those are grammar,
vocabulary and phonology. Cunningsworth (1995) sees that those three elements are
important in term of learning the language. Therefore, in his textbook checklist, he
tried to find out whether a textbook provides adequate grammar item for the students
to learn autonomously and teachers as the source of grammar knowledge. Besides the
grammar item that should be presented in textbook, vocabulary in terms of quantity
and range must be adequately provided in a textbook. Another important aspect that
should be presented in textbook related to content is the pronunciation. A textbook
has to give teachers and students the pronunciation work in terms of individual
sounds, word stress and intonation.
Related to the content of textbook, language skills presented in textbook
should be carefully paid attention of since referring to what Tomlinson (2008)

14
suggests in his critical review book that an effective textbook must provide the
learners with the experience of extensive listening and/ or extensive reading. His
opinion definitely supports the input Hypothesis by Krashen (1981).
The topic presented in a textbook ideally should have enough variety and
sufficiently fulfill the learners’ interest. The topic presented in a textbook also should
be content-wise enough for the learners’ language level. The social cultural context
presented in a textbook is also being considered to the use of English in the real life.
A textbook must also consider the social structure such as gender group, ethnic
group, occupational group and other group such as disability (Cunningsworth, 1995;
Brown, 2001; and Tomlinson, 2008).
Many experts in textbook evaluation field share ideas regarding the criteria of
a textbook. They seem explaining and putting the name for each criterion in different
way. However, whatever the term they are using for each criterion, actually they are
referring to the same ideas. Therefore, in this study the researcher respectively
adapted Cunningsworth’s (1995) evaluation checklist which has covered all criteria
used in this study. The six aspects of textbook will be evaluated are its aims,
approach, design and organization, language content, skills and topic. In this study
the researcher came with six aspects a textbook should have. Each aspect should at
least meet with one criterion. The criteria used in this study are as follow:
15
Table 2.1: Criteria of an Effective Textbook
Aim The aims of a textbook should correspond with the aim

of the syllabus and curriculum (Cunningsworth, 1995; Nunan,
1989; and McDonough and Shaw 1993:64).
Approach
a. A textbook should be flexible to any teaching and
learning situation and style (Cunningsworth, 1995 and
Brown, 2001).
b. Material should take into account that learners differ in
learning styles (Tomlinson, 2008)
Design and a. attractive illustration and its design can facilitate

organization learners in language acquisition and development
(Tomlinson, 2008)
b. Therefore the sequence and the grading of textbook
should be able to make the learners at ease and suitable
for the learners (Cunningsworth, 1995).
c. a good textbook should help the learners feel at ease
(Rowntree,1997 and Tomlinson, 2008).
Language a. a textbook should provide adequate grammar item for

content the students to learn autonomously and teachers as the
source of grammar knowledge (Cunningsworth, 1995)
b. vocabulary in terms of quantity and range must be
adequately provided in a textbook (Cunningsworth,
1995)
c. A textbook has to give teacher and student the
pronunciation work in terms of individual sounds, word
stress and intonation (Cunningsworth, 1995)
Language
a. Integrated language materials should provide language
Skills skills needed
b. Materials should facilitate learners with adequate and
rich reading text (Tomlinson, 2008)
Topic a. To achieve impact, a textbook should present appealing

topics of interest to the target learners. (Tomlinson,
2008)
b. A textbook should present topics which offer the
possibility of learning something new. (Tomlinson,
2008)
16
2. Evaluation
In the second language education, the main purpose of evaluation is ultimately
for decision-making based on the informed judgment. That is why enough data
gathering is highly important in the evaluation process (Richards, 1996). The
information can be gained from the programmer, administrators, teachers, learners or
the stakeholders as part of the implementation of a program and can be used as the
program development.
a. Definition of Evaluation
As it is mentioned above, the ultimate purpose of evaluation is to collect
information systematically in order to indicate the worth or merit of a program or a
project then to make a decision. Therefore, Brown (1989: 223) suggests a good
working definition to be the systematic collection and analysis of all relevant
information necessary to promote the improvement of the curriculum and assess its
effectiveness and efficiency, as well as the participants’ attitudes within a context of
particular institutions involved. Similar definition proposed as follow, evaluation is a
natural activity as intrinsic part of teaching and learning that enables evaluators to
make judgments towards the program or curriculum that is done systematically and in
principled way (Rea-Dickinds & Germaine, 1992). Obviously, both definitions agree
that evaluation is a systematic information gathering to make a judgment. They also
mention a program and curriculum in the context of educational evaluation. In the
case of educational evaluation, it can be done before the program commences or
before. It of course can also be done during the program is being commenced
(Brown, 1994). This is supported by Hewings (1996). The scope of evaluation has
17
moved from a concern with test result to the need to collect information and make
judgements about all aspects of the curriculum, from planning to implementation.
In relation with language program, in this context is English program,
evaluation may focus on many different aspects. First, it may focus on curriculum
design which is to provide insights about the quality of program planning and its
organization. Second, the concern can be on syllabus and program content in which it
will deal with the relevancy and engagement of the syllabus and program content and
the successfulness of the assessment and test procedures. Third, that is classroom
process which aims to provide insights about the extent to which a program is being
implemented appropriately. Fourth, material of instructions where evaluators may
focus on materials of instruction in which provides insights about specific materials
are aiding student learning. Fifth, that is the teachers which need to consider main
questions such as who conducted the teaching, what was the teachers’ perception of
the program, and what did the teachers teach. Sixth, it can be about teacher training
which aims to assess whether training teachers have received is adequate. Seventh,
the point is about the students in which the research focuses on investigating what the
students learn from the program, what their perception toward the program is, and
how the students participate in the program. Eighth, the concern is on monitoring of
the students’ progress.
One of evaluation process which is categorized based on time frame it is
conducted is formative evaluation or in-progress evaluation of student learning.
Based on Sanders (1992) and Weir and Roberts (1994), the focus aims of conducting
this type of evaluation are on five matters. First, that is learner motivation where
18
evaluators try to seek insights about the effectiveness of teachers in aiding students to
achieve goals and objectives of the school. Second, that is learner motivation where
the evaluators try to seek what drives learners to learn. Third, it is about the
institution where the evaluators try to what administrative support was provided, what
resources were used, what communication networks were employed. Fourth, learning
environment is to provide insights about the extent to which students are provided
with a responsive environment in terms of their educational needs. Fifth, staff
development aims at providing insights about the extent to which the school system
provides the staff opportunities to increase their effectiveness. Last but not least, that
is about decision making which is to provide insights about how well the school staff-
principals, teachers, and others – make decisions that result in learner benefits.
According to the above explanation, it can be concluded that evaluation is a
long process conducted to provide information. The information is to improve
curriculum, teacher development, students, the effectiveness of teaching and learning
process and materials used. All things mentioned are to achieve educational and
instructional goals.
b. Type of Evaluation
In general, evaluation in ELT mainly occupies two kinds of evaluation namely
macro-evaluation and micro-evaluation. The distinction between two evaluations is
that macro-evaluation deals with the administrative and curriculum matters that
undergo evaluation for the whole programs. Besides, in micro-evaluation, it only

19
deals with the break-down of the curricular matters such as teachers, materials and
learners (Ellis, 1998).
Weir and Roberts (1994) distinguish the types of evaluation for language
program based on two major purposes, program accountability and program
development. Accountability refers to the quality of the work of those involved in a
program (Richards, 2001). He also makes a point that accountability evaluation
usually examines the effect of a program and its significance towards educational
system is usually conducted by an independent institution as the external audience of
the education system or by the decision maker. In contrast, development-oriented
evaluation aims to improve the educational quality of a program or project (Weir and
Roberts, 1994). This kind of evaluation is normally done while the program is in
progress.
Richards (2001) gives different opinions about purposes of evaluation to what
Weir and Roberts suggested. He distinguishes it into three different kinds of
evaluation. The first type is formative evaluation. Formative evaluation refers to an
evaluation that is undergone as part of the process of the program to seek what is
working and what is not and what problems need to be addressed. Since this kind
evaluation focuses on ongoing development and improvement of the program, it is
usually conducted during the program. An evaluative question related to textbook that
typically raised in formative evaluation is how well textbook is being received.
Another type of evaluation is Iluminative. In this kind of evaluation,
evaluators try to find out the different aspects in the program and their
implementation. This kind of evaluation focuses on the teaching and learning process
20
that occurs in the program. Therefore, it is rarely to ask about the materials in this
kind of evaluation.
The last type suggested by Richard (2001) is summative evaluation. This third
type of evaluation is the type with which most teachers and program administrators
are familiar. Summative evaluation concerns on whether a program is effective and
efficient and to some extent acceptable. Richard points that this kind of research is
conducted after the program has been implemented. This kind evaluation is also
asking whether the materials work well. Brown (1994), related to the material
evaluation, makes an important point that any evaluation can be carried out before the
program commences or after and or during the program.
By relying on the above explanation, this present study agrees on what Brown
pointed that any evaluation can be carried out before, after or during the program. In
this study, the evaluation is conducted during the program. Another point to
reconfirm is that this present study undergoes a material evaluation in form of
textbook evaluation.
c. Check List of Textbook Evaluation

Related to textbook evaluation, McDonough, Shaw & Masuhara (2013) give
two different stages instead of the type of textbook evaluation. The first stage is a
brief external evaluation. This includes criteria which gives an overview of the
organizational foundation of the book. Cunningsworth (1995) give similar ideas with
different term, impressionistic evaluation. In this stage an assessor only does a brief
look to textbook to give a brief description about textbook. Following this is an in-
21
depth internal investigation of textbook, ‘to see how far the materials in question
match up to what the author claims as well as to the aims and objectives of a given
teaching program’ (McDonough, Shaw, & Masuhara, 2013). In other words, in-depth
internal investigation must be done to analyze to which extent result of external
investigation match up with internal investigation.
Another criterion is illuminated by McDonough, Shaw & Masuhara (2013).
They comprise nine criteria before conducting in-depth internal evaluation. These
criteria are usually found at the cover or in the introduction of the book. The criteria
or checklist are in a form of questions that will be answered by evaluators.
In parallel with Shaw (2013) and Rea-Dickins (1992) suggest a checklist in a
form of preliminary questions. Then, evaluators answer the questions before
analyzing which teaching and learning materials are suitable. The questions are as
follows.
Rea-Dickins (1992) provide these preliminary questions above. Though in the
end of the procedure, they point important consideration that since teaching and
learning materials are sensitive to the language process, evaluation criteria should
relate not only to the aims and contents of language learning but also to the
procedures of working with texts and performing tasks in the classroom. That is why
they give credits to Cunningsworth’s work since as it can be seen in appendix, the
criteria cover Germain have pointed. To compare to the criteria suggested by
McDonough, Shaw & Masuhara (2013), what Cunningsworth marked gives more
22
detailed and comprehensive questions related to the methodology in which Rea-
Dickins (1992) suggested.
Table 2.2 Checklist Suggested by Rea-Dickins & Germaine
Criteria 1: What do materials mean for you?

1. Do you refer exclusively to textbooks, or do you include teachers’
guides, teaching manuals, supplementary units, readers, audio and
visual materials, etc?
2. Do you make a distinction between materials designed specifically for
first and second language teaching, and also between those targeted
specifically for use in school and materials that are non-pedagogic but
authentic?
3. Do you include materials produced by the teachers and the learners?
Criteria 2: What are the roles of material within your teaching and learning
context?
1. What role(s) are they expected to play?
2. What role(s) are they expected to achieve?
Criteria 3: How are the materials to be used?

1. Are they to be used as the sole source and resource for teaching?
2. Are they one of several available resources?
A more recent study held by J. Mukundan, (2011) in developing English
language textbook evaluation checklist was done in a focus group discussion. Six
participants joined the developers to improve the items of checklist generated by
Cunningsworth’s evaluation checklist (1995) in reference to their clarity and
inclusiveness. In this study, researchers successfully developed an evaluation
checklist for ELT textbook. They added, omitted, and revised the Cunningsworth’s
evaluation checklist.
In relation to Indonesian context and considering the above explanation, this
present study sees that Cunningsworth’s evaluation check list will work well to
23
evaluate the examined book. Therefore, this present study will be spelt out the
evaluation criteria by Cunningsworth in form of questionnaire to be given to the
teachers to evaluate the English textbook of seventh grade junior high school in
Indonesia.
3. Review on Indonesia National Curriculum 2013

A new curriculum has been implemented in Indonesia since June 2013. This
new curriculum is called curriculum 2013 (C13) which is based on the Regulation of
Ministry of Education of Indonesia number 22, 2006. C13 has been implemented in
all around Indonesia for all levels of education from Elementary School (grade 1-6),
Junior High School (grade 7-9), and High School (grade 10-12). In the first step of
the implementation, not all grades were using the new curriculum. Only grade 1, 4, 7
and 9 run the C13. This new curriculum was also not implemented in all school in
Indonesia. For the first stage of the implementation, Ministry of Education only
pointed some schools to become the pioneering schools for this curriculum
implementation.
To give an illustration on the C13, the following explanation is a brief
description on the core competence and base competence. There are four core
competences for English subject in K 13. All of them are related to religious and
moral conduct and conceptualized in the base competence.
To make the explanation crystal clear, samples of a base competence stated in
C13 (Core and Basic Competence of Curriculum 2013) in the following six points.
First, it is composing oral text to express and respond greeting, leave taking,
24
thanking, and apologizing correct language features in context. Second, it is
composing oral and written text to express and respond introducing oneself in short
and simple way by paying attention to social function, text structure, and language
features correctly and contextually. Third, it is composing oral and written text to
mention names of days, month, and time in days, time in numeric form, dates, and
year by using the language features correctly and contextually. Fourth, it is
composing oral and written text to mention names of animals, things, and public
places found in the real life by using the language features correctly and contextually.
Fifth, it is composing oral and written text to make names in label and list down
things by using language features correctly and contextually. Sixth, it is composing
oral and written text to describe personal characteristics, animal and things by using
the language features correctly and contextually.
As stated above, it is very obvious that current curriculum deploy three main
aspects reflected in the core competence. Those are attitude, skills and knowledge. In
addition to the explained aspects, C13 performs the thematic and integrated concept
of syllabus which can be seen in the official website of education and cultures
ministry.
Taken from the preface of the teacher’s book of English textbook, it is said
that C13 gives a significant change in terms of teaching and learning process.
Previous curriculum enables teacher to deliver the subject or knowledge. In contrary
to the prior curriculum indeed, C13 enables the students to share knowledge among
them by the guidance of the teacher. In pedagogical term, the teaching and learning is
a two-way teaching and learning process. Though, in the real implementation some
25
teachers in Indonesia have implemented a two-way teaching process instead of one
way. However, such approach, had not been realized until the C13 was issued. As a
consequence, by applying this curriculum, students are intended to be more active in
doing observation, thinking, questioning, and communicating as well as presenting
what they have earned from the knowledge receiving process
(http://kemdikbud.go.id/kemdikbud/uji-publik-kurikulum-2013-1).
Four standards that happen to be changed in curriculum 2013 from curriculum
2010 covers graduate competence, process content, and standard of learning
evaluation. Related to learner evaluation/assessment, there are four criteria of learner
assessment: 1) level of achievement of the use of social function text; 2) level of
completion and arrangement of text structure; 3) accuracy of language features such
as grammar, vocabulary, pronunciation, word stressing, intonation, and spelling; and
4) level of appropriateness of writing and delivery format. Four methods of learner
assessment are provided in this curriculum to meet with the core and base
competence. The four methods of learner assessment are assessing student
performance, student observation, student portfolio, reflection and pair assessment.
In conclusion to the explanation above, it is highly required to evaluate
textbook of new curriculum 2013. It is since many new constructs applied in the
curriculum that become valuable source to adjust the materials used. In addition,
significant shifted paradigm such as the teaching methodology suggested in the
curriculum.
26
4. Review of Previous Studies

In this section of the study, two previous studies will be reviewed briefly to
give depiction and broader knowledge in doing the research. The review of previous
studies focus on the methodology of each research to compare one to another in order
to get depiction of what and how the researcher conduct an evaluation especially
textbook evaluation.
A thesis by Alamri (2008) entitled An Evaluation of Sixth Grade English
Language Textbook for Saudi Boy’s School was conducted to evaluate a new
textbook. Conducted in King Saud University, Riyadh, this research was a survey
research which used a set of questionnaire to elicit the perspective of 93 English
language teachers and 11 supervisors in Riyadh Educational Zone about textbook in
question. The questionnaire comprised 64 groups under 12 main categories. For the
questionnaire design, he adopted four different evaluation checklists from four
different experts and researchers. As the result of the adoption of those four
checklists, (a) general appearance, (b) design and illustration, (c) accompanying
materials, (d) objectives, (e) topic contents, (f) language contents, (g) social and
culture contexts, (h) language skills, (i) teachability, (j) flexibility, (k) teaching
methods, and (l) practice and testing are used as the categories in the questionnaire.
After getting the data, the researcher analyzed them through descriptive statistic. The
findings showed out of 64 items in the questionnaire, only 13 items had means less
than 2.50. The category that had the highest mean was the one on learning
components. Meanwhile, the category that had the lowest mean was the one on the
teaching methods. Based on the findings, it was also found that there were no
27
significant differences between the means of two populations of the study, teachers
and supervisors, except on the flexibility of textbook. The different natures of their
jobs could be a possibly reason for this difference.
Another example of impressionistic evaluation using an empirical qualitative
approach is the one done by Nemati (2009) in India. This research involved 26 Pre-
University teachers. Because this research also focused on vocabulary factors, besides
a questionnaire, the data were also taken from vocabulary response. The
questionnaire that was distributed to the respondents then followed by an interview
for each respondent to triangulate the data, it was arranged based on textbook
evaluation checklist. She adopted several checklists from the authors such as
Cunningsworth, (1995) and Ur (1996). After studying the checklists from the authors,
she decided to select features which are in common. The criteria used in the
questionnaire are as follow (1) logistical factors, two questions about the availability
and price of the book, (2) pedagogical factors, or methodology, with two subparts of
skills and contents, (3) human factors, including teacher friendly and learner friendly
factors, (4) vocabulary factors, entail questions regarding vocabulary learning
strategies. From the research result, teachers’ belief toward textbook was revealed
based on the criteria. In addition, suggestions to improve textbook were also
obtained.
The other example of textbook evaluation research discussed in this section is
conducted by Zamanian (2015). Aiming at evaluating ‘Learning to Read English for
Pre-University Students’ used in Iranian high schools, the researchers distributed a
questionnaire to 75 high school English as a Foreign Language teachers. In designing

28
the questionnaire, Litz’s checklist (2000) was adopted. After the data were collected,
those are analyzed based on a Likert scale. Based on the research results, it was
known that textbook was quite far from teachers’ standard. It happened because
textbook ignored listening and writing skills. Moreover, it did not follow a
communicative approach in teaching different skills and language components.
Similar to those three research, this study is conducted empirically. After
being used to teach junior high school students in Indonesia, textbook entitled ‘When
English Rings the Bells’ is evaluated. Data are going to be collected by distributing
questionnaire to evaluate whether the material is in line with what teachers expect.
Similarly, this research also captured textbook evaluation process done by Indonesian
teachers after they used textbook in a classroom context. Framework used for this
research was Cunningsworth’s (1995). Using the framework, the research focuses on
the aim, the approach applied, the language skills presented in textbook, the design
and organization, the content of the language, and the topic.
B. Theoretical Framework
From the literature reviewed above, the theoretical framework can be drawn
as follows. In conducting this evaluation research, the criteria of a book evaluation
are the most important and significant aspects to be paid attention of in resulting a
reliable research. The criteria established in this research are selectively adapted from
the Tomlinson’s, Bworn’s, Cunningsworth’s and Nunan’s book evaluation criteria.
Furtherly, the criteria are conceptualized into a research instrument in form of
questionnaire. The instrument is used to measure textbook by the practitioners in this

29
case are English secondary school teachers. Each item of questionnaire is also
considering what is currently formulated in C13. Therefore, the concept of textbook
evaluation by Tomlinson (2008), Brown (2001), Cunningsworth (1995) and Nunan
(1989) contributes largely in drawing the concept of the criteria applied by the
researcher in this evaluation study.
The framework is developed from some underlying theories about evaluation
in English language teaching (ELT) and textbook. Those encompass the framework
on textbook evaluation and textbook development. From the literature reviewed
above, the framework can be theoretically drawn as follows.
Tomlinson (2008)
Brown (2001)
Textbook Curriculum
Evaluation 2013 (C13)
Cunningsworth (1995)
Nunan (1989) Checklist
Aim Approac Language Design and Language

Topic
Organization Content
Figure 2.1 The Theoretical Framework

CHAPTER III
RESEARCH METHOD
This chapter discusses the research methodology which was employed in an
attempt to answer the research question empirically. In the relation to the path of data
collection, this chapter discuss research method applied. After that, research setting
and participants, research instrument, data gathering and analysis technique are
scrutinized in this chapter.
A. Research Method
This research was under an umbrella of applied research that enabled the
researcher to evaluate a program, in this case, it was to evaluate a textbook. It was
adopted in this research since it embraces a set of process in evaluating the quality of
textbook. Evaluation research study was designed to find out to what extent a new
program, policy, textbook, a new way of doing something and so forth, has met the
criteria mentioned in the second chapter of this report.
Evaluation research can be undergone quantitatively or qualitatively. In favor
to conduct quantitative research, researcher may use a survey research. To conduct
evaluation research using qualitative data, the researcher must use the field research
(Neuman 2006). In survey research Neuman suggests to researchers to collect data
from a sample or could be smaller group of selected people. He suggests around 150
respondents to generalize result to a larger group of 5,000 people.
30
31
B. Research Respondents
This research targeted junior high school teachers who implemented
curriculum 2013 as the respondents. As it has been described in the previous chapter,
the implementation of curriculum 2013 did not cover all junior high schools
throughout Indonesia. Each region and province in Indonesia has pointed selected
school to implement and use textbook of curriculum 2013 in odd semester. In odd
semester 2014, more schools implemented C 13.
Since it was survey research, it was conducted in 10 junior high schools in
Special Province of Yogyakarta. These schools were pointed to implement the C13 in
Yogyakarta. Special Province of Yogyakarta has five districts including the city of
Yogyakarta. The researcher picked up two schools for each district as sampling. From
each school, the researcher took two teachers to be participated in the survey
research. The teachers vary in terms of the length of teaching experience. The very
least year of experience was two years. While the most experienced teacher had 35
years of teaching experience.
To sum up, this research was supposed to involve twenty teachers teaching in
different junior high schools in the province of Yogyakarta. However, due to some
circumstances, there were two teachers who refused to join the survey and one
teacher filled out the questionnaire with the same answers for every item. As result,
this respondent is considered failed. Therefore, by the end of the data gathering, there
were only 13 respondents whose data was considered as valid.

32
C. Research Instrument
Two research instruments were utilized in order to attain feedback from this
study. First, it was an evaluative questionnaire adopted from Cunningsworth’s
checklist. Second, to triangulate the data taken, an open ended questionnaire was
distributed then.
For the evaluative questionnaire, the items were established by adopting
Cunningsworth’s checklist which focused on the aim, approach, language skills,
design or organization, language content, and topic used. There were seventy-one
items prescribed in the questionnaire which these items were measured using Likert
scale. In order to reduce the number of invalidity of the survey result, the researcher
only used 4 scales instead of 5 scales namely scale 1 for strongly disagree, 2 for
disagree, 3 for agree, and 4 for strongly agree to the item.
Beside closed item which was prescribed in form of statement, the
questionnaire also encompassed open questions for the respondents to answer. This
was aimed to obtain the triangulation data. Besides using open questions, the
researcher also used textbook close reading to get the factual evidence from textbook
to crosscheck whether the survey results were confirmed.
D. Data Gathering Techniques
In gathering the data, at first, the researcher made an appointment with twenty
teachers. However, like what it has been stated above that due to some circumstances,
two teachers refused to join the survey and one teacher filled out the questionnaire
with the same answers for every item. As result, only thirteen teachers who kindly
agreed to the appointment.

33
The data gathering was conducted by the end of the even semester 2012/2013
Indonesian academic year in June 2013. The researcher started to meet respondents
from school to school. So, it took a month to take the data. From thirteen sets of
questionnaire were distributed, amazingly all of them returned. However, there was
one questionnaire was returned remain unfilled by a respondent.
E. Data Analysis
The very first step of data gathering was distributing questionnaire to the
respondents. The respondents were given a week to fill out the questionnaire
completely. The researcher came over to each respondent to explain the procedure in
filling out the questionnaire although the researcher had written a clear instruction in
the questionnaire. This aims to make sure that the respondents were able to fill out the
questionnaire correctly. Then, after a week the questionnaires were collected. After
the questionnaire were returned to researcher, the results then were put into the tally
system as seen in the table below.
After putting all the result in tables, the researcher calculated the data to get
the MEAN for each item of the questionnaire. In addition, the standard deviation was
also gained to find out whether each item was valid. In the table presented in this
survey research report, it comprises the percentage of all fourth scales (1 = strongly
disagree, 2= disagree, 3 = agree, and 4= strongly agree), the Mean, and the standard
deviation.
To read the table, it can be inferred from the mean of the item. If the mean
more than 2.5, it means that respondents or teachers agree to the statement. In
34
contrary, if the mean of an item is showing less than 2.5, it means the item is not
agreed by the respondents. Therefore, the researcher presented the data for each item
in two digits of decimal. It aims to see the difference one item to another more
precisely.
After completing the tally sheet, the researcher then created tables based on the
result of the survey. The table helps the researcher to describe the result of the survey easily.
Hence, later on the researcher will be able to analyze then discuss the result based on the
diagram and the answers of the open questions.
After obtaining the result survey, the data inform of score are then presented
into table in order to make the interpretation easier to elaborate. The researcher later
describes the survey result elaborately in chapter IV section one. And then, in the
second section of chapter IV the researcher is discussing the survey results and
relating them with the theory in learning and material development. This discussion is
aimed to get a comprehensible conclusion and decision toward textbook since the
main goal of evaluation is to make decision based on the data gathering.
Although the researcher had time constrain, the researcher successfully interviewed
respondents to triangulate the data. In addition, the researcher also happened to give them
several open questions on the questionnaire. They had to write the open questions on the
questionnaire sheet. While the researcher collecting the questionnaire, the researcher tried to
confirm all the answers written on the sheet by. By doing this, the researcher was able to
understand the answers from the respondents comprehendingly. Then, all the answers of the
open questions were written in a table and categorized. Hence, these data are ready to use in
the analysis to support the survey result.

CHAPTER IV
RESEARCH FINDINGS AND DICUSSION
In this chapter, it is described the survey result begin with the general
findings. The following elaborative description is explaining the survey result of each
aspect. Six aspects of textbook being evaluated are its aim, approach, design and
organization, language content, skills and topic.
A. General Findings
In general, the evaluated textbook remained credible as a textbook used in
English language classroom for secondary school. Despite the debate regarding the
C13 and its textbook, the survey result proved everything regarding the textbook. The
result for each aspect of the evaluated textbook shows that teachers believe this
textbook is still credible and reliable to be used in the English language classroom for
secondary school students.
Later in this section, a table showing the overall result of the survey is
displayed. The Mean of each aspect is telling the average of the respondents’
response toward the aspects. Each aspect was evaluated by several statements
formulated in the questionnaire. Therefore, the mean represents how each aspect
meets the criteria based on the teachers’ or respondents’ perspective. Since the scale
employed in the questionnaire is 1 to 4, the Mean which is more than 2.5 means that
the aspect meets the criteria. To see to what extent the aspect meet the criteria, it can
be seen how far the Mean overly reach the limit (2.5).
35
36
Teachers believed that the aims of the textbook were definitely in line with the
curriculum and the program objectives. As it is seen from the table, the mean for this
aspect shows 2.97 and is ranked as the highest mean among other five aspects to be
evaluated. The second highest mean of the aspect is 2.76 for the topic of the textbook.
This unavoidably shows that teachers consider the topics of the textbook are
appropriate for Indonesia context as well as representing different group and
ethnicity. As though aims and topic of the textbook, the approach used in the
textbook happens to be one of the aspect proving that this textbook remains reliable.
This is shown in the table which 2.61 happens to be the mean for approach of the
textbook. The other three aspects namely the design and organization, content, and
language skill have almost the same result for each.
Table 4.1: Overall Survey Result

Aspect Mean
No
2.97
1. Aims
2.61
2. Approach
2.55
3. Design and organization
2.52
4. Content
2.32
5. Language skill
2.76
6. Topic
However, teachers and material developer are obliged to see further and
deeper on each aspect. Therefore, addition to the material as well as revision to the
existing material may be necessary to be done in favor to the improvement of the
textbook. Hence teacher may find the additional material focus on the specific
37
component of the textbook and the material developer may revise the textbook on that
specific component as well. Accordingly, the next subchapter will elaborate deeper
on the survey result of each item of the aspects.
B. Detail Findings
In this part, all criteria will be elaborated in detail by giving the real survey
data and related theory as well as data taken from textbook to enrich findings. The
researcher will also add small part of discussion related to certain item of the criteria.
In the next section, the researcher will give general discussion about the evaluated
textbook. In this sub-section, the detailed findings will start from the very first
criterion.
1. The Aims
The first analysis goes to the first aspect, the aims of textbook. As it is seen in the
table, there are four items or statements to measure the agreement of the aims of
textbook. The first item is rated at 3.11 by the respondents. This mean shows that it is
higher than 2 and even higher than 3. This means that teachers are mostly agree on
the statement in item 1. Teachers saw that the aims of this textbook closely
correspond with the aim of the curriculum. The respondents rated the second item the
aims of textbook correspond closely with the needs of learner at 3 for the mean. As
though the first item, the second item was agreed by the respondents. In conclusion,
almost all respondents saw that the aim of this textbook has no issue and think that
the aim of textbook had been in line with the curriculum and learner needs. See the
table for the detail.

38
Table 4.2: Survey Result on the Aims of the Textbook

Item Statement Strongly Disagree Agree Strongly Mean SD
Disagree Agree
(%) (%) (%) (%)
1. The aims of 0 11.1 66.7 22.2 3.11 0.582
the textbook
correspond
closely with
the aim of
curriculum.
2. The aims of 0 16.7 66.7 16.7 3 0.594
the textbook
correspond the
core of the
competence.
3. The aims of 0 16.7 66.7 16.7 3 0.594
the textbook
correspond
closely with
the aim of the
teaching
program.
4. The aims of 0 27.8 66.7 5.6 2.76 0.545
the textbook
correspond
closely with
the needs of
learners.
However, teachers and material developers must carefully see that the mean
of item number 4 namely the aims of the textbook correspond closely with the needs
of leaners is not that high. It is only 2.76 a little higher than 2. In other words, 27.8 %
of the respondents disagreed to the statement. Therefore, to determine whether each
statement in this criterion is factually confirmed, the researcher needs to look at the
results of the next criteria and the actual evidence found by reading closely to
textbook and contrasting them with what the survey results are. Therefore, the
researcher will elaborate each statement at the discussion section of this chapter.
39
2. The Approach
The second aspect to measure is the approach of textbook. There are seven
statements to measure this criterion. See table 4.3. It can be clearly said that
respondents agreed that textbook is suited to the learning situation. However as a
researcher, it is important to analyze it deeply. Based on the survey, the score of this
statement is 2.7. It means the textbook suited to the learning situation. Respondents
see that textbook is effectively suited to the learning situation although the score is
low. It is a survey research, therefore teachers’ statement towards this topic cannot be
proven empirically since the researcher did not conduct observation to see how
effective and judge whether or not the aim of textbook is suited to the learning
situation.
The second statement is seen as effective one by the respondents. It means
that respondents agree that textbook is suited to the teaching situation. However, the
number of agreement to this statement is absolute which means some other
respondents see that textbook is not suited to the teaching situation.
Remarkably, most respondents disagree to the third statement. The Mean is
2.23 See table 4.3. Respondents did not agree that textbook covers most of all student
needs since it does not reach 2.5. Another negative result happens at the fourth
statement. Respondents did not agree that textbook is a good learning resource for
student. But the value of the disagreement is low. It is 2.47.
Most teachers agreed that textbook does not cover all students’ need
(Diagram). For sure it is not easy to facilitate all students need. However, at least a
40
textbook must be able to give the most important need for the students for example
the need of language input. This study attempts to confirm the empirical data taken
from the survey and textbook itself with the theory. As Krashen (1982) proposed the
input hypothesis, comprehensible input is very important for second or foreign
language learners. Comprehensible input means the learners know what people say
and what the message is, instead how people say. Let us refer to textbook –When
English Rings the Bells. Here are two picture captured from the book on page 7.
Table 4.3 Survey Result on the Approach of the Textbook

Disagree Agree
(%) (%) (%) (%)
5. The textbook 5.6 22.2 66.7 5.6 2.70 0.665
suited to the
learning situation.
6. The textbook is 5.6 22.2 66.7 5.6 2.70 0.665
suited to the
teaching
situation.
7. The textbook 5.6 61.1 33.3 0 2.23 0.545
covers most of all
student needs.
8. The textbook is 5.6 44.4 44.4 5.6 2.47 0.696
good resource for
students.
9. The textbook is 0 44.4 50 5.6 2.58 0.599
good resource for
teachers.
10. The textbook is 0 27.8 61.1 11.1 2.82 0.616
flexible so it
allows various
teaching styles.
11. The textbook is 0 27.8 66.7 5.6 2.76 0.545
flexible so it
allows learning
styles.
Another important aspect in language learning is grammar (Krashen, 1982).
Grammar inevitably cannot be separated from the language learning since grammar
41
has a great deal in language learning as Monitor (Krashen, 1982). Obviously, a good
textbook is when textbook can be used by the students for individual learning
(Tomlinson, 2008). Then, the question is ‘what is the relation between grammar
reference and individual learning?’
As it is mentioned above according to Krashen (1982) that grammar plays
important role as Monitor while in individual learning, students have no supervision
from the source of knowledge except textbook itself. Therefore, grammar reference is
surely needed in a textbook to fulfill students’ need in individual learning process. To
support individual learning in ELL, a textbook must be equipped with features or
section which can be used without any supervision or assistance from teachers or
tutors. Hence, the grammar reference can be the Monitor in their language learning
process.
Figure 4.1 Sample of activity found in textbook: Activity 5

On page seven in activity 5 as you can see at figure 1 it can be seen from the
instruction that students are asked to practice the dialogs given with people around
42
them. In the dialog it is found words or phrases like feeling great and fine to express
positive response in greetings as reading input applied in activity. On this page at
chapter 1 is the only presentation of positive response to greetings.
In activity 6, different type of input is applied. See figure 4.2. The teacher has
to read aloud the dialog then students have to repeat after him/her. As seen in the
dialogs, new expressions are introduced such as have a headache and got flu. Inputs
are very important for students since they will be able to produce the language.
Learners will not be able to produce adequate outputs if learners receive very few
inputs.
Figure 4. 2 Sample of activity found in textbook: Activity 6

A case where learners cannot produce adequate language outputs in terms of
writing and speaking are very possible to happen in the next activity, activity 8 on
page nine. As you can see, textbook does not give adequate inputs for learners to
produce the language effectively. Only small number of words and expressions are
43
given in activity 5 and 6. In fact, in productive activity on this chapter, textbook
requires students to ask 10 people about their feelings.
Figure 4.3 Sample of activity found in textbook: Activity 8

Textbook does not anticipate any possibility might happen in this activity
such as students ask more vocabularies to express feelings. Textbook doesn’t provide
a rich vocabulary range related to expressing feelings. In contrary, textbook provides
picture expressing feeling without its English word.
Interestingly, the respondents who are teachers see that this textbook is a good
learning resource for teachers. Yet the value of agreement for this statement is low. It
is 2.47.
Related to the flexibility of various teaching and learning styles, respondents
see that this textbook is flexible for both teaching and learning style. However, in fact
teachers must follow the teaching method used in the curriculum. To find further
factual evidence stating about the teaching method used in C13 curriculum, the
44
researcher went thoroughly to the teacher manual. It was found that in C13, teachers
must implement the learning process suggested in the curriculum.
The first learning process is observation. At this beginning process, students
are aimed to read/watch/listen to the examples of the text being learnt directly from
the sources and/ or from the recorded sources by paying attention to the social
function, text structure, linguistics elements and the writing format. Then, teachers
must scaffold the students to imitate/ copy/ use the examples.
Second, students are aimed to questioning what they have observed at the
observation stage. In this stage, teachers must assist the students to question the text
being learnt regarding social function, utterance and text structure, linguistics
elements as well as the writing format. After questioning the text, students are
supposed to learn the social function, utterance and text structure, linguistics elements
and writing format directly from the teacher or other sources.
The third learning process is exploration/experiment. At this point, according
to the brief explanation written in the teacher manual, students are aimed to read/
listen/ watch similar kinds of the text being learnt from other different kind of sources
including text book and guide book by paying attention to the social function of the
text, structure of the text, linguistics elements and the format of the text. However,
the researcher failed to understand what ‘guide book’ means.
The fourth learning process suggested in the manual book is associating/
analyzing. The students are put in groups then asked to study a set of text with their
45
group. Subsequently, they have to indicate the social function, utterance and text
structure, linguistics elements and text format of the texts they are discussed. After
that, teachers must give feed back towards their discussion.
The last stage is communicating. Students are aimed to
read/listen/present/demonstrate/ publicize/speak/recite/watch the texts being learnt.
Then teacher has to give feedback to what they have done on the texts and message
they can get from the text. Eventually, students are asked to share what difficulty and
ease they face in learning the text. They are also asked to share their strategies to deal
with their difficulties.
Back to the diagram, the score of both statements number 6 and 7 is almost
the same, 2.70 and 2.23. The researcher cannot give any further comments on this
finding since the researcher did not conduct any observation which actually can be
one the data to evaluate the teacher’s method and approach in teaching activities.
Therefore, researcher in this study could not give any justification towards textbook
flexibility in terms of teaching and learning styles. However, the researcher will
explain and give a very gentle comment towards this particular topic in the discussion
section of this chapter.
Generally, respondents see the approach of this textbook has met the criteria
rated at 2.71. It means that respondents agree that the approach of this textbook has
met the criteria since the mean of this aspect is more than 2.5.
46
3. Design and Organization of textbook

Various results happen in this criterion that some statements are disagreed and
some others are agreed. The mean of each statement is also varied. Respondents seem
do not agree with that this textbook support media such as CD, workbook, and
teacher’s manual. The mean is 2.47. It means half of respondents agree to the
statement and another half do not. It can happen since textbook support by teacher’s
manual but it does not support CD and workbook.
However, the result shows that respondents still consider that textbook is not
effective as it does not contain workbook and CD(s). It is definitely true that this
textbook is not accompanied with workbook and CD(s) for multimedia support. In
other words, the survey does reflect the reality. Unfortunately, it is an inconvenient
truth that textbook does not fulfill what Tomlinson (2008) suggest that ELT materials
must make use of multimedia to provide reach language experience.
A finding based on the survey raised an intriguing question whether the result
of the survey does represent the reality. The result shows that teachers saw textbook
has a well sequencing in term of complexity. To be surer about the result, the
researcher tried to look closely at the content of textbook. The focus on the
investigation was the spread of textbook and the sequencing of textbook. To make the
explanation become clearer, Chapter 1in textbook is used as an example. See figure
4.4. As it can be seen, students will learn ‘how to greet’, ‘how to thank’, ‘how to take
leave’ and ‘how to apologize’.

47
Table 4.4 Survey Result on the Design and Organization of the Textbook
Disagree Agree
(%) (%) (%) (%)
12. The textbook is supported 11.1 38.9 38.9 11.1 2.47 0.874
with media such as CD,
workbook, and teacher’s
manual.
13. The organization of the 0 50 38.9 11.1 2.58 0.691
textbook is right for the
teacher.
14. The organization of the 0 44.4 50 5.6 2.41 0.844
textbook is right for the
learner.
15. The content is well 0 33.3 66.7 0 2.64 0.477
sequenced on the basis of
complexity.
16. The content is well 0 27.8 66.7 5.6 2.76 0.545
learnability.
17. The content is well 0 27.8 72.2 0 2.70 0.455
usefulness.
18. There is adequate 11.1 33.3 55.6 0 2.41 0.691
recycling.
19. There is adequate review. 5.6 44.4 44.4 5.6 2.47 0.696
20. There is reference section 16.7 61.1 22.2 0 2 0.594
of grammar in the book.
21. The materials promote 0 50 33.3 16.7 2.47 0.775
individual study.
22. It is easy for learners to 0 41.2 52.9 5.9 2.64 0.588
find their way around the
textbook.
23. It is easy for teacher to find 0 38.9 55.6 5.6 2.64 0.588
his/her way around the
textbook.
24. The textbook has good 0 33.3 44.4 22.2 2.88 0.757
layout.
48
Talking about sequence on basis of complexity, this book has fulfilled by
placing the learning materials in sequence since ‘how to greet’ and ‘how to thank’ are
easier than ‘how to take leave’ and ‘to apologize’. The investigation was not finished
to this point. A deeper analysis was done and it was found that the book consists of
18 activities (activity 1 to activity 18). There are five activities teaching and
practicing ‘how to greet’. They are activity 1, 2, 3, 4 and 11.
Figure 4.4 Aim of Chapter 1

Then, there are six activities presenting and practicing ‘how to thank’. They
are activity 5, 6, 7, 8, 14 and 16. ‘How to take leave’ is presented and practiced in 2
activities only, namely activity 9 and 10. Finally, ‘how to apologize’ is also presented
and practiced only in 2 activities, activity 17 and 18.
In the prior paragraph, it seems that there is no problem about sequencing in
terms of content. In contrary, the sequencing of the activity seems to be problematic
since there is an activity which is misplaced. Activity 11 practicing ‘how to greet’,

49
but it is placed in activity number 11. It should be placed in activity 5. Activity 14
and 16 should be placed in activity 9 and 10.
It is proven that actually this book, even the first chapter of this book does not
seem well sequenced in term of complexity. Hence the result of the survey for this
statement does not really represent the fact. As Richards (2001) suggested that in
developing ELT materials, developers must consider the sequencing of the content.
They must determine which content need to be placed prior to other content.
Table 4.5 Complexity Level of the Activity

Chapter Activity Topic Complexity level
(1= lowest level,
5= highest level)
1 1 How to greet 1
2
3
4
5 How to thank 2
6
7
8
9 How to take leave 3
10
11 How to greet 1
12 How to greet and take 1 and 3
leaver
13 How to thank 2
14
15 How to ask for help / favor 5
16 How to thank 2
17 How to apologize 4
18
Another related topic to be discussed is the scope of the content. As Richards
(2001) purposed that material developer must consider the scope of content. It means
how many meetings students need to learn a certain topic. Developers must be able to
determine such detail. They have to know whether or not the time allocation is
50
enough, not enough or too long. In this textbook, it seems that the material developers
didn’t really pay attention to the scope of the content of each chapter. In chapter 1 as
explained above, there are 5 activities to learn and practice ‘how to greet’. The fact,
as it is concluded from the teacher’s manual, 2 activities are taught in one meeting. It
means to learn and practice a simple thing called greeting, students must spend a
week. Surprisingly, chapter 1 is created for 4 up to 5 weeks learning. Take a look at
the table. It is obviously seen that the highest level of complexity indeed has only 2
activities and is taught in 1 meeting only.
Table 4.6 Complexity Level and Number of Activity
Complexity Content in number of number of

level Chapter 1 activity meeting
1 greet 6 2½
2 thank 7 3½
3 take leave 2 1
4 apologize 2 1
5 ask for help 1 1
This section of textbook teaches how to greet and respond the greeting. The
respondents also agree that this textbook does not have adequate recycling in it and
respondents give 2.41 to measure the adequateness of recycling in this textbook. It
shows half of the respondents agree and another half disagree. Textbook does not
have adequate review either according to the result as it is 2.47. The next item should
be paid attention more by the teachers and textbook developer is that the result shows
51
there is no reference section of grammar in this book. Almost all respondents agree
that this textbook is very poor of reference section of grammar. The result was 2. The
last for this category, respondents do not see that this textbook promote individual
study.
4. Content of the textbook

Next result to be elaborated is the result of content of textbook. The result
apparently is pretty shocking that mean for all statements in this criterion were rated
at score 2 meaning that the respondents tended to disagree to each statement. As it is
seen in the table 4.7 on page 52, most teachers responded almost disagree to each
statement. According to the teachers, this textbook does not cover main grammar
item. While Brown (2001) explained in his book that teaching grammar is always
needed since students need to know the correct form and structure of English
sentences. Then he proposed judicious question what kind of techniques would be
best for teaching grammar. Then Brown (2001) suggested referring Fotos (1994) and
Doughty & Williams (1998) who talked about appropriate techniques in teaching
grammar.
To be thoughtful responding to the result of the survey, it is better to take a look
at the textbook itself. What is in it? Does this book really not covering the grammar
item? If it is not so, then does this book use the techniques proposed by Fotos (1994)
and Doughty & Williams (1998)?
This textbook seems to be on track with what both experts suggested about
appropriate teaching grammar techniques. At list the first technique they proposed
52
that it must be embedded in meaningful and communicative context (Fotos 1994,
Doughty & Williams 1998). To be surely informed that this discussion is not
discussing the teaching method. Instead, it is still teaching the material (textbook)
itself, since it seems in this textbook teachers must follow the instruction step by step
told in textbook. Therefore, it is clearly seen in textbook that textbook reflects the
teaching techniques should be used in the class.
Table 4.7 Survey Result on the Content of the Textbook

Disagree (%) (%) Agree
(%) (%)
25. The textbook covers 16.7 61.1 22.2 0 2.05 0.638
the main grammar
item.
26. Material for 5.6 55.6 33.3 5.6 2.35 0.680
vocabulary teaching
is adequate in terms
of quantity.
27. Material for 5.6 50 38.9 5.6 2.41 0.691
vocabulary teaching
is adequate in terms
of range of
vocabulary.
28. The textbook 22.2 50 22.2 5.6 2.05 0.802
includes the material
for pronunciation
work.
29. The material for 55.6 55.6 16.7 0 2 0.766
pronunciation
covers individual
sounds.
30. The material for 16.7 61.1 11.1 11.1 2.11 0.831
pronunciation work
covers word stress.
31. The material for 16.7 61.1 11.1 11.1 2.11 0.831
pronunciation work
covers intonation.
53
As seen in figure 4.5 on page 40, the grammar item is actually taught embedded
in the meaningful and communicative context. Textbook teaches the students how to
construct simple presentence using to be (is, am, are) and using verb for present tense.
This textbook also shows the concept of subject-verb agreement in simple present
sentence.
Another appropriate technique to teach grammar used in this textbook is that
this textbook does not use linguistic terminology.
Figure 4.5 Sample of activity found in textbook

Related to the reading skill, teachers from 10 different schools in Special District
of Yogyakarta said that this textbook does not provide the students adequate
vocabulary in terms of quantity. Therefore, most of the teachers had to find
supplementary materials out of this book to fulfill the need of vocabulary for students
although a textbook supposedly roles to fulfill inadequacy of materials (Ur, 1996).
Teachers also saw that the vocabulary in terms of range is not adequate either.
54
5. Language Skill
Based on Tomlinson (2008), a good ELT material must provide a rich
experience of different genre and text types. While text types can be mainly drawn as
two different types. They are spoken text and written text. Years before, Brown
(2001) strongly stated that in highly literate society like today, hundreds of different
types of written text can be used as reading inputs. He mentioned more than 20
different text types for example reports, editorial, article, novels, short stories, letters,
greeting card, etc. He even believed that the written texts are much more than spoken
ones. Unfortunately, the result of the survey shows that most of teachers who
evaluated When English Rings the Bells agree that the book even does not provide
them to improve students’ reading skill.
To assure that the book does not provide reading inputs, a close reading to
textbook should be done. It is found that four out of eleven chapters in this book
completely do not provide any kind of written text. They are chapter 5, chapter 7,
chapter 9 and chapter 10. Chapter 11 is the only chapter which provides longer dialog
and longer descriptive text comparing to other chapters. Except chapters with no text
at all, they always have lyric text. Some of them have short monolog and very short
dialog. Surprisingly, in chapter 6 the book provides a long lyric with difficult
vocabularies comparing to other chapters with simple lyrics. On page 43 figure 4.6 is
the picture showing kind of texts available in the book.
In conclusion, this book does not really provide enough reading text to make the
students experience different genre of text as mentioned by Tomlinson (2008). Even

55
more this textbook does not provide supplementary materials that make the students
experience the extensive reading as Tomlinson (2008) also stated in his book. It is the
fact that this book does not make the book itself become the source to present
adequate materials for learners (Cunningsworth, 1995).
Table 4.8 Survey Result on the Skills Presented in the Textbook

Disagree Agree
(%) (%)
(%) (%)
32. The textbook 18.8 56.3 25 0 1.88 0.757

covers reading
skills.
35. The textbook 0 27.8 66.7 5.6 2.64 0.836

covers writing
skills.
38. The textbook 16.7 66.7 16.7 0 1.93 0.658

covers listening
skills.
44. The textbook 0 17.6 82.4 0 2.81 0.390

covers speaking
skills.
A similar result shows at the item of listening skill. The questionnaire tried to find
out how teachers see textbook in relation with listening skill. Surprisingly, almost all
teachers see that this textbook does not cover any listening skill for learners. The
survey result reached 1.93. Since the Mean did not reach 2.5, it means teachers or
respondents did not agree to the statement stating that the textbook covers listening
skills. Contrary to the result of the survey, the researcher of this study sees that in fact
textbook does have listening skill for students since any text in the book can used as
56
the input. As it is explained by Krashen (1981) in his input hypothesis that students
can acquire a language when they are given lots of input. The input refers to reading
text or listening text. As in further studies, some resulted in response to the input
hypothesis that there are some characteristics of input which later can be used by ESL
teachers in their teaching. Teachers may modify the input by using caretaker talk
when they read aloud the dialog in textbook. Any dialog in textbook in fact can be
used as the listening input for the learners. Therefore, the researcher in this study
doesn’t see that this textbook does not cover listening skill.
Figure 4.6 Sample of a simple lyric and a very simple dialog

6. Topic Presented in textbook
The last criterion is the topic of textbook. C13 curriculum is well known as a
curriculum with rich of topics in each subject. Other subjects of C13 textbooks are
constructed based on the topic implemented in the base competence. This evaluation
57
is done to find out how far this textbook conceptualizes the topics in the curriculum.
Therefore, to measure the topic of textbook, the researcher gave four statements
related to this criterion. looking closely to the teacher manual. In the manual book, it
is described that textbook covers topics which related to one self and social and
natural environment around the house and school.
Table 4.9 Survey Result on the Topic Presented in the Textbook

Disagree Agree
(%) (%) (%) (%)
47. There is sufficient 0 38.9 55.6 5.6 2.70 0.570
material of genuine
interest to learners.
48. There is enough 5.6 38.9 50 5.6 2.58 0.691
variety of topics.
49. The topic presented is 5.6 38.9 50 5.6 2.58 0.691
content-wise enough
for learner’s language
level.
50. The students are able 0 22.2 66.7 11.1 2.87 0.599
to relate the social
culture context
presented in the
textbook to the use of
English in real life.
51. Female is portrayed 0 33.3 61.1 5.6 2.68 0.582
equally with male.
52. The characters in the 0 11.1 77.8 11.1 3 0.5
textbook represent
people from different
region in Indonesia.
53. Different ethnic 0 22.2 66.7 11.1 2.87 0.599
groups are well
represented.
54. Different groups of 0 16.7 72.2 11.1 2.93 0.555
occupation are well
represented.
55. Other group such as 0 44.4 50 5.6 2.62 0.599
disability is also
represented in the
textbook.
The first is statement is that material of genuine interest to learners is sufficient.
In this statement, the researcher attempts to know whether textbook covers sufficient
58
materials that accommodate students’ genuine interest for example textbook gives
topic which is recently relevant to young students. As shown in the table, it can be
seen that teachers see that textbook has already provided genuine topic to learners.
The teachers agreed on the first statement. However, a double check must be done to
confirm whether this textbook does provide sufficient topic of genuine learner
interest. The researcher needs to confirm it by interviewing two respondents. One the
respondent said:
“colorful, many pictures and … look at this picture [pointing to a page of

textbook] this represents Indonesia. So Indonesia, in my opinion. This is
good. The topic is varied.”
However, the researcher thinks that it is not enough to give the topic around the
school and home. In the information technology era like today, where a social media
does really affect the entire way of living most students today, this textbook has not
inserted yet any such topic in textbook.
Generally, teachers see that this textbook has no issue regarding the topic
presented in textbook. The content of textbook is even content-wise enough for
learner’s language level. In contrary to the survey result, the open question results
indicate that this textbook generally is too easy for the students. Teachers also see that
students are able to relate the social context presented in textbook to the use of
English in the real life.

59
C. Discussion
To make this research more valuable and useful, a fruitful discussion regarding to
textbook and its evaluation using the survey given to 20 teachers in Yogyakarta is
explained in this section. Although the curriculum used in the evaluated textbook is
not currently administered by the Minister of Education any longer, some schools in
Indonesia especially the schools which were selected becoming the pilot school to
administer the C13 are still using it until today. Hence, the researcher believes that
this evaluation will be very useful for teachers and material developers in the future.
In order to make the data more valid, this research also uses triangulation data
obtained from interview done to two randomly selected teachers out of 20
respondents. It aims to confirm whether the survey results are confirmed by teachers
directly.
1. The Aim of the Textbook

First discussion goes to the aim of this textbook. This study tries to see whether
the aim of this textbook corresponds with the core competence, basic competence,
teaching program of the curriculum and the needs of the students. In fact, the result of
the survey shows that generally the aim of textbook doesn’t have any issue to this
point. All respondents view that the aim of textbook agrees with the aim of the
curriculum. To see whether it is true, the researcher tries to check what textbook
really aims for. In the preface of textbook, it is written:
“The English lesson for grade 7 of junior high school students presented in this book was
developed to increase the language competency.” (page iii)
60
This means that the English lesson for 7 grade of junior high school presented in
this book was developed to improve English language competency. Meanwhile, a
high English competence student masters the fourth English skills – listening,
reading, speaking and writing skill. In contrary, the fact obtained from the survey
result and close textbook reading show that this textbook does not supply the students
with enough reading texts and its comprehension tasks as well as listening tasks. As it
has been discussed in the result section, most teachers agree that this textbook does
not provide enough reading text. In the interview, which is used as the triangulation
data, both respondents said that this book is lacking of reading texts and listening
activities.
To clarify that this textbook does not provide enough reading texts and listening
activities, two respondents who were interviewed after filling out the questionnaire
elaborated that:
“Definitely, speaking and writing skills are well covered in this book. But the sample of the
conversation for speaking task is too short and simple. It is not challenging. This book doesn’t
cover listening and reading skill pretty well.” (respondent 1)
“I am a little bit confused to answer this. Well, this book doesn’t have listening section. It is
not good because teachers then must find additional material for listening. It bothers us as
teacher [laughing]. And what else? [laughing].” (respondent 2)
Hence, from the evidence above the researcher strongly believe that this textbook
does not celebrate the aim of the syllabus and more over the aim of the curriculum
since this textbook does not give learners room to improve their reading and listening
skills through this textbook.
However, the researcher believes that the first reason underlies the result on the
aim of textbook since no one can tell whether the aim of a textbook correspond to
61
curriculum until the lesson material presented in each chapter is evaluated. If the
lesson material presented in each chapter is not in line with the aim of textbook, this
means that textbook doesn’t correspond the curriculum. Therefore, the researcher
believes that the adapted questionnaire used as the instrument in this research has no
issues regarding the item on the aim of textbook.
2. The Approach of the Textbook

A surprising finding from the survey happened during some interviews conducted
by the researcher. In some interviews, many of the teachers apparently either did not
understand what approach is or they were drowned in a confusion while being asked
about the difference between approach and method. This situation is also reflected in
the way they answer the open questions in the questionnaire about the approach. In
fact, more than half of the teachers have more than 10 years of teaching experience.
From the situation above, the researcher is confident enough to make a
preliminary conclusion that the quality of the English teacher in Indonesia needs
more attention from the government. This can be assured that the quality of English
teacher in remote area throughout Indonesia is even worse.
Officially the C13 applied a new approach called Scientific Approach (SA) which
is derived from the science education framework (Helena, 2014). As the
consequence, from the teachers’ point of view (taken from the interview), most of
ELT teachers find difficult to follow the curriculum since the ELT teacher must build
the skill to communicate while in other hand the government imposed SA, which is
basically aimed to gain or develop new knowledge.

62
Obviously and many would agree that the curriculum seemed to have much
bigger shockwaves to the teachers than ever before (Helena, 2014). Although C13
was only implemented in some appointed school when this study was conducted,
other schools seemed to have the same tense since it seemed that teachers were not
ready for the new curriculum with its new approach. Specifically in ELT context, C13
adopts a Genre-Based Approach (GBA) which aims to develop the ability to
communicate in both spoken and written language.
Both contradictory results which are explained above may happen for reasons.
One of them is that the respondents might not evaluate the book deeper by looking
thoroughly what is presented in textbook and realize that what are presented in
textbook should be established based on the aim of textbook as suggested by Nunan.
Another reason for this was that the questionnaire itself did not give a clear
instruction and explanation.
3. The Design and Organization of the Textbook

The same case happens to the content of textbook. As it is prescribed in the
preface of textbook as well as the teacher manual book, this book is aimed to help the
students to be able to speak English spontaneously or in guided situation with correct
pronunciation, intonation and word stress in the transactional communication.
Surprisingly, there is no single material or activity assisting the students to be able to
speak with correct pronunciation, intonation and stress found in this textbook. The
teachers have the same opinion as the researcher which is shown by the result of the
survey.
63
Still related to the aim of textbook, in the new version of Bloom’s Taxonomy of
learning, there are six steps of difficulty in learning. The very first step of learning is
remembering. Then it is followed by understanding applying, analyzing, evaluating
and creating.
Figure 4.7 The new version of Bloom's taxonomy

Source:(http://sites.duke.edu/eelandscape/2015/03/31/blooms-taxonomy/)
The developer of this textbook definitely does not consider this taxonomy as the
basis to establish the aim of this textbook. After reading closely to the basic
competence written in the teacher manual book, the researcher found that there are
two major problems occur in the basic competence of C13. First, that is the basic
competence seems does not consider Bloom’s Taxonomy as the basis in formulating
the basic competence. As the evidence, there are only two steps from the Bloom’s
taxonomy are applied in the basic competence of C13. The first is understanding,
then jump to the final step which is creating. So in term of difficulty order, this this
textbook doesn’t really follow the order as it is seen from the survey result as well.
The second issue is about the base competence of textbook. The researcher could
not see that the first four base competences mentioned in the teacher manual have any
64
relation to the English language teaching and learning. To make this clearer, the
translation of the first four base competences are provided as follow:
1. To be grateful of having opportunity to learn English as the means of International

communication.
2.1 To appreciate polite and care attitude in teacher-student communication and among the
students’ communication.
2.2 To appreciate the attitude of being honest, disciplined, self-confident, and responsible in
transactional communication to the teacher and other students.
2.3 To appreciate the attitude of being responsible, caring, helpful in group working, peaceful
in doing functional communication.
To evaluate student base competence mentioned above, in the manual of textbook
it is described the criteria to evaluate the first four base competences. Those four
basic competences are categorized into the aspect of attitude encompassing respect,
honesty, care, bravery, confidence, communication, social awareness, and curiosity.
In each criterion, teachers must evaluate the students using score from 1 (the lowest)
to 5 (the highest). The students can be evaluated using four different kinds of
evaluation. The first is performance assessment which is meant for the learning result
(outcome) and the learning process. It means that teachers must assess student
performance through their learning process and outcome. In the manual, however, is
not explained the specific form of the learning outcome should be applied, neither the
learning process assessment. It means that this curriculum give teachers freedom to
decide the form of assessment they are desiring to administer. The second is In the
manual of textbook, it is also explained. As the consequence, many teachers complain
about the assessment systems since the variables in base competence of number one
and two are difficult to assess. Both variables do not related to the English teaching
65
and learning. However, teachers must evaluate the students using them during the
program.
These three cases indicate that the developer of this textbook does not pay
attention to the basic principles of materials development and to important theories in
developing teaching materials. Lesson units presented in textbook are supposed to be
based on what the aim of textbook has established. Interestingly, most teachers as the
evaluators of this textbook also have the same notion regarding material
development. Because, logically if the teachers saw that the lesson units presented in
textbook had the problem such as no enough reading and listening text or no
pronunciation materials, they then would not say that the aim of textbook did not
have any problem at all. However, fortunately there are still two teachers out of 17
who see that the aim of textbook does have problem. Surely, these two teachers are
highly able to evaluate textbook comprehensively.
Another evidence shows that this textbook is not a good resource for the students
when the teachers answer the open question in the questionnaire. Many of the
respondents say that this textbook has very limited materials. In addition to it,
textbook does not challenge the students. It is considered too easy. This definitely is
against what Piaget has proposed about cognitive development which is divided into
4 stages according to the age.
The typography of the textbook is intangible and the lively colorful pictures seem
to well suit the needs of adolescents for some respondents. However, the number of
the pictures presented in the textbook seemed to be not well suited and tended to be
66
distracter since in the age of junior high school students, usually called as adolescents
(between 11 – 15 years old), the students can learn something more abstract and
containing logical ways. Therefore, for late language learners such as adolescents and
adults, the most cases occur in Indonesia, new vocabulary is easier to learn than new
sounds or new grammar. In contrary, as the result survey shows, this textbook hardly
provides adequate vocabulary in term of quantity. This condition was substantiated
by the respondent as shown in the interview transcript below:
“Oh God, this textbook is definitely under standard. I am really sure that this book was made
in a rush. A very noticeable part is that many typos in this book. And it is too easy for my
students, too many pictures but lack of tasks and texts. The government really didn’t pay
attention to it, I think.” (respondent 2)
“This book is good. It is colorful. Many pictures support the activity. But sometimes I feel it
is too many. There is no listening section I guess and the text is very limited though I add
with my own texts.” (respondent 2)
4. Language Content of the Textbook
The survey result also tells that this textbook does not provide pronunciation
exercise as well as word stress and intonation. According to the survey result, this
textbook do not promote either the vocabulary building or grammar exercise. In
which, again, this proves what Tomlinson (2008) explained that textbooks fail to help
the learners to make full use of the language experience available to them outside the
classroom. The researcher seems to doubt that this textbook was created and
developed based on the theories underlying the material development.
5. Language Skill Presented in the Textbook

Those two survey results are confirmed by other survey result. The teachers
disagree that this textbook provides adequate reading and listening texts for the
students. In result, it is not surprising that teachers do not promote this textbook as a
67
good resource book for the students. In the survey result, the teachers do not agree
either this textbook covers the students need or a good textbook. In fact, Nation
theorizes that material developers must agonize over the four strands to create and
develop a balanced textbook. They have to assure that the learners later can learn
from textbook through the reading and listening materials presented in textbook as
the meaning-focused input. In other word, in Krashen’s (1981) term, this textbook
does not provide enough language input text namely listening and reading text. In
short, this textbook obviously is not considering the language learning and teaching
principles as well as the material development’s principles.
6. The Topic Presented in the Textbook

The research shows that this textbook entitled When English Rings the Bell
presents some positive qualities. This textbook represents both gender groups, various
ethnic group in Indonesia and different religion groups. This textbook has
accommodated the cultural and group diversity in Indonesia. This textbook is also
easy to get because students and teachers can get this book by downloading from the
Indonesia Education Ministry ‘s website. Regarding the topic presented in textbook,
this book also has accommodated the relevance topic to students although in terms of
the content, teachers found that this textbook was too easy for the students at that
time. The writing and speaking skill are adequately presented in textbook according
the survey result.
In spite of the positive quality of textbook, the research also presents the pitfalls
of textbook. It is worth mentioning that the criteria which teachers least satisfied is
68
the skill presented in the textbook and the language content that comprises grammar
reference, recycling and review, and vocabulary. With respect to the big number of
the writing and speaking exercise of this textbook, teachers felt shortcomings of this
textbook in terms of providing listening and reading exercises are real. It seemed that
the developer of this textbook managed to take the emphasize on the productive skills
namely writing and speaking skill. As the matter of fact language learning is
definitely needing the receptive and productive skills in balanced as suggested by
Krashen (1981). On different sub criterion, it was found that this textbook seemed to
not giving enough room for students to use this book independently since it is lacked
of grammar reference. This textbook also shows that there is inconsistency between
the curriculum and the lesson unit presented in textbook. The researcher found that
this textbook was not developed with a strong foundation of basic principles in
developing textbook. Shortly this textbook in some aspects is positively responded
and in some more other aspects negatively responded by the teachers.
Commenting on this research itself, the researcher believes that this research
stands close to other similar research conducted by experts and other scholars in
terms of the method by conducting a survey. It is worth mentioning such as what Litz
(2002) researched in South Korea on a textbook evaluation and ELT management.
Another scholar to mention is Sadehvandi (2012) who also conducted textbook
evaluation by administering a survey in Iranian EFL context.
However, this research unavoidably has shortcomings that need to be improved in
the future. The main shortcoming that is obviously seen and conceptually may raise
69
questions is the validation process. A single validation process, namely expert
judgment, was only undergone for this research which undoubtfully it is still under
the standard of validation process. This research should undergo the validation
process to test the items of the questionnaire. Therefore, in the future the researcher
is validating the items of the questionnaire to administer other same kind of
evaluation research.
CHAPTER V
CONCLUSION
In this very last chapter, the conclusion chapter consists of two parts namely, the
conclusion and suggestions. The conclusion is briefly restating the answer of the
research question which are 1) is the textbook entitled When English Rings the Bells
effective? And 2) What are the strengths and weaknesses of the textbook according to
the survey? In answering the research questions, the researcher found some
drawbacks of the study. Therefore, besides presenting the conclusion, this very final
chapter is also presenting suggestion for future researchers.
A. CONCLUSION
By the end of the day, there are scarcely textbook or materials which are
developed close to the state of perfect. Students, teachers and readers must be able to
locate the pitfalls or the drawbacks of any material, not least this textbook. This
textbook according to the survey result, in some aspects is very effective and
ineffective in some other aspects. To start the concluding chapter, the researcher is
giving the strength of this textbook prior to the drawbacks of the textbook.
The most prominent strength of this textbook would be the attractiveness. This
textbook was designed in a colorful way and illustrated with attractive comic-like
characters. This definitely will impress the students in a good first impression.
However, the attractiveness of a textbook is not the only aspect to determine whether
a textbook is good and effective. It is true that it is quite effective to attract the
students to learn using this textbook. But the teachers must be very critical to the
70
71
students’ response since however, by the end of the day students are autonomously be
able determine whether a book is worth to study of not considering the content and
the usefulness of the book itself. For the Indonesia context, this textbook effectively
accommodates the plurality in terms of culture, ethnic group and religion as well as
gender since it is easily pointed that there are many pictures symbolizing certain
ethnic group, culture, religion and gender. In regard to the discussion of the language
skill presented in the textbook, it seems that writing skill exercises are quite well
developed although the some of the instruction of the tasks need to be revised. The
speaking skill exercises and tasks pretty much show their quality though some
revisions need to conduct.
In some other aspect, the textbook is see as ineffective to the respondents. The
evaluation result on the aim of this textbook shows that the developing of this
textbook did not consider the basic foundation which should be generated in the very
beginning process of its development. Although in some ways around in this textbook
the developers state clearly the basic competence and foundation, the lesson units in
this textbook do not confirm what have been set in the basic competence.
Regarding to the approach, this textbook was written according to the newest
curriculum in Indonesia called C13. The teachers looked over textbook then decided
that this textbook has been in line with what the C13 suggested as approach to be
employed namely scientific approach (SA). However, the results of the evaluation
show that many English teachers in Indonesia especially in junior high schools have
very limited knowledge regarding the approach of teaching. Besides, according to

72
some experts in English language teaching in Indonesia, the approach implemented in
the curriculum is hardly suitable for English teaching and learning process. As the
consequence, this textbook raised confusion among the teachers as it is shown in the
result of the open question questionnaire. The ultimate conclusion regarding approach
of this textbook is that material developers fail to determine and describe rigidly to
the EFL teachers in junior high school in Indonesia.
The respondents of this textbook evaluation appreciate the organization of this
textbook in various results. The respondents rate that this textbook does not hold
enough recycling and review. Even though this textbook is aimed to improve
communicative competence, this textbook was assessed by the teachers as textbook
which does not have grammar reference where students can refer to in their individual
learning. Therefore, teachers do not promote this textbook for independent learning.
In terms of the content of textbook, teachers positively agree that this materials
ware very poorly developed. Besides the teachers couldn’t find any grammar
reference in the textbook, apparently, the grammar items are hardly found in the
textbook. The teachers even got difficulty to conduct their classes using this textbook
since this book does not provide either adequate vocabulary learning in terms of
quantity and range or pronunciation works in terms of individual sounds, word stress
and intonation. As the result, this textbook alone is far from being enough to
becoming the main textbook in the class. As the evidence, all teachers who
participated in this survey gave their students more materials outside the textbook as
the supplementary materials.

73
This textbook is not either completely effective or ineffective. In some aspects of
the textbook, When English Rings the Bells is really effective. While in some more
other aspects of the textbook, it does not support effective teaching and learning
process. Shortly, the researcher sees that there are more aspects of the textbook which
do not support EFL teaching and learning process. Therefore, it is concluded that this
textbook tends to be ineffective one.
B. Suggestions
All the drawbacks resulted in the evaluation of this textbook are believed because
the textbook was not designed based on a strong fundamental grounded theory of
developing materials. Additionally the curriculum itself was prematurely tested and
validated prior to the field implementation. Therefore, this study highly recommends
the curriculum and materials developers to carefully develop them by considering the
most fundamental principles in developing curriculum and materials. In addition, they
also need to consider theories on English language teaching and learning to support
their decision in order to create a reliable and effective textbook.
Considering the trend of Indonesia’s government in keep changing the curriculum
every time a new government is elected, teachers in Indonesia must be able to
conduct autonomous textbook evaluation to make decision in choosing the teaching
and learning materials in their class. They can use the available evaluative questions
such as developed by Cunnisngworth, Nunan, etc. Thus, teachers in Indonesia are not
depending on the unreliable and effective materials. However, to do so, teachers must
improve their knowledge in EFL learning and teaching principles as well as materials
74
development by having focused group discussion among the teachers in certain
region or attending national and international conference and training in teaching
EFL.
The researcher of this textbook suggests to the next researcher to limit and narrow
down the scope of the evaluation. In other words, future researcher must be more
specific to the certain elements of the textbook. If the research will be conducted in a
survey research like this, the future researcher must be able to give clearer instruction
in the questionnaire by giving guideline to fill out the questionnaire. However, the
researcher suggests that future researchers must do a more empirical evaluation
respectively like what Ellis (1997) pinned in her article entitled The Empirical
Evaluation of Language Teachinng Materials using pre-test and post-test to see a
more precise evaluation on the materials although it is believed that there is no
definite yardstick or evaluation system to analyze a book.

75
Bibliography
Alamri, A. A. (2008). An Evaluation of Sixth Grade English Language Textbook for

Saudi Boy's School. Kingdom of Saudi Arabia: King Saud University.
Alderson, J. C. (1992). Guidelines for the Evaluation of Language Education. In A.

Baretta, & J. C. Alderson (Ed.), evaluating second language education.
Cambridge: Cambridge University Press.
Allwright, R. L. (1981). What do we want teaching materials for? ELT Journal, 1, 5-

18.
Brown, H. D. (2001). Teaching by Principles: An Interactive Approach to Language

Pedagogy. New York: Addison Wesley Longman.
Cunningsworth, A. (1995). Choosing Your Coursebook. Oxford: Heinemann.
Doughty, C., & Williams, J. (1998). Focus on Form in Classroom Second Language
Acquisition. New York: Cambridge University Press.
Ellis, R. (1997). The Empirical Evaluation of Language Teaching Materilas. ELT, 36-
42.
Ellis, R. (1998). Evaluation of Communicative Task. Cambridge: Cambridge

University Press.
Fotos, S. S. (1994). Integrating Grammar Instruction and Communicative Language

Use through Grammar Consciousness-raising Tasks. TESOL Quarterly 28, pp.
51-323.
Graves, K. (2000). Designing Language Course: A Guide for Teachers. Boston:

Heinle & Heinle Publishers.
Hamied, F. A. (2014). Curriculum Change: What Does It Mean to Indonesia TEFL?

In H. P. Widodo, & N. T. Zacharias, Recent Issues in English Language
Education: Challenges and Directions (pp. 13-37). Surakarta: UNS Press.
Helena, I. R. (2014, October 7). The English Curriculum: The Paradigm,

Interpretation, and Impementation . The Recent Issues in English Language
Education: Challenges and Directions, pp. 39-64.
76
Hewings, M., & Dudely-Evans, T. (Eds.). (1996). Evaluatiion and Course Design in
EAP. Hertfordshire: Prentice Hall Macmillan.
Hutchinson, T., & Waters, A. (1991). English for Specific Purposes. New York:
Cambridge University Press.
J. Mukundan, V. N. (2011). Developing an English Language TextbookEvaluation

Checklist: A Focus Group Study. International Journal of Humanities and
Social Science, 1(12), 100-106.
Jones, S. (2009). A Retrospective Evaluation of ELT for Korean University

Conversation Course. Seoul.
Karnodi, O. R. (2014). Implementing the Authentic Assessment Embodied in the 2013

English Curriculum. School of Postgraduate Studies. Bandung: Indonesia
University of Education.
Krashen, S. (1981). Second Language Acquisotion and Second Language Learning.

Pergamon Press Inc.
Krashen, S. (1982). Principles and Practice in Second Language Acquisition.

Pergamon Press Inc.
Lawrence, W. P. (2011). Textbook Evaluation: A Framework for Evaluating the

Fitness of the Hong Kong New Secondary School (NSS) Curriculum. Hong
Kong City: University of Hong Kong.
Litz, D. R. (2002). Textbook Evaluation and ELT Management: A South Korean

Case Study. Asian EFL Journal, 48, 1-53.
McDonough, J., Shaw, C., & Masuhara, H. (2013). Materials and Methods in ELT.
Oxford: Balckwell.
Nemati, Z. (2009). Evaluation of an ESL English Coursebook: A Step towards

Systematic Vocabulary Evaluation. Karnataka: Univeristy of Mysore.
Neuman, W. L. (2006). Basic of Social Research: Qualitative and Quantitative

Approaches (second edition). Boston: Pearson Education Inc.
Nunan, D. (1989). Designing Tasks for the Communicative Classroom. Cambridge:

the Press Syndicate of the University of Cambridge.
Rea-Dickinds, P., & Germaine, K. (1992). Evaluation. Oxford: Oxford University

Press.
77
Richards, J. C. (2001). Curriculum Development in Language Teaching. Singapore:

Cambridge University Press.
Rowntree, D. (1997). Making Materials-based Learning Work. London: Kagan Page.
Sadehvandi, M. A. (2012). Textbook Evaluation: EFL Teachers' Perspective on

"Pacesetter Series". ELT Journal, 5, 64-74.
Sadehvandi, M. A. (2012). Textbook Evaluation: EFL Teachers' Perspective on

"Precestter Serries". ELT Journal, 5, 64 - 74.
Sanders. (1992). Evaluating School Programs: An Educator's Guide. Newburry Park:

Corwin Press.
Sheldon, L. E. (1988). Evaluating ELT Textbooks and Materials. ELT Journal, 237-
246.
Textbook Evaluation: EFL Teachers' Perspective on "Pacesetter Series". (2012). ELT

Journal, 5, 64-74.
Tomlinson, B. (2008). English Language Learning Materials: A Critical Review.

New York: Continum International Publishing Group.
Tsiplakides, I. (2011). Selecting an English Coursebook: Theory and Practice. Theory

and Practice in English Studies Volume 1 No 7, pp. 758-764.
Ur, P. (1991). A Course in Language Teaching. Cambridge: Cambridge University

Press.
Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell.
Zamanian, M. R. (2015). Textbook Evaluation ELT Teachers' Perspective on

"Learning to Read English for Pre-University Students". International Journal
of Research Studies in Language Learning, 4, 105-118.
78
Appendices
Appendix 1: Questionnaire Draft 1
Categories Questions YES NO
A Aims and 1 Do the aims of textbook correspond closely with

approaches the aims of the teaching program and with the
needs of the learners?
Apakah tujuan buku ‘When English Rings the Bell’

sejalan dengan tujuan program pengajaran dan
sejalan dengan kebutuhan pembelajar?
2 Is textbook suited to the learning/teaching

situation?
Apakah buku ‘When English Rings the Bell’cocok

untuk situasi belajar mengajar di kelas Anda?
3 Does it cover most of all what is needed?

Apakah buku ‘When English Rings the Bell’
mencakup semua apa yang dibutuhkan?
4 Is it good resource for students and teachers?

merupakan sumber yang bagus bagi murid dan
guru?
5 Is textbook flexible? Does it allow teaching and

learning styles?

fleksibel? Dalam hal ini, apakah buku ini memberi
keleluasaan dalam model 78u8 mpengajaran dan
pembelajaran?
B Design and 1 Is there any other supporting media such as CD,

organization workbook, teacher’s book?
Apakah ada media pendukung lainnya untuk buku
‘When English Rings the Bell’ contohnya seperti
79
CD, buku latihan, atau buku guru?
2 Is the organization right for teachers? (e.g.

According to structures, functions, topics,
skills, etc)
Apakah pengorganisasian buku ‘When English

Rings the Bell’ sudah baik untuk guru? (missal
berdasarkan struktur bahasa (grammar), fungsi,
topik, ketrampilan dll)
Is the organization right for the learners? (e.g.

According to structures, functions, topics,
skills, etc)
Apakah pengorganisasian buku ‘When English

Rings the Bell’ sudah baik untuk pembelajar?
(missal berdasarkan struktur bahasa (grammar),
fungsi, topik, ketrampilan dll)
Is the content sequenced on the basis of

complexity?
Apakah isi buku ‘When English Rings the Bell’

disusun berdasarkan tingkat kerumitan?

‘learnaibility’?
disusun berdasarkan tingkat keterbelajaran?

usefulness?

disusun berdasarkan kegunaan?
6 Is the grading and the progression suitable for

the learners?
Apakah system penilaian dan aktifitas di

dalam buku ‘When English Rings the Bell’ sesuai
80
dengan pembelajar?
Does it allow them to complete the work

needed to meet any external syllabus
requirements?

memberi kesempatan pembelajar untuk
mengerjakan semua tugas-tugas di dalamnya
dalam rangka memenuhi ketentuan-ketentuan
dalam silabus?
5 Is there adequate recycling and revision?

Apakah ada daur ulang materi dan revisi
materi?
6 Are there reference sections for grammar, etc?

Apakah ada bagian referensi untuk grammar
dll?
Is some of the material suitable for individual

study?
Apakah beberapa materi yang ada di dalam
buku ‘When English Rings the Bell’ cocok/sesuai
untuk pembelajaran secara individual?
7 Is it easy to find your way around textbook?

Apakah mudah menggunakan buku ini?
Is the layout clear?
Apakah layout buku ini jelas?
C Language 1 Does textbook cover the main grammar items

content appropriate to each level, taking learners’ need
into account?
Apakah buku ini mencakup item grammar
yang sesuai dengan level, yang
81
mempertimbangkan kebutuhan murid?
2 Is material for vocabulary teaching adequate

in terms of quantity and range of vocabulary,
emphasis placed on vocabulary development,
strategies for individual learning?
Apakah materi untuk pengajaran kosakata
cukup memadai dalam hal kuntitas dan range
kosakata, penekanan pada pengembangan
kosakata, strategi untuk pembelajaran
mandiri?
3 Does textbook include material for

pronunciation work?
Apakah buku ini memasukkan materi untuk
pengnucapan?
If yes, does the pronunciation work cover

individual sounds?
Jika iya, apakah materi tersebut mencakup

bunyi yang berdiri sendiri?
Does the pronunciation work cover word

stress?
Apakah materi tersebut mencakup penekanan

pada kata?
Does the pronunciation work cover sentence

stress?
Apakah materi tersebut mencakup penekanan

pada kalimat?
Does the pronunciation work cover

intonation?
Apakah materi tersebut mencakup intonasi?
82
4 Does textbook deal with the structuring and

conventions of language use above sentence
level, eg. How to take part in conversations,
how to structure a piepce of extended writing,
how to identify the main points in a reading
passage? (More relevant at intermediate and
advanced levels.)
5 Are style and appropriacy dealt with? If so, is

language style matched to social situation?
Categories Questions YES NO
D Skills 1 Are all four skills adequately covered,

bearing in mind your course aims and
syllabus requirements?
Apakah keempat skill berbahasa

secara proporsional tecakup dalam
buku ini dengan mempertimbangkan
tujuan pembelajaran di kelas Anda dan
persyaratan silabus?
2 Is there material for integrated skill

work?
Adakah materi di dalam buku ini

dengan skil terpadu (integrated)?
3 Are reading passages and associated

activities suitable for your students’
levels, interests, etc?
Apakah teks bacaan dan latihan-
latihannya sesuai dengan :
a. level pembelajar di kelas Anda?
b. minat pembelajar di kelas Anda?
Is there sufficient reading material?

83
Adakah meteri bacaan yang

mencukupi?
4 Is listening material well recorded, as

authentic as possible, accompanied by
background information, questions and
activities which help comprehension?
Apakah materi mendengarkan

(listening) :
a. direkam dengan baik?
b. otentik?
c. disertai infomasi pendahuluan?
d. disertai pertanyaan dan aktifitas
yang membantu pemahaman?
5 Is material for spoken English

(dialogues, roleplays, etc) well
designed to equip learners for real-life
interactions?
Apakah materi untuk percakapan

bahasa Inggris ( seperti dialog dan
role play) dirancang dengan baik
untuk membekali pembelajar di
kehidupan nyata?
6 Are writing activities suitable in terms

of amount of guidance/control, degree
of accuracy, organization of longer
pieces of writing (eg paragraphing) and
use of appropriate styles?
Apakaha aktifitas menulis sudah sesuai

dalam hal
a. jumlah panduan/control
b. tingkat keakuratan
84
c. pengorganisasian panjang
pendeknya tulisan
d. penggunaan gaya penulisan?
Categories Questions Yes No
E Topics 1 Is there sufficient material of genuine

interest to learners?
Adakah materi yang cukup berkenaan
dengan minat asli untuk pembelajar?
2 Is there enough variety and range of

topic?
Adakah variasi dan rentangan topic

yang cukup ada di buku ini?
3 Will the topics help expand students’

awareness and enrich their experience?
Akankah topic yang disajikan
membantu memperluas kesadaran
pembelajar dan memperkaya
pengalaman mereka?
4 Are the topics sophisticated enough in

content, yet within the learners’
language level?
Apakah topic yang disajikan cukup

canggih/rumit/sulit dalam hal:
a. isi?
b. tingkat penguasaan bahasa para
pembelajar?
5 Will your students be able to relate the

social cultural contexts presented in the
course book?
85
Akankah pembelajar Anda mampu

menghubungkan konteks sosial budaya
yang disajikan dalam buku ini?
6 Are women portrayed and represented

equally with men?
Apakah figure perempuan

digambarkan dan ditampilkan
seimbang dan sepadan dengan figur
laki-laki?
7 Are other groups represented, with

reference to ethnic, origin, occupation,
disability, etc?
Apakah kelompok minoritas dari:
a. suku
b. ras
c. pekerjaan
d. orang-orag cacat
e. agama
juga direpresentasikan?
F Methodology 1 What approach/approaches to language

learning are taken by textbook? Is this
appropriate to the learning/teaching
situation?
Pendekatan pemebelajaran bahasa apa
saja yang digunakan di dalam buku
ini?
Apakah pendekatan ini sesuai untuk
situasi pembelajaran dan pengajaran
di kelas Anda?
86
2 What level of active learner

involvement can be expected? Does
this match your students’ learning
styles and expectations?
Level pembelajar apa yang bisa secara

aktif terlibat dalam pembelajaran
menggunakan buku ini?
3 What techniques are used for

presenting/practicing new language
items? Are they suitable for your
learners?
Teknik apa yang digunakan untuk

mempresentasikan/ mempraktekkan
item bahasa yang baru?
4 How are the different skills taught?

Bagaimana setiap skill berbahasa
diajarkan?
Membaca:
Menulis:
Mendengarkan:
Berbicara:
5 How are communicative abilities

developed?
Bagaimana kemampuan berkomunikasi
dikembangkan di dalam buku ini?
6 Does material include any advice/help

to students on study skills and learning
strategies?
Apakah buku ini juga memasukkan
tips/saran kepada pembelajar dalam
87
mempelajari skill dan strategi belajar?
7 Are students expected to take a degree

of responsibility for their own learning
(eg by setting their own individual
learning targets)?
Apakah ada di antara

pembelajar/murid yang mempunyai
target pribadi mereka dalam hal
belajar bahasa Inggris? (missal
setelah lulus harus sudah bisa lancer
berbahasa Inggris)
G Teacher’s 1 Is there adequate guidance for the

books teachers who will be using textbook
and its supporting materials?
Apakah ada panduan yang menuckupi

untuk para guru yang akan
menggunakan buku ini?
Dan adakah materi-materi
pendukungnya?
2 Are the teachers’ books comprehensive

and supportive?
Apakah buku guru tersebut lengkap
dan membantu?
3 Do they adequately cover teaching

techniques, language items such as
grammar rules and culture-specific
information?
Apakah buku guru dan materi

pendukung mencakup:
a. teknik pengajaran?
b. item kebahasaan seperti grammar?
88
c. serta catatan informasi budaya

secara spesifik?
4 Do the writers set out and justify the

basic premises and principles
underlying the material?
Apakah penulis penulis buku guru

tersebut memeberi pernyataan dan
penilaian tentang aturan dasar dan
prinsip dasar tentang buku murid ini?
5 Are keys to exercise given?

Apakah kunci jawaban latihan
diberikan di buku guru?
H Practical 1 What does the whole package cost?

considerations Does this represent good value for
money?
Berapa harga buku ini?
Apakah gratis?
2 Are the books strong and long lasting?

Are they attractive in appearance?
Apakah buku ini akan tahan lama?
Apakah buku ini cukup menarik secara
penampilan?
3 Are they easy to obtain?

Apakah buku ini mudah didapatkan?
4 Do any parts of the package require

particular equipment, such as a
language laboratory, listening center or
video player? If so, do you have the
equipment available for use and is it
89
reliable?
Apakah di antara bagian di dalam
buku ini membutuhkan peralatan
khusus lainnya seperti laboratorium
bahasa, listening center atau videa
player?
Jika iya, apakah sekolah Anda

mempunyainya?
90
Appendix 2: Questionnaire Draft 2

A Questionnaire of
English Textbook Evaluation of Curriculum 2013
A. Statements
Please, tick accordingly.
1=strongly disagree, 2=disagree , 3= agree, 4=strongly agree
Statements 1 2 3 4
1. The aims of textbook correspond with the aim of the curriculum.
2. The aim of textbook correspond the core competence (KI)
3. The aims of textbook correspond with the aim of the teaching

program.
4. The aims of textbook correspond with the needs of learners.
5. Textbook is suited to the learning situation.
6. Textbook is suited to the teaching situation.
7. Textbook covers most of all student needs.
8. The courebook is a good learning resource for students.
9. Textbook is a good learning resource for teachers.
10. Textbook is flexible so it allows various teaching styles.
11. Textbook is flexible so it allows the learning styles.
12. The book support media such as CD, workbook, and teacher’s
manual.
13. The organization of the book is right for the teacher. (for
example the organization is based on the structure order, topic
based order, or difficulty based order)
91
14. The organization of the book is right for the learner. (for
example the organization is based on the structure order, topic
based order, or difficulty based order)
15. The content is well sequenced on the basis of complexity.
16. The content is well sequenced on the basis of learnability.
17. The content is well sequenced on the basis of usefulness.
18. There is adequate recycling. (Example of recycling is an

expression such as refusal “I am sorry, I am afraid I can’t” is
presented in unit 1 and it is repeated in unit 3, 6 and 8.
19. There is adequate review. (For example there is always review

after finishing 3 units)
20. There is reference section of grammar in the book.
21. The materials promote individual study.
22. It is easy for learners to find their way around textbook.
23. It is easy for teacher to find his/her way around.
24. The book has good layout.
25. Textbook covers the main grammar item.
26. Material for vocabulary teaching is adequate in terms of

quantity.
27. Material for vocabulary teaching is adequate in terms of range of

vocabulary.
28. Textbook includes the material for pronunciation work.
29. The material for pronunciation work covers individual sounds.
30. The material for pronunciation work covers word stress.
31. The material for pronunciation work covers intonation.

92
Continued
1=strongly disagree, 2=disagree, 3= agree, 4=strongly agree
Statements 1 2 3 4
32. Textbook covers the reading skill.
If your respond for this statement is Agree (3) or Strongly Agree (4), and then please
answer number 33 and 34.
33. The level of difficulty of the reading passages is appropriate to

the students.
34. The level of associated activities for reading passages is

appropriate to the students.
35. Textbook covers the writing skill.

If your respond for this statement is Agree (3) or Strongly Agree (4) and then please
36. The level of writing activities is appropriate to the students.
37. The writing activities are suitable in terms of amount of

guidance/control.
38. Textbook covers the listening skill.

answer number 39 up to 43.
39. The level of listening texts is appropriate to the students.
40. The listening material is well recorded.
41. The listening material is authentic.
42. The listening material is accompanied by background

information which helps comprehension.
43. The listening material is accompanied by question and activities

which help comprehension.
93
44. Textbook covers the speaking skill.

45. The level of speaking activities is appropriate to the students.
46. The material for speaking activities such as dialogues and

roleplays is well designed to equip students for real-life
interactions?
47. There is sufficient material of genuine interest to learners.
48. There is enough variety of topics.
49. The topic presented is content-wise enough for the learner’s

language level.
50. The students are able to relate the social cultural context
presented in textbook to the use of English in the real life.
51. Female is portrayed equally with male.
52. The characters in the book represent people from different region
in Indonesia
53. Different ethnic groups are well represented.
54. Different groups of occupation are well represented.
55. Other group such as disability is also represented in textbook.
Continued
1=strongly disagree, 2=disagree, 3= agree, 4=strongly agree
Statements 1 2 3 4
56. The teacher’s manual is comprehensive.
57. The teacher’s manual is supportive.
58. The teacher’s manual adequately covers teaching technique.

94
59. The teaching technique in the teacher’s manual book in line with
what the curriculum proposes.
60. The teacher’s manual adequately covers language items such as

grammar and pronunciation.
61. The teacher’s manual book adequately covers culture-specific

information.
62. The keys to exercises are given in the teacher’s manual book.
63. The book is affordable for students.
64. The book is affordable for the haves-not students.
65. The book is strong and long lasting.
66. The book is easy to obtain.
67. Some parts of textbook require a language laboratory.
68. Some parts of textbook require listening center.
69. Some parts of textbook require video player.
70. Your school has the equipment available for the proper use of
the book.
71. The equipment your school has is reliable.
B. Open Questions
1. What approach/approaches to language learning are taken by textbook?
2. Is the approach to language learning taken by textbook appropriate for your class
situation? Why? (Please give a brief explanation of the reason.)
95
3. What approach/approaches to language teaching do you use?
4. What level of active learner involvement can be expected from textbook? Does this
match your students’ learning styles and expectations?
5. What techniques are used for presenting/practicing new language items? Are they
suitable for your learners?
6. How are the different skills taught? Explain briefly.

a. Reading
b. Writing
c. Listening
96
d. Speaking
7. How are communicative abilities developed?
8. In your opinion, what is the major problem of this textbook? (Please give a brief
explanation)
9. Do you always make lesson plan for every single lesson for you class? If not, why?
10. Do you think that lesson plan is important? Why or why not?
11. If the book is accompanied by a set of lesson plan for each unit to help the teacher,
do you think that you will use it or adapt it? Why or why not?
97
98
BLUEPRINT
Appendix 3: The Blueprint of the Questionnaire
A Questionnaire for Teachers to Evaluate
A Textbook Entitled When English Rings the Bell
Dear participants,
This questionnaire is a part of a research for attaining a Master degree in English
Language Studies, Sanata Dharma University. The research is to find out whether
textbook proposed by the Indonesia Education Ministry entitled When English Rings
the Bells is effective according to the result survey to teachers who are applying the
C13 curriculum and using this book in their class. The aim of this questionnaire is for
the participants to check give evaluation to each statement according to the
experience and knowledge as English teacher of Junior High School in Indonesia.
There are two parts of the questionnaire should be fill out. Part A is the statement in
which the participant must tick to each statement accordingly. Part B is the open
questions. In this part, participant must answer the questions briefly and clearly by
writing the answer at the provided blank space.
Name :
School :
Experience
(in year) :
C. Statements
Please, tick accordingly.
1=strongly disagree, 2=disagree , 3= agree, 4=strongly agree
Statements 1 2 3 4
72. The aims of textbook correspond with the aim of the

curriculum.
73. The aim of textbook correspond the core competence (KI)
74. The aims of textbook correspond with the aim of the

teaching program.
99
75. The aims of textbook correspond with the needs of

learners.
76. Textbook is suited to the learning situation.
77. Textbook is suited to the teaching situation.
78. Textbook covers most of all student needs.
79. The courebook is a good learning resource for students.
80. Textbook is a good learning resource for teachers.
81. Textbook is flexible so it allows various teaching styles.
82. Textbook is flexible so it allows the learning styles.
83. The book support media such as CD, workbook, and

teacher’s manual.
84. The organization of the book is right for teacher (for

example the organization is based on the structure order,
topic based order, or difficulty based order).
85. The organization of the book is right for the learner (for
example the organization is based on the structure order,
topic based order, or difficulty based order).
86. The content is well sequenced on the basis of complexity.
87. The content is well sequenced on the basis of learnability.
88. The content is well sequenced on the basis of usefulness.
89. There is adequate recycling. (Example of recycling is an

expression such as refusal “I am sorry, I am afraid I can’t”
is presented in unit 1 and it is repeated in unit 3, 6 and 8.
90. There is adequate review. (For example there is always

review after finishing 3 units)
100
91. There is reference section of grammar in the book.
92. The materials promote individual study.
93. It is easy for learners to find their way around textbook.
94. It is easy for teacher to find his/her way around.
95. The book has good layout.
96. Textbook covers the main grammar item.

quantity.

range of vocabulary.
99. Textbook includes the material for pronunciation work.
100. The material for pronunciation work covers

individual sounds.
101. The material for pronunciation work covers word

stress.
102. The material for pronunciation work covers

intonation.
103. Textbook covers the reading skill.
If your response for this statement is Agree (3) or Strongly Agree

(4), and then please answer number 33 and 34.
104. The level of difficulty of the reading passages is

appropriate to the students.
105. The level of associated activities for reading

passages is appropriate to the students.
106. Textbook covers the writing skill.

101
If your response for this statement is Agree (3) or Strongly Agree

(4) and then please answer number 36 and 37.
107. The level of writing activities is appropriate to the

students.
108. The writing activities are suitable in terms of

amount of guidance/control.
109. Textbook covers the listening skill.

If your response for this statement is Agree (3) or Strongly
Agree (4) and then please answer number 39 up to 43.
110. The level of listening texts is appropriate to the

students.
111. The listening material is well recorded.
112. The listening material is authentic.
113. The listening material is accompanied by

background information which helps comprehension
114. The listening skill material is accompanied by

question and activities which helps the comprehension.
115. Textbook covers speaking skill

If your response for this statement is Agree (3) or Strongly
Agree (4) and then please answer number 45 and 46.
116. The level of speaking activities is appropriate to the

students.
117. The material for speaking activities such as

dialogues and roleplays is well designed to equip students
for real-life interactions?
118. There is sufficient material of genuine interest to

learners.
119. There is enough variety of topics.

102
120. The topic presented is content-wise enough for the

learner’s language level.
121. The students are able to relate the social cultural

context presented in textbook to the use of English in the
real life.
122. Female is portrayed equally with male.
123. The characters in the book represent people from

different region in Indonesia
124. Different ethnic groups are well represented.
125. Different groups of occupation are well represented.
126. Other group such as disability is also represented in

textbook.
127. The teacher’s manual is comprehensive.
128. The teacher’s manual is supportive.
129. The teacher’s manual adequately covers teaching

technique.
130. The teaching technique in the teacher’s manual

book in line with what the curriculum proposes.
131. The teacher’s manual adequately covers language

items such as grammar and pronunciation.
132. The teacher’s manual book adequately covers

culture-specific information.
133. The keys to exercises are given in the teacher’s

manual book.
134. The book is affordable for students.

103
135. The book is affordable for the haves-not students.
136. The book is strong and long lasting.
137. The book is easy to obtain.
138. Some parts of textbook require a language

laboratory.
139. Some parts of textbook require listening center.
140. Some parts of textbook require video player.
141. Your school has the equipment available for the

proper use of the book.
142. The equipment your school has is reliable.
D. Open Questions
12. What approach/approaches to language learning are taken by textbook?
13. Is the approach to language learning taken by textbook appropriate for your
class situation? Why? (Please give a brief explanation of the reason.)
14. What approach/approaches to language teaching do you use?
15. What level of active learner involvement can be expected from textbook?
Does this match your students’ learning styles and expectations?
16. What techniques are used for presenting/practicing new language items? Are
they suitable for your learners?
104
17. How are the different skills taught? Explain briefly.

e. Reading
f. Writing
g. Listening
h. Speaking
18. How are communicative abilities developed?
19. In your opinion, what is the major problem of this textbook? (Please give a
brief explanation)
20. Do you always make lesson plan for every single lesson for you class? If not,
why?
21. Do you think that lesson plan is important? Why or why not?
105
22. If the book is accompanied by a set of lesson plan for each unit to help the
teacher, do you think that you will use it or adapt it? Why or why not?
Appendix 4: The Respondents’ on Statement
respondent
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
Experience 16 - 19 15 - 20 7 16 10 10 10 20 26 14 17 35 2
(years)
Statement
1. 3 3 3 3 3 3 2 3 2 4 3 4 4 4 3 3 3
2. 3 3 3 3 3 3 2 3 2 3 3 4 4 4 3 2 3
3. 3 3 2 3 3 3 2 3 2 3 3 4 4 4 3 3 3
4. 3 3 2 3 3 3 2 2 2 3 2 4 3 3 3 3 3
5. 3 3 3 3 3 2 1 3 2 3 4 2 3 3 3 2 3
6. 3 3 3 3 3 3 1 2 2 3 4 2 3 3 3 2 3
7. 3 3 2 2 2 2 1 2 2 3 2 2 3 2 2 2 3
8. 3 3 1 3 3 2 2 2 2 3 4 2 3 2 2 2 3
9. 3 3 3 3 3 2 2 2 2 2 4 2 3 2 3 2 3
10. 3 3 2 3 3 2 2 3 2 3 4 2 3 3 3 3 4
11. 3 3 3 3 3 2 2 2 2 3 4 2 3 3 3 3 3
12. 3 3 3 2 3 2 1 2 2 3 4 3 4 1 2 2 2
13. 3 3 2 2 3 2 2 2 2 3 4 3 4 3 2 2 2
14. 3 3 2 3 3 2 2 2 2 3 4 3 3 2 2 2
15. 3 3 2 3 3 2 2 3 2 3 3 3 3 3 3 2 2
16. 3 3 3 3 3 2 2 3 2 3 3 3 4 3 3 2 2
17. 3 3 3 3 3 2 2 3 2 3 3 3 3 3 3 2 2
18. 3 2 3 3 3 2 3 3 2 3 1 2 3 1 3 2 2
19. 3 2 3 2 2 2 3 3 2 3 3 2 4 1 3 2 2
20. 3 2 2 2 2 2 1 3 2 2 2 2 3 1 2 1 2
106
21. 2 3 2 3 2 2 2 3 2 3 2 2 4 1 3 2 4
22. 3 2 2 3 2 2 2 3 3 3 3 2 4 3 3 2 3
23. 3 3 2 3 2 2 2 3 2 3 3 2 4 3 3 2 3
24. 3 3 2 4 4 2 4 3 2 3 3 2 4 3 2 2 3
25. 2 2 3 3 2 2 1 3 2 2 2 2 3 1 2 1 2
26. 3 2 2 2 2 2 1 2 3 3 4 2 3 2 3 2 2
27. 3 2 2 3 2 2 1 2 3 3 4 2 3 2 3 2 2
28. 3 2 1 2 2 2 1 2 2 2 3 2 4 2 3 1 1
29. 3 2 1 2 2 2 1 2 2 2 2 2 4 2 3 1 1
30. 3 2 2 2 2 2 1 2 2 2 4 2 4 2 2 1 1
31. 3 2 2 2 2 2 1 2 2 2 4 2 4 2 2 1 1
32. 3 2 2 2 2 2 1 3 0 2 2 2 3 1 2 1 2
33. 3 0 0 0 0 0 0 3 0 0 0 0 4 0 0 0 0
34. 3 0 0 0 0 0 0 3 0 0 0 0 4 0 0 0 0
35. 3 3 3 3 3 2 2 3 0 3 3 2 4 3 3 2 3
36. 3 3 4 3 3 0 0 3 0 3 4 0 4 3 3 0 3
37. 3 3 3 3 2 0 0 3 0 3 3 0 3 3 3 0 3
38. 2 2 3 3 2 2 2 2 0 2 1 2 2 1 3 1 1
39. 0 0 3 3 0 0 0 0 0 0 0 0 0 0 3 0 0
40. 0 0 2 2 0 0 0 0 0 0 0 0 0 0 3 0 0
41. 0 0 3 2 0 0 0 0 0 0 0 0 0 0 2 0 0
42. 0 0 2 2 0 0 0 0 0 0 0 0 0 0 2 0 0
43. 0 0 3 2 0 0 0 0 0 0 0 0 0 0 2 0 0
44. 3 3 3 3 3 2 2 3 0 3 3 3 3 3 3 2 3
45. 3 3 3 3 3 0 0 3 0 3 4 3 3 3 3 0 3
107
46. 3 2 2 2 2 0 0 3 0 3 4 2 3 3 3 0 3
47. 3 2 2 3 2 3 2 3 3 3 4 2 3 3 3 2 3
48. 3 2 3 3 3 3 2 2 2 3 4 2 3 2 3 1 3
49. 3 3 2 3 3 3 2 3 2 3 4 2 3 2 3 1 2
50. 3 3 3 3 3 2 2 3 3 0 4 2 4 3 3 2 3
51. 3 3 2 3 3 2 2 3 2 0 3 3 4 3 3 2 2
52. 3 3 2 3 3 2 4 3 3 0 3 3 4 3 3 3 3
53. 3 3 2 3 3 2 4 3 3 0 2 3 4 3 2 3 3
54. 3 3 3 3 3 2 4 3 2 0 2 3 4 3 3 3 3
55. 3 3 3 2 2 2 4 3 3 0 2 2 3 2 2 3 3
56. 3 3 3 3 3 2 2 3 3 0 4 3 3 1 3 3 3
57. 3 3 3 3 3 2 2 3 3 0 4 3 3 1 3 3 3
58. 3 3 3 3 3 2 2 3 3 0 4 3 4 1 3 2 2
59. 3 3 3 3 3 2 2 3 3 0 4 3 3 1 3 2 2
60. 3 2 2 2 2 2 2 3 3 0 3 2 4 1 2 3 3
61. 3 2 2 3 2 0 1 3 3 0 4 2 4 1 3 2 3
62. 2 3 4 2 2 2 1 1 3 0 4 2 3 1 2 2 2
63. 2 3 4 3 3 3 3 2 2 0 4 3 3 3 3 3 3
64. 2 3 3 3 3 3 3 2 2 0 4 3 3 3 3 3 3
65. 2 3 3 3 3 2 1 2 2 0 4 3 3 1 2 3 3
66. 2 3 1 3 3 3 4 2 2 0 4 3 3 3 3 3 3
67. 2 2 3 3 1 3 3 2 2 0 3 2 4 2 3 2 2
68. 2 2 1 3 1 2 1 2 2 0 4 2 3 2 3 2 2
69. 2 3 1 4 2 2 2 2 2 0 3 2 3 2 3 2 2
70. 2 3 2 3 3 2 4 2 2 0 4 3 3 4 3 3 3
108
71. 2 3 1 3 3 3 4 2 2 0 3 3 3 4 3 3 3
Note:
1. Zero (0) means the participant didn’t respond to the statement.

2. One (1) means the participant strongly disagree with the statement.
3. Two (2) means the participant disagree with the statement.
4. Three (3) means the participant agrees to the statement.
5. Four (4) means the participant strongly agrees to the statement.
109
Appendix 5: The Respondents’ Answer on Open Question Questionnaire
What What Is the approach to In your opinion

Respondent
Experience
approach/appr approach/appro language learning what is the
oaches to aches to taken by the major problem
language language coursebook of this
learning is teaching do you appropriate for you coursebook?
taken by the use? class situation? Why?
coursebook?
1 - Scientific Scientific The steps are okay but It is not enough

approach approach the forms of task in if we as teachers,
some ways are not solely use the
suitable. As teachers, I book as a source.
have to find other tasks Teachers should
which are suitable for make/create
my students. Most of other learning
tasks are not experiences for
challenging for my their students in
students. the form of task
and practice.
2 20 Scientific Scientific Yes, because the The content of

approach approach students’ English the book is too
competence is above easy for the
the average. students. As the
teachers, we
need much
sources for the
teaching
materials.
3 7 Scientific Scientific Yes, because the book The coursebook

approach approach is too easy for the is too easy for
students of SMP 1 our student so
Sleman. So the teacher we need another
must make another sources to teach
work sheet to our student. We
influence their make worksheet
understanding. to influences the
students’
knowledges.
4 26 - The students are Because it contains the Students can

asking actively picture which suitable only do exercise
with the situations of if they are
explained by
110
our area. their teacher.
5 14 Based on the Various Yes. Because a text- This coursebook

statement in the approaches based approach is too simple. In
coursebook, the provides a very wide other words, the
approach is And space for the teacher book is not so
text-based communicative and students to vary deep and doesn’t
learning. approach their class with any give a challenge
creative method. to the students.
6 35 Scientific Scientific No, too simple. There No text. So the

approach approach are no task for students have no
exposure. chance to
observe.
7 2 Scientific Scientific - It’s used some

approach approach text so student
can practice their
reading
(pronunciation).
But using some

pictures can
make student
explore more
information from
theirself.
8 20 The curriculum So far, based on Quite appropriate as There are no

suggests that all KTsP I use far as the teachers are clear tasks
subjects are genre-based creative enough to designed to
taught based in approach to create students tasks develop which
scientific language by themselves, skills. This book
approach i.e. teaching. This because the presents mostly
observing, approach is used coursebook doesn’t pictures and
questioning, to teach year 8 display any clear steps simple
exploring, and 9. For year 7 or tasks for students. expressions or
associating, and I am suing the sentences with
communication. Scientific no clear
This time being, approach as purposes until
I am still trying recommended by the teachers refer
to find the link the curriculum. themselves to the
between this teacher’s book.
approach and
language
learning
approach I’ve
111
known.
9 11 Scientific Both Scientific Not really. Sometimes It is lack of texts.

learning learning and my students still need
three phase to be guided in
technique understanding the
material. For example
teacher must write the
explanation on the
board, etc.
10 17 The book takes Communicative Yes it is. The text- The materials is
a text-based and text-based based approach no so deep and
learning approach. provides the teaches to doesn’t give
approach. use various methods. more challenge.
11 10 The coursebook I think, we will No, it is not. The coursebook

use Speech be better if we doesn’t complete
approach use easy Because many students
approach (like don’t know the The materials of
pictures, CD and knowledge of English the coursebook
video) especially the student are limited.
that have low
understanding in The arrangement
English when they are of the
in SD. coursebook is
not good.
12 10 Discovery Discovery No, because the I think, the

learning, project learning for approach to language coursebook of
based learning, reading and learning in general 2013 curriculum
communicative writing. appropriate of the is more colorful
learning (CLT), Communicative material. picture so it is
role play, CTL learning for suitable for
teaching. speaking. speaking.
But in teaching
reading and
writing is not
suitable. So my
problems are in
teaching reading
and writing if
used this
coursebook of
the ring bells. So
I used the others
book to teach
112
reading and
writing.
13 16 Scientific Scientific Yes, because it is not Limited

approach approach only transfer materials
knowledge from
teacher to student but There are not
student can see , ask example of
and analize what conversation
teacher present. enough.
14 19 Scientific Scientific Yes it is but not There is no

approach approach always. It depends on grammar focus.
the materials. There is no
listening and
speaking.
15 15 Scientific I use active I think it is appropriate The book covers

approach learning but sometimes it minimal
approach. I try to depends on the topic of material, so the
Communicative make the learning. teacher has to
approach students active find other
in learning, and Using 5 steps in resources to
the most learning (observing, develop the
important thing questioning, material.
that they can associating,
apply it in their experimenting, and
real life. communicating/
networking) the
Sometimes I also students can learn
use co-operative more active and enjoy
and collaborative the learning.
learning.
16 16 Scientific Scientific Yes, sesuai. Membuat Masih ada

approach approach anak lebih creative. beberapa
kekurangan/
kelemahan
diantara:
Skill listening
tidak ada
Skill reading
kurang
113
17 10 Learning based Contextual Yes, it is. Because It is very simple

on text. Speech language students can because our
or written by an teaching comprehend easily if students have
English in language learning had the materials
language as a use learning by texts from primary
communication school.
language
114
Appendix 6: Translated transcript of the interview

Interview 1
Researcher: What is your comment towards English textbook for grade 7
entitled When English Rings the Bell?
Respondent 1: Oh my God, this textbook is definitely under standard. I am really
sure that this book was made in a rush. A very noticeable part is that many typos
in this book. And it is too easy for my students, too many pictures but lack of
tasks and texts. The government really didn’t pay attention to it, I think. Really…
I can’t believe it. At the end, as usual… teachers must work harder to find more
materials which are suitable for the students. There are too many problems in this
book AND the curriculum itself.”
Researcher: What is wrong with the curriculum?
Respondent 1: Well, the major problem is the approach. In the curriculum it is
clearly stated that teachers must apply the scientific approach. However, I never
heard of such approach in language teaching and learning. Neither have other
teachers. We are in the big confusion because we are mandated to apply what the
curriculum says but in other hand we ARE aware that such approach doesn’t exist
in language teaching and learning.
Secondly, one of the scoring aspects for final examination is way too much. How
can we measure the attitude through English language learning process? Yes we
are equipped with teacher’s guide. However, the implementation is way more
difficult than what it is prescribed in the teacher’s manual. Too many aspects that
are not correlated to language learning need to be measured. Teachers are
overwhelmed.
Researcher: What does this book lack?
Respondent 1: Hmm,, obviously, this book lacks reading texts and listening tasks.
Do you agree, mas? Right? If you take a look at it closely, this book is lacked of
reading texts. And there is no listening section. So teachers must prepare them all.
Researcher: So do you think that this book doesn’t cover all skills?
Respondent 1: Definitely, speaking and writing are well covered in this book. But
the sample of the conversation for speaking task is too short and too simple. It is
not challenging. This book doesn’t cover listening and reading skill pretty well.
Researcher: Do you think this book has covered all the student needs?
Respondent 1: Absolutely no. You know, this book is too easy and students need
more texts, need more vocabulary. Look, even there is no grammar in it.
Researcher: Sorry to interrupt, so you think that grammar is important to be
taught to the students?
Respondent 1: Well, not directly. But at least students can refer to the book for
grammar if they find any difficulties when they study at home by themselves.
Researcher: Sorry, so are you saying that this book does not promote auto
learning or independent learning?
Respondent 1: Hmm… yes, sort of. Students cannot learn by themselves. They
need teachers or instructors when they study using this book.
Researcher: So what do you do responding this situation?
Respondent: I do anything possible to do for my students such as finding more
relevant materials. I mostly find more texts to support my teaching. And to make
more challenging I change a little bit the tasks in textbook and sometimes I
115
replace it with other different tasks because, you know, the exercises in the book
are too easy for my students.
Interview 2
Researcher: What is your comment towards English textbook for grade 7
entitled When English Rings the Bell?
Respondent 2: [release breath] This book is good. It is colorful. Many pictures
support the activity. But sometimes I feel it is too many. Hmm … [opening
textbook then look at it briefly] oh ya… there is no listening section I guess and
the text is very limited though I add with my own texts. I use previous textbook
and other textbook to complete When English Rings the Bell.
Researcher: So what is the strength of this textbook?
Respondent 2: colorful, many pictures and … look at this picture [pointing to a
page of textbook] this represents Indonesia. So Indonesia, in my opinion. This is
good. The topic is varied.
Researcher: And what do you of the lack of this textbook?
Respondent 2: I am a little bit confused to answer this. Well, this book doesn’t
have listening section. It is not good because teachers then must find additional
material for listening. It is bother us as teacher [laughing]. And what else mas?
[laughing]
Researcher: What about the approach, Sir?
Respondent 2: Ah… that one. It is good. Sometimes the Scientific Approach (SA)
can be applied in my class but sometimes cannot. And I think I never heard of
scientific approach in English teaching, is it? It’s been years since I graduated
from my university. I forgot many things about theory of teaching English
language. [laughing]
Researcher: So do you think the SA is suitable for your class?
Respondent 2: [sigh] sometimes [smiling]. I think there isn’t such approach in
English teaching. But since the government told us to apply it in our class, so I
must apply it.
Researcher: So do you apply this approach completely?
Respondent 2: Sometimes. So I modify the approach sometimes.
Researcher: So do you use textbook every time you teach?
Respondent 2: Not all the time. But most of the time, I use textbook as the guide.
So when I feel I need to add materials, then I will find it because this textbook is
too easy for many of my students.
Researcher: Do you have anything else to say about this textbook?
Respondent 2: Yes, this book still needs to be revised. [smile]
116

Full

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Full

Caricato da

Copyright:

Formati disponibili

PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI

THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

THE GRADUATE PROGRAM OF ENGLISH LANGUAGE STUDIES

SANATA DHARMA UNIVERSITY

THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

I owe an enormous debt of gratitude to F.X. Mukarto, Ph.D., my supervisor,

STATEMENT OF THE ORIGINALITY .................................................................... iv

D. Data Gathering Techniques ................................................................................ 32

Figure 2.1 The Theoretical Framework ..................................................................... 29

Table 2.1 Criteria of an Effective Textbook .............................................................. 15

C13 Curriculum 2013

Appendix 1: Questionnaire Draft 1 79

As the process of curriculum development, Indonesia developed a new national

Sebagai bentuk proses perkembangan kurikulum, Indonesia mengembangkan

Kajian ini digolongkan dalam sebuah penelitian kuantitatif diskriptif.

Pada kesimpulannya, peneliti melihat bahwa buku teks terebut memunyai

Kata kunci: Evaluasi materi pembelajaran, evaluasi bahan ajar, evaluasi.

research, and the research benefits.

is a partial procedure of a macro evaluation of curriculum development. Third,

Sheldon (1988) said that a textbook evaluation needs to be administered since a

an evaluation process of a textbook is fundamental because it affects effectiveness of

and feedback resulted from the process for its revision.

Another reason why this research is about material evaluation is because

textbook is an important resource for (1) presenting language components, (2)

Agreeing to Cunningsworth (1995), Tomlinson (2008) stated that textbook evaluation

evaluation can be carried out to help a publisher or an institution to make decisions,

teachers, stakeholders and the program.

should do a material evaluation besides determining whether the material “works” or

not. Empirical evaluation of teaching materials can be found in accounts of trialing of

considered textbook evaluation to be one of the most important components of a

teacher training course (Hutchinson & Waters, 1991).

current negative phenomena especially among young people and discouraging

perceptions among Indonesians regarding education (Hamied, 2014). However,

C13 is in the urge to be conducted.

and broader is to facilitate the whole process of curriculum development (Ellis,

B. Problem Identification and Limitation

toward its administrative matters to pedagogical matters. Administrative matters

include dissemination of the curriculum, budget, distribution of textbooks and teacher

evaluation employ in it. These evaluations later will be called as macro-evaluation.

task evaluation can be done in micro-evaluation.

Too many aspects must be evaluated in macro-evaluation. As a matter of fact,

more specific subject. A micro-evaluation will be undergone to examine the English

on textbook published by the Indonesia Ministry of Education entitled When English

will be elaborated in detailed in chapter 2.

using 2013 curriculum WERB research questions are raised as follows:

1. To what extent does textbook meet the criteria?

E. Benefits of the Study

teachers, learners, and stakeholders.

For materials developers, this research evaluation is beneficial as reference in

do in relation to the findings.

in building up theoretical framework of the research. Based on the research problems,

it is recognized into three major constructs, namely English textbook, Evaluation in

English Language Teaching, and Indonesian curriculum of 2013.

as supplementary materials in the English course program. Therefore, it is important

to clarify the definition on what it is called as a textbook.

It seems that Ur’s (1991) definition about textbook is the easiest to

comes to the necessity of using a textbook. It is explained that textbook is book of

process it is followed systematically as the basis of a language course.

Textbooks now become a must-have element in EFL/ESL teaching. Those are

common element in the teaching of English as the second language or foreign

time by providing ready-made teaching texts and learning tasks.

Therefore, there is a sense of structure and progress. A textbook can be also a

syllabus of an EFL/ESL course since a textbook should provide a syllabus in it.