Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Article views: 30
This paper investigates attitudes among educational organisations towards the use of
system evaluation for quality improvement in Swedish higher education. The subject of this
investigation is the evaluation of the pilot project on advanced vocational education (AVE), which
was conducted between 1997 and 1999 by a research team at the Lulea Technological University.
The investigation was carried out in the form of a telephone inquiry with a strategic selection of the
education organisers. The investigation shows that 65% of the respondents regard the work of the
research team, built on the system-evaluation model, as an instrument to inspire con dence and
contribute to renewal.
ABSTRACT
Introduction
Evaluations have become an increasingly important instrument in carrying out reforms of
higher education in Sweden. There are several reasons for this development. One is a
desire to acquire knowledge of the objectives set up for certain educational activities and
to check whether these have been attained (Franke-Wikberg & Lundgren, 1979). A second
motive is the desire to know what primary and secondary effects have resulted from the
educational activities in question. A third motive is that representatives of the political
establishment want more information on the economic resources used to implement the
activities (Stevrin, 1991; Franke-Wikberg, 1992).
In recent years, a parallel development has become discernible in view of educational
organisations increasing interest in the evaluation models developed in quality management. There are two main motives behind this development. First, evaluation for improving quality is perceived by the educational organisers to be particularly important in
maintaining a competitive edge over the growing market of private education and training
companies in Sweden (Lindell, 2001). Second, the models illustrate the importance of
taking advantage of various stakeholders needs and expectations in the best way possible.
The needs to be considered are often complex and varied since they are derived from
stakeholders with different motives, such as, educational and policy planners at the state
level, employer and employee bodies, local companies and students. Evaluation and
quality assurance, in the sense that the expectations are met, are very important for
providing education that is sought after and ts the demands of working life. This interest
can be exempli ed by the establishing of quality awards such as the European Quality
ISSN 1353 8322 print; 1470 1081 online/02/020127 10
DOI: 10.1080/135383202200000432 2
128
Award and Malcolm Baldrige National Quality Award. In both Sweden and abroad
(Muller & Funnell, 1992), this interest has also been carried over to organisations in the
educational eld. In Sweden, for instance, a special award for schools, the Swedish Schools
Award, was established in the mid 1990s.
With the increased importance of evaluation in higher education, research and debate
has focused on the choice and design of models of evaluation. The most widely used
models in Sweden for evaluating educational reforms are often referred to as `engineersmodels. In these, evaluators attempt to be independent and impartial, avoiding, as far as
possible, contacts and communication with respondents. Nor is the evaluator allowed to
make suggestions for improvements or support during the process of the evaluation:
comments are communicated separately (Vedung, 1995).
However, this attitude is changing in Sweden with the development of various alternatives to the `engineers-models. According to several studies, this has taken place because
of increased political awareness of the demands of pressure groups during the evaluation
(Karlsson, 1995). System evaluation is one of a number of models that build on this new
evaluation concept.
One major problem is the lack of research into how interactive system-evaluation models
are perceived by those evaluated. Only a few of the national evaluation projects for higher
education reforms conducted in Sweden during the 1990s have used systems evaluation
(Kim, 1997). This investigation starts from a national evaluation project on Advanced
Vocational Education (AVE) that uses the concept of systems evaluation. It studies how the
evaluation process was perceived by those evaluated. The evaluation was performed
between 1997 and 1999 by a research team from Lulea Technological University (SOU,
1999).
This investigation aims at answering two questions. The rst concerns the extent to
which the work of the research team on the AVE-project is perceived as a support for
development of the educational organisations. The second aims at answering whether the
work of the research team on the AVE evaluation is viewed with con dence or mistrust
among the educational organisations.
129
system. Other criteria are adaptability, the ability to identify changes in requirements, and
mechanisms for feedback of information (Stevrin, 1991).
130
performed. At local level, the research team performs visitations and face-to-face interviews and telephone-interviews.
National Level: the overall design and objectives of the trial activities
On the national level, the research team based their work on reports and surveys of the
Swedish labour market with a view to catering for future requirements. These included the
need for a higher vocational educational level and a more versatile workforce. One
problem in this form of post-secondary education is the uncertainty about how and what
students should learn in a rapidly changing labour market, which is characterised by
international competition.
In view of this uncertainty, the research team found it important not to use an objective
model but instead use a model that would support the improvement work during the trial
period rather than checking the extent to which the goals have been met after conclusion
of the project. Furthermore, the research team found it valuable to use a model that permits
the subject of evaluation to change its established goals in the course of the trial activities
but is still able to judge whether or not the change implies an improvement. The principal
means of supporting improvement work among the AVE organisations, and at the same
time taking care of data collection for the evaluation work, was self-assessment in the
respective educational organisations. All AVE educational organisations were, therefore,
offered a tool for self-assessment to support improvement work. Self-assessment also
constituted a means of collecting data required for the evaluation. A second information
channel was the collection of national statistics for the educational system, which were
then compared with AVE.
Self-assessment in AVE Organisations
A special tool for the self-assessment, based upon TQM and called `Laroverket [1], was
constructed at the Division of Quality Technology & Statistics at Lulea Technological
University (Svensson & Klefsjo, 2000). Here TQM is de ned as `a management system
consisting of core values, methodologies and tools, which support each other in order to
increase customer satisfaction with less amount of resources (Hellsten & Klefsjo, 2000).
Self-assessment, developed and supported by various quality awards during the last
decade, has become established as a tool for quality improvements. In this project,
self-assessment has been introduced as a process consisting of the following four phases:
planning, description of the organisation, analysis of the description, and creation of a plan
for improvement (Svensson & Klefsjo, 2000, p. 238). These four phases are closely related
to the four phases of the improvement cycle `plan do study act described by Deming
(1994).
`Laroverket was developed as a basis for the self-assessment work and is built on the
core values by Bergman and Klefsjo (1994). The structure of `Laroverket is inspired both
by the criteria for the Swedish Quality Award and a tool named `Springboard (Hellsten,
1997a,b). `Laroverket has ve evaluation dimensions and from these a number of typequestions are formulated intended to stimulate the systematic work with quality improvement. The dimensions and the questions are:
1.
2.
3.
4.
131
132
According to Reitberger and Uhlin (1996), older cognitive models of evaluating trial
activities are no longer valid when the trend of development is towards new types of
educational organisations. New technology, new sets of values and new geopolitical
conditions have created not only new educational forms but also new leadership ideals
and new forms of communication. Thus, it is not only the physical, organisational and
social infrastructures that have changed but also the mental ones. Examples that support
Reitberger and Uhlins hypotheses in the AVE case are the ability of the evaluator to
evaluate the part of the education known as on-the-job learning, which accounts for
one-third of the total course time.
On the basis of the above discussion, the research team visited 25 education providers.
During these visits, the school management, a number of teachers and students were
interviewed, together with several administrators and representatives of working life, all
of whom must, according to the ordinance, be associated with all levels of education.
Telephone-interviews were conducted with several representatives of organisations that
have arranged places for the advanced application of theoretical knowledge at a workplace, and of educational co-ordinators.
In summary, the evaluation study of AVE is an example of the concept of system
evaluation in two respects. First, it emphasises multiple channels for input of data, on both
national and local level. Second, it offers educational organisations active participation in
the evaluation process.
Attitudes towards System Evaluation: model for analysis
The Lundin and Soderholm model (1995) has been used to investigate the attitudes
towards the use of system evaluation in the AVE-project. According to this, the encounter
between an evaluator and the activity to be evaluated depends in part on the expectations
the two parties have and in part on how the party being evaluated experiences the
evaluation.
Expectations may either be positive, in the sense that the work of evaluation will lead to
development potential being demonstrated, or negative, in the sense that an evaluation is
seen as a threat to existing activities. In the former case the evaluation will be viewed with
con dence and in the latter with mistrust. Evaluations may also be experienced as
`backward-pointing, where evaluators appear only to look for shortcomings, or as `forward-pointing, where there is a sense of renewal.
By a combination of the con dence mistrust and `forward/backward-pointing dimensions it is possible to construct a four- eld matrix that describes the four different types of
evaluators (Figure 2).
An evaluator who is regarded primarily as an instrument of authority can be denoted
type I. In this role evaluators are viewed with mistrust as they intervene in the life of those
being evaluated with the approval of those in power. This is done, with little interest in the
activity, to record discrepancies that it will later be possible to correct. In addition, the
corrections made will be mainly of a negative nature. Type I evaluators often encounter
various defence mechanisms. Type I lies close to what is perhaps the traditional evaluators
role, that is to say, evaluation as a form of supervision for someone elses purposes and
with a clear power relation in the background.
Type II evaluators represent the face of benign power. They are met with con dence as
they are seen as positive, with a well-developed feeling for how the results of the
evaluation will be handled. Those being evaluated have, in this case, much less fear of
sanctions and punishment than the type I role gave rise to but this is nonetheless a
133
supervising function. To summarise, type I and II, respectively, can be seen as comparable
to the earlier mentioned roles within the `engineers-model.
The characteristic feature of type III evaluators is that their methods do not aim
primarily at looking for and reporting faults and inconsistencies but rather they are
concerned with the future welfare of the subject. Type III is thus the true activity developer
and assistant, with whose aid those being evaluated can hope to develop their job in a
forward-pointing direction. Another important characteristic of type III is the lack of clear,
strong connections with the assigner of evaluators and thus, at least in theory, there is no
possibility of sanctions. A metaphor often used for the type III evaluator role is an `angel.
Finally, type IV evaluators are the representative of an `evil power, who tries, with
various temptations, to induce those being evaluated to enter into certain contracts for the
future that are not perceived as legitimate. It is true that type IV is future-oriented but, in
contrast to type III, it represents an undesirable and threatening future against which it is
important to protect oneself. In evaluation contexts, the type IV evaluator is referred to as
a `demon.
Description of Methods, Selection and Implementation
Telephone-interviews were used to investigate the opinion of those being evaluated within
the AVE-project. The interviews consisted of six questions, divided into two separate parts.
The selection of respondents was made on the basis of strategic considerations. Of the total
population of 123 educational organisations, those organisations and within them those
individuals who, conceptually, were most familiar with the work of the evaluation team,
insofar as that they had participated in part of the `Laroverket education, administered
student inquiries and participated in the programme of visits of the evaluation team. The
selection frame encompassed a total of 22 educational organisers. Before telephone-interviews were performed, the selected educational organisers had received inquiry forms.
The interviews consisted of six questions, and were divided into two separate parts. The
rst part of the telephone-interviews consisted of four questions with xed alternative
answers arranged on a Likert scale (Hellevik, 1984). They were created on the basis of
Lundin & Soderholms analysis model and the concept pairs `con dence/mistrust and
`forward/backward-pointing. Each pair of concepts was operationalised into two contradictory assertions. Each respondents attitude was placed in the four- eld matrix (Figure
2). First, the given alternative responses were graded on the Likert scale with numerical
134
FIG. 3. Overall picture of the respondents attitude to the work of the evaluation team; each cross represents
one organisation.
values from 1 to 5. The numerical values assigned to the respondents responses were
compiled for each of the assertions. Also, the mean values for the four concepts were
calculated and entered into the box-plot, which were combined with the four- eld matrix
(Figure 3).
The second part of the inquiry consisted of two questions with open alternative answers.
The questions served to reduce any ambiguity and contradictory answers given in the rst
part. In addition, background data on the respective educational organisations that can be
assumed to bias the answers has been used, that is line of business, ownership form,
geographic location and economic nancing.
Demons, Angels or What?
Of the educational organisations consulted, all 22 respondents answered the rst part of
the telephone inquiry. However, two respondents chose not to answer the rst part of the
inquiry with xed alternative responses with Likert scales, which gave a response frequency of 91%. The investigation shows that 65% of the respondents regard the work of
the research team, built on the system-evaluation model, as an instrument to inspire
con dence and contribute to renewal.
The overall picture of the results of the study (Figure 3) shows that 10 of the respondents
(50%) denoted the work of the research team as forward-pointing. Five respondents (25%)
considered it backward-pointing. The remainder were `undecided. Sixteen (80%) had
con dence in the evaluation and three (15%) mistrusted it. All 10 who indicated forwardpointing evaluation also exhibited con dence in the process. Fifty percent of the sample
denoted the research team as type III (angels). No one identi ed the team as type IV
(demons), although two respondents were on the borderline in the matrix.
Looking more closely at the organisations that express the evaluation of the research
team with con dence and as forward-pointing in the matrix, the data do not give any
evidence that the organisations have something in common that could explain the concentration. However, geographically, there is a tendency for organisations near Lulea Technological University to have greater con dence.
135
On the other hand, looking at the organisations that express mistrust in the matrix, two
of the three are private companies. If number 16 (in Figure 3), who lies on the boundary
between con dence and mistrust, is also included in this group the result would be that
three of the four who lack con dence in the evaluation project are private companies. Also,
three of the ve who are of the opinion that the evaluation project has not been
forward-pointing have chosen not to work with `Laroverket.
Although the study indicates a positive view of the work of the evaluation team, there
are three conceivable sources of error. First, the small and selective choice of respondents:
given that the selection constitutes only 16% of the population, the results should be
approached with caution. Second, the selected group that was chosen had participated in
the evaluation teams programme of visits. Thus, they have good knowledge of the method
of working but, in addition, one cannot ignore the likelihood that personal contacts and
conversations have toned down earlier critical opinions. Third, the analysis of the interview transcriptions that was done afterwards showed that some respondents had clearly
been bent on giving their judgement of `Laroverket and its use, in spite of the study being
concerned with their views on the work of the evaluation team as a whole.
However, the study shows that a majority of the respondents perceive the work of the
evaluation team as both forward-pointing and con dence-inspiring. Most respondents
perceive the evaluators roles as of type III. Thus, there is support for the use of models of
the systems-evaluation type based on an interactive and learning attitude.
Acknowledgements
We would like to thank Research Engineer Gorgen Edenhagen, Lulea Technological
University, for his help with the telephone-interviews. We would also like to thank our
supervisors, Professor Jan Johansson at the Division of Industrial Work Environment and
Professor Bengt Klefsjo at the Division of Quality Technology & Statistics. Also, we would
like to thank the referee for all the suggestions that have improved this article.
Note
[1] The name `Laroverket is not translated into English, because of dif culties in achieving the two
interpretations in Swedish. It means a secondary grammar school, but can also mean a book, or
something from which you can learn.
References
BERGMAN , B. & KLEFSJO , B., 1994, Quality Satisfaction: from customer needs to customer satisfaction (New York,
McGraw-Hill).
DEMING, W.E., 1994, The New Economics for Industry, Government and Education (Boston, Massachusetts,
Center for Advanced Engineering Study).
FRANKE-WIKBERG, S., 1992, Utvarderingens Mangfald (Stockholm, Universitets-Och Hogskoleambetet).
FRANKE-WIKBERG, S. & LUNDGREN, U.P., 1979, Att Utvardera Utbildning. Del 1 (Stockholm, Wahlstrom &
Widstrand).
GARVIN, D.A., 1988, Managing Quality (New York, The Free Press).
HARVEY, L. & GREEN , D., 1993, `De ning quality, Assessment & Evaluation in Higher Education, 18, pp. 9 35.
HELLEVIK , O., 1984, Forskningsmetoder I Sociologi Och Statsvetenskap (Stockholm, Natur Och Kultur).
HELLSTEN , U., 1997a, `The Springboard a TQM-based tool for self-assessment , Licentiate thesis, Division
Of Quality Technology & Statistics, Lulea Technological University.
HELLSTEN , U., 1997b, `The Springboard: a strategy for continuous improvement of small and medium-sized
companies, Total Quality Management, 8(2/3), pp. 183 186.
HELLSTEN , U. & KLEFSJO, B., 2000, `TQM as a management system consisting of values, techniques and tools,
TQM Magazine, 12(4), pp. 238 244.
136
K ARLSSON , O., 1995, Att Utvardera Mot Vad? Om Kriterieproblemet Vid Intressentutvardering (Stockholm, Hls
Forlag).
K IM, L., 1997, `Utvardering Av Hogre Utbildning Pa Nationell Niva Systemutvardering Eller Kvalitetskontroll? , paper presented at the 1st Conference on Quality in Higher Education (Uppsala, National Agency
for Higher Education).
LINDELL , M., 2001, `Trends and development in vet systems: exibility, transferability, and mobility issues,
International Journal of Vocational Education and Training, 9(2), pp. 33 46.
nglar Och Demoner. Om Utvarderarroller , in ROMBACH, B.
LUNDIN , R.A. & SO DERHOLM, A., 1995, `Bland A
& SAHLIN-ANDERSSON , K. (Eds.) Fran Sanningssokande Till Styrmedel. Moderna Utvarderingar I Offentlig
Sektor (Stockholm, Nerenius & Santerus Forlag).
MULLER , D. & FUNNELL , P., 1992, `An exploration of the concept of quality in vocational education and
training, Educational & Training Technology International, 5, pp. 257 261.
PROPOSITION, 2000/01, `Kvali cerad Yrkesutbildning [`Advanced vocational education], Swedish Government Proposition Series, No. 63, Stockholm.
., 1996, `Vaxtkraft Mal 4 Ett Mangdimensionellt Och Dynamiskt Larsystem,
REITBERGER, G. & UHLIN , A
personal correspondence.
SOU, 1999, Evaluation of Advanced Vocational Education, Swedish Of cial Report Series (Stockholm, Ministry
of Education and Science).
STEVRIN, P., 1991, Utvardering for Forandring (Lund, Studentlitteratur).
SVENSSON , M. & KLEFSJO, B., 2000, `Experiences from creating AV quality culture for continuous improvements in Swedish school sector by using self-assessments , Total Quality Management, 11, pp. 800 807.
VEDUNG , E., 1995, `Utvardering Och De Sex Anvandningarna, in ROMBACH, B. & SAHLIN- ANDERSSON , K.
(Eds.) Fran Sanningssokande Till Styrmedel. Moderna Utvarderingar I Offentlig Sektor (Stockholm, Nerenius
& Santerus Forlag).