LEARNING OUTCOMES IN BOLOGNA TOOLS:

The strengths, weaknesses and challenges represented by learning outcomes with particular reference to curricula development

Stephen Adam
stephenadam@orange.fr
 IMPACTS ON:

Education
systems
Educational
structures
Educational
processes
Delivery
mechanisms
+
Students /
citizens
Employers
Academics
Administrators
+
International
transparency,
recognition,
mobility,
competitiveness,
efficiency and
attractiveness of
European higher
education

A huge reform agenda to modernise European higher education

- still in panic about globalisation and competitiveness
A multitude of tools, initiatives and CONTEXT:
• Growth in demand
realities – all linked! • Constrained funding
• Demographic change
• Increased competition
Qualification  • Globalisation
New technology Frameworks
(QF‐EHEA + EQFLLL  Bologna and 
+ NQF) Copenhagen 
processes – not 
complete!
Mission 
statement/diversit
Mobility Targets
y in university 
Rethink role roles (By 2020 20% 
Autonomy
study/training 
Modernisation Revenue sources –
agenda + CURRENT EUROPEAN HIGHER abroad)
Restructure Finance and 
mergers! EDUCATION REFORM ISSUES
S U R V I V A L?

International  Curriculum 
Re-think reform/developme
quality assurance ‐ curricula and
Standards and  nt maintaining 
qualifications diversity
Guidelines (ESG) + 
or…
EQAR Student‐ centred
learning

Recognition issues
(Diploma  Markets 
Supplement +  Borderless  Employability
Lisbon Recognition  Credit Systems +  education (TNE)
Convention + RPL) confusions 
Internationalisation
ECTS v ECVET Learning 
outcomes –
delivery 
assessment
 We will strive for more coherence between our policies, especially in completing the
transition to the three cycle system, the use of ECTS credits, the issuing of Diploma
Supplements, the enhancement of quality assurance and the implementation of
qualifications frameworks, including the definition and evaluation of learning outcomes.

 To consolidate the EHEA, meaningful implementation of learning outcomes is needed. The

development, understanding and practical use of learning outcomes is crucial to the
success of ECTS, the Diploma Supplement, recognition, qualifications frameworks and
quality assurance – all of which are interdependent. We call on institutions to further link
study credits with both learning outcomes and student workload, and to include the
attainment of learning outcomes in assessment procedures. We will work to ensure that the
ECTS Users’ Guide5 fully reflects the state of on-going work on learning outcomes and
recognition of prior learning.

 We welcome the clear reference to ECTS, to the European Qualifications Framework and to
learning outcomes in the European Commission’s proposal for a revision of the EU Directive
on the recognition of professional qualifications. We underline the importance of taking
appropriate account of these elements in recognition decisions

 National priority by 2015 - Ensure that qualifications frameworks, ECTS and Diploma
Supplement implementation is based on learning outcomes;

 European priority by 2015 - Work to ensure that the ECTS Users’ Guide fully reflects the
state of on-going work on learning outcomes and recognition of prior learning;

 European priority by 2015 - Coordinate the work of ensuring that qualifications frameworks
work in practice, emphasising their link to learning outcomes and explore how the QF-EHEA
could take account of short cycle qualifications in national contexts;
Focus on:

1. Definition
2. Learning outcomes ‐ a difficult paradigm change
3. Typology of learning outcomes ‐ multiple applications
4. Bologna ‐ learning outcomes and curricula development
5. Problems + benefits of using learning outcomes
6. Good and bad practice creating learning outcomes
7. Examples of learning outcomes
8. How to write good learning outcomes
9. Curricula development – checklist of questions
 A statement of what a learner is expected to know,
understand and/or be able to demonstrate at the end of a
period of learning.

 Learning outcomes (are) statements of what a learner is

expected to know, understand and/or be able to demonstrate
after a completion of a process of learning.

 Statements of what a learner can be expected to know,

understand and/or do as a result of a learning experience.

 Student learning outcomes are properly defined in terms of

knowledge, skills, and abilities that a student has attained at
the end (or as a result) of his or her engagement in a
particular set of higher education experiences.

 Learning outcomes are statements that specify what a

learner will know or be able to do as a result of a learning
activity. Outcomes are usually expressed as knowledge,
skills, or attitudes.
2. Learning outcomes - a difficult paradigm change
‘what a successful student knows, understands and can do’

Represents a move from teaching to learning ‐ from 
an input to an output focus (permeates the Bologna + 
EQF reforms) – ubiquitous by reference but poorly 
understood and implemented across Europe

Often strongly resented by ‘conservative’ academics 
(re‐thinking qualifications)

Raises fundamental questions about the learning‐
delivery‐assessment relationship + purposes of 
qualifications (curriculum development)

Represents a move towards student‐centred learning 
= a difficult paradigm shift 

Difficult to create and write well …
 National generic
qualification
descriptor
Unique National subject
qualification (BA, MA, PhD} benchmark/sector
descriptor descriptor/National
(BA History =- with Occupational
6-8 overall LO?) Standard

National generic
Assessment and level descriptor
grading criteria

MULTIPLE
Module/unit International
precise descriptors LEARNING generic cycle/level
descriptor (Dublin
(6—8 LO?) OUTCOMES + EQF)
APPLICATIONS

Learning outcomes permeate directly or

indirectly all the Bologna reforms
including QA, recognition (Diploma
Supplement), mobility, credits (ECTS),
qualifications frameworks, etc.
 EQF and Bologna
National meta-descriptors
Qualifications International
Frameworks (NQF) peer group
descriptors/standards comparisons
+ input

New national NEW IMPROVED Subject

and international QUALIFICATIONS Benchmark
approaches to quality AND CURRICULA Statements,
assurance BASED ON LEARNING sectoral statements,
(internal + external) OUTCOMES NOS

Inclusion of generic
Use of learning
transferable/transversal
outcomes + new modes
skills + employability
of delivery & assessment
agenda + more
+ student-centred
Stakeholder input New delivery learning
and assessment
methods
+ new technology
NEGATIVE
[Might inhibit/constrict the learning process –
yes ‐ if written badly]
 May suit training better than open-
ended higher education
 Can create a target-led culture +
tick box mentality
 Attacks the liberal conception of
the university
 Technically difficult and expensive
to introduce + resented by staff
 Cannot treat students like Pavlov’s
dogs …
POSITIVE
(for course design and learners)
 Helps ensure consistent delivery
across programmes of learning
 Provides more precision to
curriculum/course design + stops
overlap
 Informs student choice
 Highlights teaching, learning and
assessment relationship
 Benefits quality assurance
 Potential to link vocational
education & training (VET) &
higher education (HE) and
facilitate the creation of lifelong
learning systems
 Only effective and fair way to
recognise non-formal and informal
learning (RPL)
 Increases transparency for
stakeholders
 Improved national/international
recognition and transparency of
qualifications
GOOD
PRACTICE
 Writing good learning outcomes takes time
and reflection.
 It is pointless to write them to fit existing,
unmodified modules.
 The benefits in the creation of learning
outcomes result from the dynamic and
cathartic process of creation. This will
involve a simultaneous reflection on
possible learning outcomes, their mode of
delivery and their assessment.
 The creation of learning outcomes is not a
precise science and they require
considerable thought to write – it is easy to
get them wrong and create a learning
straitjacket.
 Learning outcomes are commonly divided
into different categories of outcomes. The
most common sub-divisions are between
subject specific outcomes and generic
(sometimes called transferable or
transversal skills).
 The best learning outcomes are the product
of sincere reflection about realistic and
attainable combinations of any of the
following: knowledge and understanding,
practical skills (including applying
knowledge and understanding), subject
specific and transversal/transferable skills,
etc. (see BLOOM: cognitive, affective and
psychomotor domains).
BEWARE
 Do not be too prescriptive or too vague
 Avoid the use of simplistic terms such
an ‘understand’ or ‘explain’ as these are
imprecise and convey little.
 Generic qualifications descriptors,
subject specific benchmarks/sectoral
statements and national level
descriptors should always be presented
as guidance. They are not straitjackets.
 Existing qualifications should never be
repackaged with newly minted but fake
learning outcomes used to decorate old
and substantially unchanged units.
 Beware of creating an assessment-
driven curriculum where learning
outcomes are over-prescribed and
confine the learners’ ability to make
imaginative jumps and insights.
 The adoption of learning outcomes
should never be regarded as part of a
move towards the national or European
standardisation of content.
On successful completion of this module/programme the student will be able to:

MODULE IN RESTORATIVE DENTISTRY: (2/7)

 Examine a patient extra-orally and intra-orally;
 Formulate an appropriate treatment plan based on an understanding of the disease
process present and a prediction of the likely success;

MODULE IN ECONOMICS: (3/9)

 Interpret national income and expenditure accounts; Are any
 Differentiate between monetary and fiscal policy; questionable?
 Criticise budgetary decisions using economic criteria;

PROGRAMME LEARNING OUTCOME FOR A SECOND CYCLE COMPUTER SCIENCE

DEGREE: (3/8)
 Use, create and manipulate large computational systems;
 Work effectively as a team member;
 Write thesis/report to a professional published standard;

PROGRAMME LEARNING OUTCOME FOR A FIRST CYCLE ENGINEERING DEGREE:

(2/5)
 Identify, formulate and solve engineering problems;
 Design a system component or process to meet specified needs and to design and Examples from: Writing and
using learning outcomes – a
conduct experiments to analyse and interpret data
practical guide by Declan
Kennedy (UCC 2007)
ESSAY (BA Politics)
 Compare the notion of freedom in Marx’s ‘Communist Manifesto (1848)’ with that which Rousseau
presents in the ‘The Social Contract (1762)’. Which provides most insight into understanding current
issues of freedom in Syria?
8. How to write good learning outcomes:

Initial institutional steps – audit of current situation

• Undertake an audit of the existing curricula, their content, relevance, 
effectiveness, delivery and assessment. 
• Draw upon multiple independent information sources including 
students, alumni, employers, academics (internal and external), 
success/failure rates, employment record of graduates. 
• The results of such analysis should feed into any overall plans to 
reposition the institution in terms of its mission statement and the 
direction and plans for the future.
 Curriculum policy development (teaching and learning strategy)

• Develop a teaching and learning strategy
• There is no agreed format for these (see examples on web)
• They have direct links to quality assurance (internal/external)
• You can include: vision, objectives, learning outcomes, generic skills, 
employability, student‐centred learning, flexibility programmes, wider 
variety of assessment, standards for feedback, new delivery methods, wider 
participation, learning environment, increased choice and subject 
combinations, use of new technologies, etc. 
• Establish all its implications (physical + human resource)
'What' may include subject knowledge and understanding; a range of intellectual, 
subject based and transferable skills and their application in a range of contexts ‐
perhaps also competency to practise; values and other qualities. Programme 
outcome statements can be created by completing sentences like:

 This programme is distinctive because it develops...
 The most important values which inform this programme are...
 The academic content of this programme concentrates on...
 The most important intellectual skills developed in the programme are...
 The most useful practical skills, techniques & capabilities developed are...
 The most important ways in which a student will learn are...
 On completing the programme we want students to know & understand...
 On completing the programme we want students to be able to....
Think about the module/qualification developmental
process:

SOURCE:
Dr Louise
Naylor,
University
of Kent)
1. Thinking critically and making judgements (Developing
arguments, reflecting, evaluating, assessing, judging)

2. Solving problems and developing plans (Identifying

problems, posing problems, defining problems, analysing
data, reviewing, designing experiments, planning,
applying information)

3. Performing procedures and demonstrating techniques

(Computation, taking readings, using equipment, following
laboratory procedures, following protocols, carrying out
instructions)

4. Managing and developing oneself (Working co-operatively,

working independently, learning independently, being self-
directed, managing time, managing tasks, organising)
5. Accessing and managing information (Researching, investigating,
interpreting, organising information, reviewing and paraphrasing
information, collecting data, searching and managing information
sources, observing and interpreting)

6. Demonstrating knowledge and understanding (Recalling, describing,

reporting, recounting, recognising, identifying, relating and
interrelating]

7. Designing, creating, performing (Imagining, visualising, designing,

producing, creating, innovating, performing)

8. Communicating (One and two-way communication, communication

within a group, verbal, written and non-verbal communication.
Arguing, describing, advocating, interviewing, negotiating,
presenting, using specific written forms)
A. Self Management This refers to a student's general ability to manage her own
learning development. Abilities required to do this successfully include:
 an ability to clarify personal values
 an ability to set personal objectives
 an ability to manage time and tasks
 an ability to evaluate one's own performance

B. Learning Skills This refers to a student's general ability to learn effectively and be
aware of her own learning strategies. Abilities required to do this successfully include:
 an ability to learn both independently and co-operatively
 an ability to use library skills, to find and organise information
 an ability to use a wide range of academic skills (research, analysis, synthesis etc.)
 an ability to identify and evaluate personal learning strategies

C. Communication This refers to a student's general ability to express ideas and

opinions, with confidence and clarity, to a variety of audiences for a variety of purposes.
Abilities required to do this successfully include:
 an ability to to use appropriate language and form when writing and speaking
 an ability to present ideas to different audiences using appropriate media
 an ability to listen actively
 an ability to persuade rationally
D. Teamwork This refers to a student's general ability to work productively
in different kinds of team (formal, informal, project-based, committee
based, etc.) Abilities requires to do this successfully include:
 an ability to take responsibility and carry out agreed tasksan ability to
take initiative and lead others
 an ability to operate in a range of supportive roles within teams
 an ability to negotiate, asserting one's own values and respecting others
 an ability to evaluate team performance

E. Problem solving This refers to a student's general ability to identify the

main features of a given problem and to develop strategies for its
resolution. Abilities required to do this successfully include:
 an ability to analyse
 an ability to think laterally about a problem
 an ability to identify strategic options
 an ability to evaluate the success of different strategies

F. Information Technology This refers to a student's general ability to

use IT appropriately for their learning and employability. Abilities required
to do this successfully include:
 an ability to use IT as a communication and learning tool
 an ability to use IT to access and manage information
 an ability to use IT to present ideas
 an ability to use specialist software where relevant to the discipline
Many common problems associated with learning outcomes:
[Linkages between learning outcomes, curriculum development, qualifications frameworks
and quality assurance are not fully understood]

1. Learning outcomes often created with no link to external reference points (standards)
2. Generic national level descriptors are too vague to provide useful guidance
3. Assessment is often traditional and inappropriate to learning outcomes
4. Assessment criteria are missing or lack detailed learning outcomes
5. Learning outcomes employ inappropriate active verbs; they are vague and meaningless
6. Module/unit learning outcomes fail to support overall qualification LO
7. Lack of academic autonomy /staff autonomy stifles creative learning outcomes
8. The context of delivery prevents creation of good learning outcomes (+ finance)
9. ECTS credit allocation fails to be linked to learning outcomes
10. Staff development /support re learning outcomes development is missing
11. HEI lack teaching, learning delivery policies to support curriculum development
12. Insufficient stakeholder involvement in the creation of learning outcomes
13. Transferable skills missing from learning outcomes – including ‘employability’ aspects
14. Internal and external quality assurance not focusing on learning outcomes
15. Learning outcomes fail to embrace student-centered learning + support RPL
16. New ‘cosmetic' learning outcomes placed on unreformed curricula academic (inertia)
9. Curricula development checklist of questions:

1. How is your HEI strategically placed (e.g. undertake SWOT/PEST/Decision Tree analysis)?
2. What is your current institutional profile and how do those outside perceive it?
3. How would your like your institution to be perceived + what are its natural + emerging
markets
4. Given globalisation, competition and market realities, do you need to change your mission
statement – do you need to change institutional priorities and directions?
5. Do you need to conduct an internal curriculum audit of the current situation + on what
basis?
6. What are your current curriculum strengths and weaknesses (teaching/research/subjects,
etc.)
7. What sort of qualifications do your students require and how do you know this?
8. Is there a need to introduce more intermediate qualifications (i.e. within the first Bologna
cycle) and new qualification types (i.e. Professional/Industrial Doctorates, Master
programmes)?
9. Are your current qualifications fit for purpose - what is the evidence for this?
10. Are your questionnaire/feedback and stakeholder (alumni, student/employer) involvement
mechanisms effective in obtaining practical information to help curriculum development?
11. What areas of the curriculum do you need to develop/re-develop/drop?
12. Can you develop/exploit links with other HEI (national and international) to aid the internal
process of curriculum change?
13. What are the implications of curriculum change on: management;
Institutional organisation and structures; learning resources; staffing and
staff development; building/room configuration; management Information
systems; finance, quality assurance, etc.
14. Do new curriculum development approaches conflict with any existing
institutional (and national) regulations (e.g. subject combinations, joint-
degrees, admissions, recognition of prior learning)?
15.What are the limits (resource constraints) to the introduction of more varied
assessment and course delivery?
16.Do you have a policy towards curriculum development - are staff aware of it
and is it effective?
17.Is there an institutional teaching and learning strategy document? When
was it last reviewed and does it have any impact?
18.Which stakeholders were consulted, and by what mean, when any such
policy/strategy was created?
19.Does your teaching and learning policy reflect the institutional mission
statement – do you need to reconsider the institutional orientation?
20.Is curriculum development given sufficient priority within the institution – is
it a standing item on the highest university committees? Is it considered a
dynamic element within the HEI?
21.What do you need to include in your teaching and learning strategy
document – main focus, deadlines, measurable outcomes, rolling plan,
feedback/review mechanisms, etc?
22. What sort and level of support do you need to give to those charged with policy
implementation?
23. What key staff do you need to appoint and how should they be organised – a
special unit (with rotating seconded personnel?) and with what level of rectorate
backing and resources?
24. If established, what might be the most effective way forward for any staff
development unit - offer walk-in sessions, public events, newsletters, etc?
25. How will you train the trainers (capacity building) - do you require a staff
development manual, in-house training modules?
26. What impact do you expect your teaching and learning policy to have on new
staff appointments and promotions?
27. How do you overcome staff resistance to curriculum change (carrot v stick or
something else)?
28. How can curriculum development link to staff research expertise + should it?
29. Do staff fully understand ‘learning outcomes’ and their major impact on
curriculum building?
30. How does your teaching and learning policy link to internal and external quality
assurance?
31. How can you use the internal validation/review process as a way to change the
curriculum?
32. How does curriculum development link to the introduction of more flexible study
programmes and the use of ECTS credits?
 ARISTOTLE
TWO FINAL
THOUGHTS:

‘What we have to learn

to do, we learn by
doing.’
[active learning, student-centred
learning, outcomes-based learning]

CHARLES DARWIN
'It is not the
biggest, the brightest
or the best that will
survive, but those who
adapt the quickest.’

• Learning outcomes are the basic building blocks of the Bologna HE reforms (permeate all)
• Provide a framework for quality & standards by relating to external reference points.
• Increase national and international transparency and recognition + improve qualifications
• Also help us reconsider the ‘who, how, when, where and why’ of education in 21st Century
References + further information:
A large number of guides and handbook exist that give step by step
advice on creating effective and appropriate learning outcomes
including:

Adam S (2004) Using learning outcomes, Scottish government. Download

available: http://www.scotland.gov.uk/Publications/2004/09/19908/42704
Adam S (2008) An introduction to learning outcomes, EUA Bologna handbook,
RAABE publishing. Download available:
http://is.muni.cz/do/1499/metodika/rozvoj/kvalita/Adam_IH_LP.pdf
Baume D (2011) Writing and using good learning outcomes, Leeds Met university.
Download available: http://www.leedsmet.ac.uk/Learning_Outcomes.pdf
Bologna Process (2008) The shift to learning outcomes, CEDEFOP.
Download available: http://www.cedefop.europa.eu/EN/publications/12952.aspx
CEDEFOP (2008) The shift to learning outcomes,, CEDEFOP. Download available:
http://www.cedefop.europa.eu/EN/publications/12952.aspx
CEDEFOP (2011) Using learning outcomes ,EQF Note 4, CEDEFOP download
available:
http://www.cedefop.europa.eu/en/news/19115.aspx
Kennedy D (2008) Writing and using learning outcomes, EUA Bologna Handbook,
RAABE publishing. Download available: http://www.procesbolonski.uw.edu.pl/dane/learning-
outcomes.pdf
Moon J+ Gosling (2007) How to use learning outcomes and assessment
criteria, SEEC publications. Download available: http://www.seec.org.uk/publications/how-
use-learning-outcomes-and-assessment-criteria