Quality Assurance in a Student-Based Agile Software Engineering

Process
Andrew Marrington, James M. Hogan and Richard Thomas
Centre for IT Innovation
Queensland University of Technology
GPO Box 2434, Brisbane, QLD, 4001.
{a.marrington,j.hogan,r.thomas}@qut.edu.au

Abstract
Agile methodologies have proven to be a popular
addition to the software engineering toolbox, promising
significant advances in quality for small teams, with
developers no longer weighed under by the
documentation said to characterise traditional
approaches. Yet, in spite of these claims, quality
assurance remains a question to which agile
methodologies have given only a partially satisfactory
answer. This work examines quality assurance practices
among student developers working within a lightweight
iterative development process. Our focus is upon the
linkages between their reported practices and the quality
levels observed in the systems delivered. The study has
identified a range of approaches to improving quality for
small, inexperienced teams, with applications to both the
educational and SME environments.

Introduction

Recent years have seen the emergence of new,

supposedly revolutionary, software engineering processes
which abandon a linear development model in favour of
iterative approaches based around short development
cycles. Such Agile, or lightweight software engineering
processes [1] exemplified by Extreme Programming
(XP) [2] - have become increasingly prominent in
industry. As modern software engineering education is
driven by an expectation that projects should reflect
industry best practice and utilise state of the art software
technologies, it is important that software engineering
students be exposed to agile processes in the classroom.
The Real World Software Process (RWSP) [3] is an
agile development process designed at QUT specifically
for small inexperienced teams: students and small
commercial software houses. The RWSP is an iterative,
adaptive, and scalable process used by all software
engineering students enrolled in the Bachelor of
Information Technology degree, and by a number of
commercial development teams in Queensland and other
states of Australia. This paper is concerned with the

development of strong quality assurance practice within

agile development teams, particularly those with limited
software engineering experience.

1.1 Quality Assurance: Background and

Traditional Practice
Regardless of the choice of process, quality assurance
is vital to the success of development projects. Software
quality is readily visible to the client in the delivered
product the so-called external quality of the system
but this view reflects aspects of software practice more
directly under the developers control. In this work we are
concerned with the internal quality of the system, the
developer-centric viewpoint focused on characteristics
amenable to management within the software engineering
process: the standard of the source code, the elegance of
the softwares design and the strategies for validation are
commonly examined. The quality of the software process
(expressed by the degree of internal quality achieved),
therefore has a direct relationship with the quality of the
software produced [4]. There is a large body of literature
and widely accepted standards against which software
processes may be accredited as part of a mature quality
assurance regime. However, traditional quality assurance
standards are often seen as too cumbersome and
expensive for use in a student environment, given the
limited scope of student projects and the inexperience of
student software engineers. Such standards are also often
regarded as antithetical (perhaps unfairly [5]) to Agile
software processes.
Agile software development methodologies thus
require educators to rethink the quality assurance
practices they teach to software engineering students.
Since the management of the software process is an
integral facet of quality assurance, it is particularly
important that students have a thorough understanding of
the process itself. It is not sufficient for students to be told
to follow the prescribed process as closely as possible
they must understand the benefits of doing so. Moreover,
if these practices are to be embedded within their
professional consciousness prior to graduation, satisfying
assessment requirements cannot be the primary driver of
compliance. The link between mature QA practices and

Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)

1530-0803/05 $20.00 2005 IEEE

system quality must be established to the extent that

students are motivated to follow the practices closely, and
to maintain their commitment even in the face of looming
deadlines.
The present study is one step in this direction, and is
intended to provide a basis for selection of practices
demonstrably successful within the inexperienced team
environment. The scope of the study and a number of the
challenges posed by lightweight processes are outlined in
the following section.

1.2 The Challenge of Agile Methods

Agile software methodologies pose several specific
challenges to conventional quality assurance practice:
Agile processes generate fewer process artefact
documents. More traditional approaches offer a
more elaborate record and audit trail than those
agile framework, thus providing established
scaffolding for formal revision and review of
practices and artefacts. The nature of the agile
doctrine, which in its extreme form asserts that
all documentation is embodied within the code
base, ensures that extraction of review material
is more difficult.
Agile processes purport to be more people
driven than process driven. This dogma
explicitly rejects the notion that software
development teams consist of replaceable
components, each performing a task prespecified in the manner of an assembly line,
instead highlighting the professional
responsibility and initiative of each developer.
While the differences are doubtless exaggerated,
there is nevertheless a perception that the
process in general and management-oriented
quality assurance activities in particular are far
less critical than the developers technical
insight.
Agile methodologies are adaptive and therefore
spread critical decisions throughout the software
process. This concern reflects the iterative nature
of the agile approach, and the adaptation of the
project to changes in requirements. Once again,
this is less distinct from traditional approaches
than some may assert, but the effects are
nevertheless more pronounced within an
iterative process.
The challenge for the researcher is thus not limited to
the evaluation of particular practices, but must
incorporate the identification of novel practices as well.

Process, proceeding from an outline to a more detailed

examination of each phase. In section 3, we discuss the
results of the student survey, before examining those
practices which appear to have had the most significant
effect (section 4). We conclude in section 5 with a
discussion of future work to address quality assurance
practices among inexperienced agile teams.

The Real World Software Process

The Real World Software Process, used by software

engineering students at QUT, combines aspects of
traditional and agile software methodologies, producing
many of the same document artefacts as the Waterfall
software process, but including agile iterations between
non-iterative set-up and finalisation phases. The agile
nature of the RWSP is a result both of growing
acceptance in the industry of agile processes, and the
belief that agile methods offer significant educational
advantages for novice software engineering students [6].
The RWSP is built around four different regimes,
known as phases:
P h a s e Z e r o the stage prior to the
commencement of the project, determining
project feasibility, and assessing risk.
Phase One initial requirements gathering,
planning of incremental releases, and building
initial functionality.
Phase N an iterative and generic phase, in
which the functionality is incrementally
extended and the specification is adapted to any
changes in the customers requirements.
Finalisation the commissioning phase of the
project, in which the software is delivered to the
customer.
More details of the process may be found in the cited
references, notably [3], or directly from the process
website: http://www.fit.qut.edu.au/~rwsp.
Within the present curriculum, software engineering
skills are developed and refined through two key projects:
an introductory unit in which students are
provided with a completed (phase 1) application
of moderate size, and are expected to complete
two incremental development cycles extending
this system; and
an advanced unit in which the now experienced
student teams develop from scratch a more
substantial application for an industry client.
Once again, two development cycles are
employed.

This paper is organised as follows. In section 2, we

give a brief description of the Real World Software
Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)
1530-0803/05 $20.00 2005 IEEE

The experiences of teams in these projects formed

the basis for the present study, and the structure of

the survey undertaken is described in the following

section.

The Survey

3.1 Survey Design

The issues discussed in 1.2 were examined through an
on-line survey of student RWSP practitioners, forming a
significant component of the honours thesis of the first
author. The format of the responses for the main body of
questions varied according to the nature of the question,
and was drawn from among these alternatives:
Likert scales when the verbal response
categories were well defined, and readily
covered the full range of likely responses.
Numerical responses when the respondent was
asked to provide a response between two
extremes (eg very satisfactory to very
unsatisfactory).
Fill in the blank-style responses when
respondents were asked to volunteer information
about additional practices they may have
implemented themselves, which had not already
been covered by previous questioning.
The data from this survey were used to identify quality
assurance practices which students had successfully
implemented within their project work under the RWSP.
The relative importance (measured in perceived impact
on software quality) of these practices was identified
through subsequent analysis of the responses, and
correlation with system quality1.

In consequence, there remain some significant

limitations on the data which form the basis of the study,
and our work is heavily reliant upon self-reports. Yet
while some concerns remain, strong support for the
reliability of the responses is provided by an at times
breathtaking degree of frankness, and there is little doubt
that the results are of value2.
Of greater concern than these judgments is the
response rate itself, and it is our intention to survey
subsequent cohorts using the same basic instruments. For
the present, however, we have in many cases enhanced
the reliability of our conclusions by aggregating sparsely
populated response bands. For example, we shall refer
throughout this analysis to quality bands encompassing
below average, average and above average ratings
of quality. These bands are aggregations from the five
level rating system of the original survey: poor,
substandard, satisfactory, good or excellent.
The majority of respondents considered that they
produced software of satisfactory quality or better (see
figure 1). This provides modest support for the contention
that the RWSP produces good quality software, and
correlates well with overall team performance across the
units considered. The responses to this question also
illustrate that students have been fairly honest in their
assessment of their work one third of respondents rated
the quality of their product as below average.
Student Opinion of their Project's Quality

Like many student-based investigations of recent

times, the present study suffered to a considerable degree
from survey fatigue, and only a modest response rate was
achieved, in spite of an initial blanket email and a
subsequent reminder. Happily, however, these responses
reflected a broad range of student experiences and
standards, and were not - as initially feared - concentrated
in the upper grade bands. Thirty-five (35) validated
responses to the survey were received from students
enrolled in our programme in 2003. Validation in this
respect refers to a check of a supplied student number
against those available from the enrolment records. In line
with the promise of confidentiality, this information was
safely discarded prior to examination of the data, and no
matching with student academic records was permitted.

Note that the requirement of respondent anonymity made

correlation with system quality more tenuous than is desirable. This
issue is examined later in the paper.

29%

31%

3.2 Survey Responses

Below Average quality (10

respondents)
Average quality (14
respondents)
Above Average quality (11
respondents)

40%

Figure 1. Respondents were asked to rank the overall

quality of their product. The distribution of responses
is roughly in line with the distribution of grades.
3.2.1
Key Practices
Most respondents believed that their requirements
specification document very accurately reflected the
users requirements (figure 2). By itself, this is
unsurprising. Students were also asked to describe the
user of their software, and on the whole many
respondents demonstrated a poor understanding of their
user. It is thus troubling that on the whole, respondents
2

The

survey

questions

can

be

viewed

http://sky.fit.qut.edu.au/~marringt/survey.shtml.

Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)

1530-0803/05 $20.00 2005 IEEE

online

at:

Opinion of Requirements Specification Quality by

Overall Quality Band

Above Average
1 out of 5
2 out of 5
3 out of 5
4 out of 5

Average

5 out of 5
Below Average

0%

20%

40%

60%

80%

100%

Figure 2. Respondent opinion of requirements

specification quality organised by overall quality
band. Here, requirements specification quality is
defined as the degree to which the user's requirements
are reflected by the document.
Respondents were presented with a number of
characteristics of software system design and asked to
select those which applied to their project. Figure 3 shows
the percentage of respondents in each quality band who
described their system design as simple, complex, easy to
understand, extensible, modular, and accounting for
future functionality. More respondents in the above
average quality band or higher kept their designs simple
(but already accounting for future functionality), easy to
understand, and extensible. Note that there is a difference
between accounting for future functionality and
extensibility designs which account for future
functionality have already begun to account for
functionality planned for future iterations, whereas
extensible designs are those which are merely well-suited
for future growth as the softwares functionality is
expanded.

Software Design Characteristics by Quality Band

Modular
Above Average

Extensible
Design Accounts for Future
Functionality
Easy to Understand

Average

Complex
Below Average

Simple
0

10

20

30

40

50

60

70

80

90 100

felt that their requirements specification documents

accurately captured the requirements of a user whom
many students struggled to identify. User descriptions
such as No idea actually, Joe Public, Anyone, and
especially Our tutor (tutors act as clients in our
programme) were not uncommon even from
respondents who rated the quality of their requirements
capture very highly. Fifteen of the thirty-five respondents
either couldnt describe their user, described the marker
instead of the user, or described the user in terms which
were too broad to be of any use. It seems probable that
students in the below average quality band in particular
have been too generous in their assessment of the quality
of their requirements specification. While there is good
correlation between student perceptions of overall system
quality and the grades distribution, this assertion is
unsustainable in this context, and Glass law [7] may be
more broadly applicable than first thought.

Figure 3. Characteristics of software design across

quality bands.
Only six respondents out of thirty-five rated the quality
of their projects source code less than 3 out of 5. While
the quality bands are too small to claim statistical
significance, respondents in the above average band
reported a higher average programming quality than those
in lower quality bands (figure 4). No respondent in the
above average band reported a code quality of less than
3 out of 5, nor did any respondent in a below average
band report a perfect code quality of 5 out of 5 (figure
5). Out of the 34% of respondents who reported that they
used the source code control system provided in our
programme (CVS), all rated their code quality at 3 / 5 or
higher.
Unit testing is an important part of the RWSPs tight
cycle of iterations; the process of detailed design, coding,
and unit testing is a sub-cycle within the larger cycle of
phase N iterations. Despite the importance of unit testing,
respondents reported a median thoroughness of 3 out of 5,
with few responses at the extremes. Respondents in the
below average quality band reported that their unit
testing was less thorough than those in the two higher
bands. Essentially, almost all respondents reported that
the thoroughness of their unit testing was mediocre,
hovering around the median.
Mean Code Quality by Overall Quality Band
5
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0

Mean Code Quality out of

5

Below
Average

Average

Above
Average

Figure 4. Mean code quality by quality band. Code

quality improved progressively in each quality band,
and greater variance was exhibited among the lower
quality systems.

Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)

1530-0803/05 $20.00 2005 IEEE

standards. Nine out of thirty five respondents, or 26%,

reported that their groups employed code reviews.

Opinion of Code Quality by Overall Quality Band

Above Average
1 out of 5
2 out of 5
Average

3 out of 5
4 out of 5
5 out of 5

Below Average

0%

20%

40%

60%

80%

100%

Figure 5. Breakdown of reported code quality

according to quality band, showing the percentage of
respondents in each quality band for each reported
code quality rating. Here the assessment is more
realistic than in the requirements activity.
Code Review, Coding Standards and Code Quality Band

Number of Respondents

7
6
5

1
2
3
4
5

4
3
2
1
0
Used Code Reviews

Used Coding Standards

Used Both

Figure 6. Coding standard and code review usage by

code quality band. Again, code quality is rated 1 to 5.
Respondents rated the thoroughness of their
acceptance testing on a five point scale. Here, the median
rating was 3 and responses were tightly bunched around
this figure. Respondents who reported above average
quality also reported more thorough acceptance testing
(median thoroughness being 4 / 5 compared to 3 for the
average and below average quality bands). It is interesting
that on the whole, respondents rated their acceptance
testing considerably higher than their unit testing, with
the mean unit testing thoroughness being 2.68 versus the
mean acceptance testing thoroughness of 3.17. One might
speculate that students have a higher opinion of the
importance of acceptance testing, or a better
understanding thereof, and therefore place a greater
emphasis on it.
Only half the respondents reported that their group
used a coding standard. Over 50% of respondents who
reported a code quality of 3 out of 5 or higher (16 out of
30) also reported that their group employed a coding
standard (figure 6). Yet only six respondents belonged to
groups which combined coding standards with code
reviews, and they all rated the quality of their coding as 3
out of 5 or higher. Thus there were eleven respondents
whose groups employed coding standards but then didnt
employ code reviews to ensure adherence to those

3.2.2
Respondent-Identified Factors
Students were given several opportunities to identify
factors which influenced the quality of their software
which werent already covered in detail by the other
survey questions. In addition, reviewing the survey
responses gave some unexpected and unsolicited insights
into student practice of the RWSP.
Students were asked to describe those aspects of their
approach which had the biggest single negative impact on
the quality of their software. Eleven respondents reported
that lack of time had the biggest negative impact more
than any other single issue although this response
appeared to be correlated with version control problems.
Students were asked to describe their release planning
and time management, and were given the following
options:
Appropriate amount of work for each release,
enough time.
Appropriate amount of work for each release,
not enough time.
Uneven distribution of work between releases,
but well-timed.
Uneven distribution of work between releases,
not well-timed.
Too much work to be distributed properly
between releases.
More respondents who used the CVS system (figure 8)
reported that the workload for each release was
appropriate - and that there was adequate time in which to
complete work on each release - than those who did not
(figure 7). Having a centralised repository from which the
latest version of the software is always obtainable
significantly reduces overheads. Without such a
repository, students have to arrange to obtain the latest
versions of source files from each other through e-mail
and face-to-face meetings the time spent doing this is
non-productive.
Time Management - Opinion of Release Planning (non-CVS
Users)

Too much work to be

distributed properly
between releases.
17%

Appropriate amount of
work for each release,
enough time.
26%

Uneven distribution of
work between releases,
not well-timed.
13%

Uneven distribution of
work between releases,
but well-timed.
22%

Appropriate amount of
work for each release,
not enough time.
22%

Figure 7. Time management and release planning for

non-CVS users.

Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)

1530-0803/05 $20.00 2005 IEEE

Time Management - Opinion of Release Planning (CVS Users Only)

Too much work to be

distributed properly
between releases.
8%
Uneven distribution of
work between releases,
not well-timed.
17%

distributed throughout all members of my team. As

with all graphs organised by quality band, no claims
of statistical significance can be made, although the
trend is obvious.

Appropriate amount of
work for each release,
enough time.
42%

Median Degree of Teamwork by Quality Band

3.5

Uneven distribution of
work between releases,
but well-timed.
8%

3
2.5

Appropriate amount of
work for each release,
not enough time.
25%

Figure 8. Time management and release planning for

CVS users.
Eight respondents reported that teamwork, or rather, a
lack thereof, had the greatest negative impact on the
quality of their software. Eight other respondents
identified a variety of factors which could likewise be
grouped under this banner as the most significant positive
factor contributing to the quality of their software. This
means that sixteen out of thirty five respondents named
teamwork on either extreme of the most significant
impact on quality scale. The importance of good
teamwork appears to have been established!
Students were asked to respond to the statement the
work was evenly distributed throughout all members of
my team by indicating whether they agreed or disagreed.
The vast majority of respondents disagreed. Students in
progressively higher quality bands agreed more and more
with the statement (figure 9), indicating that the
respondents in higher quality bands came from groups
which distributed the project workload more evenly. If we
rate a groups teamwork by the workload distribution,
then based on the responses to this question, it can be
shown that the quality of software reported by a
respondent increased with the degree of teamwork (figure
10). Rating a groups teamwork by the workload
distribution is probably the best single measure of good
teamwork, because it also tells us that a team
communicates and co-ordinates itself well enough to
equitably and effectively distribute work.
Work Distribution by Quality Band

Above Average
Disagree Strongly
Disagree
Neutral
Agree
Agree Strongly

Average

Below Average

0%

20%

40%

1.5

60%

80%

100%

Below Average

Average

Median Degree of Teamwork (1

to 5)

0.5
0
Above Average

Figure 10. Median degree of teamwork (on a scale of 1

to 5) by quality band. The degree of teamwork is the
degree to which the respondent agreed to the
statement "the work was evenly distributed
throughout all members of my team".
The complexity of the project provides a ruthless
examination of the team, with a successful outcome
critically dependent upon:
Adequate communication between the team and
the industry partner to capture user requirements
and clarify them over time;
The ability of specialists within the team to
acquire new technical skills and to educate their
team-mates; and
The ability of team members to rely upon the
timely contributions of others.
It is our experience that poor performance in any of
these areas is sufficient to ensure a substandard project.
Students noted the importance of communication and the
need for trust between team members, with compromise
and negotiated task allocation essential to team harmony
and a successful outcome.
Given the relationship between software quality and
the software development process, it is important to use a
proven process and to follow it closely, particularly for
inexperienced developers like students. Students wont
necessarily follow a software process simply because it is
the process employed by the programme in which they
are enrolled. If students have a thorough understanding of
the process, they will be more inclined to see it as a
helpful tool rather than as something imposed on their
team. Over 50% of respondents indicated that they only
followed the RWSP closely enough to satisfy their
assignments marker, and many responses indicated
fundamental misunderstandings of the RWSP, and of
software processes in general.

Figure 9. Work distribution by quality band. The

responses refer to the statement the work was evenly
Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)
1530-0803/05 $20.00 2005 IEEE

Over half of the respondents agreed that the iterative

nature of the RWSP helped their teams produce better
software. The responses confirmed our intuition about the
educational advantages of iterative development, with
familiarity and experience reinforcing and supplementing
the lessons of the early phase. A number of students
commented on the role of reflection in improving
technical practice and team performance.

Guidelines

The results of our survey suggested several areas

where additional instruction and guidance could be of
benefit to the quality of student software products. These
are considered in turn below.
Students are much more likely to successfully employ
a software development process if they understand it, yet
in spite of the extensive tutorial support provided as part
of the units and embedded within the process website
itself, many students demonstrated a complete lack of
understanding about the RWSP. It is preferable to present
a software process as a useful tool which will help the
student than to simply impose its use in the assessment
criteria.
Students should be encouraged to define the roles of
all team members early on. Our experience has shown
that teams which include a leader personality type tend
to be more successful and harmonious than those which
do not. The position of team leader (and quality manager,
if the group has one) should be filled by the team member
with the most advanced understanding of the process.
Students should become more familiar with the client
and with the clients specific needs. A quality piece of
software is one which meets the needs of its users, and it
is therefore important that students gain a more thorough
understanding of their softwares users.
When developing software, the simpler, easier to
understand, and more extensible the design is, the higher
quality the software produced. Students in particular
benefit from a straightforward design. Prototyping can
help students determine whether their chosen design
approach is practical before fully committing to it.
Students are encouraged through the use of such tools
as JUnit to unit test more frequently and more thoroughly.
Unit testing should immediately follow any and all
programming, and in the case of the RWSP, it should
form an iterative cycle within an iterative cycle, so that it
is impossible to do any programming without also unit
testing the new code.

Students are encouraged to use a coding standard and

to conduct code reviews to test compliance. Coding
standards lead to more readable and maintainable code.
Ordinarily in student projects, the future readability (and
thus extensibility) of a projects source code is not a
significant concern once the project is submitted, it will
never see the light of day again. However, in agile
software projects, students will have to be able to read,
maintain and extend code from previous iterations.
Coding standards are therefore of particular benefit.
The better a given team works together, the better
quality software it produces. Students are encouraged to
share work as evenly as possible, and to report any
problems as early as possible. It is possible for a third
party (e.g. a tutor or lecturer) to interfere in a group
without running the risk of creating a rift within that
group, and so students are encouraged to report any
problems as early as possible.

Discussion and Future Work

This work has examined quality assurance among

student developers working under a modern agile
methodology one clearly defined, regularly updated and
presented with abundant document and process support.
Notwithstanding our earlier caveats in regard to sample
size, the study has provided a range of insights into the
student agile development team, and informed a number
of revisions to the process and to our approach to
teaching introductory software engineering. A number of
these insights appear to be specific to the educational
environment, while others may have broader significance.
While there is some evidence of convergence between
Agile and more traditional development methods, we
would argue that the lighter approaches necessarily
involve special quality assurance needs, and are by their
nature less prescriptive than earlier methodologies. In
consequence, the most useful prescription to improve
development quality is to ensure that the team has an
adequate understanding of the process and its role in the
development project. To state such a pre-requisite is to
court banality, but the evidence of the student responses
provides a compelling demonstration of quality assurance
failure even when the tasks demanded are less than
onerous. One may reasonably speculate that novice
industrial developers who readily adopt the flexibility
of an agile approach with little attention to the
accompanying discipline may experience similar
difficulties.
The difficulty within an agile student team is to ensure
that sufficient weight is given to these activities. This
problem has long been the bane of software engineering
educators, and it is here that the agile approaches offer

Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)

1530-0803/05 $20.00 2005 IEEE

some comfort, with student responses confirming the

educational advantages of iterative development, with
familiarity and experience reinforcing and supplementing
the lessons of the initial phase. A number of students
commented on the role of reflection in improving
technical practice and team performance. Tight iterations
may similarly have a role to play in developing quality
assurance practices in industry, providing an opportunity
for novice developers to absorb the process more rapidly,
and to attain more rapidly a greater degree of process
maturity.

nurtured an understanding in our students of quality

assurance in the agile environment.
The assertion by certain process evangelists that agile
methods produce higher quality software products
(reported in [8], [9], and [10])
cannot justify
complacency from educators utilising agile processes.
Quality assurance in Agile software development requires
a similar degree of vigilance as that in traditional
environments.
[1]

At a purely educational level, further improvements to

our approach must rely upon some deeper examination of
the factors underpinning project success. Direct
assessment of the software process within a project is
difficult of itself, and still more difficult to isolate from
content influences when teams have barely entered the
software engineering novitiate. Future work will be
geared initially toward obtaining greater numbers of
survey responses, in order to enhance the reliability of the
results obtained. Depending upon ethical approval,
anonymous cross matching of respondents with team
metrics would be of considerable value, particularly in
determining the reliability of individual self-reports of
system quality.

[3]

The question of whether student developers produce

better software using agile approaches is more difficult to
answer, due to a lack of comparability in the data
available from earlier years. Moreover, at present the only
tools available to us in assessing team compliance with
the process are tutor observation and direct examination
of time and bug records. Can compliance with the
prescribed process be certified in such a way as to
provide better assessment of the effectiveness of agile
processes in action?

[7]

Ultimately, the long-term impact of our work can only

be assessed once student cohorts have progressed
successfully into the workforce, and been given
appropriate follow up. However informed the student
body in making judgments, there can be no substitute for
the reflection which accompanies actual practice.

[2]

[4]
[5]
[6]

[8]

[9]
[10]

This study has identified effective quality assurance

practices employed by student agile practitioners, as well
as some of the common deficiencies in their approach.
This allowed us to produce a set of easy to impart
guidelines which should encourage and assist students to
implement effective quality assurance practices, and
which should help them to avoid some of the mistakes
commonly made by student practitioners. We have
incorporated these guidelines into our software
engineering programme, and we believe that we have

Proceedings of the 2005 Australian Software Engineering Conference (ASWEC05)

1530-0803/05 $20.00 2005 IEEE

M. Beedle, A. van Bennekum, A. Cockburn, W.

Cunningham, M. Fowler, J. Highsmith, A. Hunt, R.
Jeffries, J. Kern, B. Marick, R. C. Martin, K.
Schwaber, J. Sutherland, and D. Thomas, "Manifesto
for Agile Software Development," vol. 2003, 2001.
K. Beck, Extreme Programming Explained: Embrace
Change, 1st ed. Reading: Addison Wesley Longman
Inc., 1999.
J. M. Hogan, G. Smith, and R. Thomas, "The real
world software process," presented at Ninth AsiaPacific Software Engineering Conference, 2002.,
2002.
W. Humphrey, "A Personal Commitment to Software
Quality," vol. 2003, 1994.
M. C. Paulk, "Extreme programming from a CMM
perspective," IEEE Software, vol. 18, pp. 19-26, 2001.
J. M. Hogan, R. Thomas, S. Clarke, G. Smith, M.
Adams, and C. Ho-Stuart, "The Real World Software
Project: Improving the professional education of
software developers through Problem Based Learning,
self-guiding Software Engineering Processes and
Software Quality Tools.," Queensland University of
Technology 2003.
A. Endres and D. Rombach, A Handbook of Software
and Systems Engineering. Harlow, UK: AddisonWesley, 2003.
A. Cockburn and L. Williams, "The Costs and
Benefits of Pair Programming," presented at
Proceedings of the First International Conference on
Extreme Programming and Flexible Processes in
Software Engineering ({XP2000}), Cagliari, 2000.
B. Boehm and T. DeMarco, "The agile methods fray,"
Computer, vol. 35, pp. 90-92, 2002.
M. Aoyama, "Agile software process and its
experience," presented at 20th International
Conference on Software Engineering, Kyoto, 1998.