Using Agile Project Management with Integration

Projects
I have been recently involved in a number of BizTalk 2004 integration projects using Agile Project
Management (Also known as Scrum). Conchango (http://www.Conchango.com) has also been
involved with Microsoft in designing and implementing the Agile add-in for VS2005.

If you are unfamiliar with Agile Project Management, have a look at the following links:

http://www.controlchaos.com/ (Its all about common sense) The official Website with all the
resources
http://www.scrumalliance.org/ For Certified Scrum Master Resources
http://wiki.scrums.org/index.cgi?HomePage and http://c2.com/cgi/wiki?ScrumProcess

Background

First, a bit of background, I don’t want to assume that you are an expert on the subject and will try to give a
bit of info to get you going for this post.

Agile project management was introduced for development projects. The standard sprint duration is 30
days and at the end of each sprint the project team presents “Production Quality” features. At the
beginning of the project the team organise their deliverable (Called Products) in a “Product Backlog”. A
representative (Called Product Owner) from the stake holders helps them identify the products and
prioritise the product backlog. The team starts with the most valuable tasks first, then the less valuable till
they deliver the entire system or run out of budget. The emphasis is on two things: The stake holders are
seeing results very early on. So after four working weeks the team would have produced real business
value. The second is that the products of each sprint are production ready. So, if the stake holders decide
to put this to live today, they can do that.

The Chickens and the Pigs

The first time I was called a chicken I was insulted! Then I was called a pig and I was insulted again, well
not really because by then I knew what it meant! The story is that a pig and a chicken wanted to open a
restaurant. The pig asked: What should we call our restaurant, and the chicken replied: Let’s call it Ham
and Eggs. The pig thought about it for a minute and said, I can’t have a restaurant with you because I
would be committed and you would only be involved. Therefore, only the scrum master, product owner,
and members of the team are pigs (Committed), all the rest are chickens (Only involved). The advantages
of this are countless: How many times were projects ruined/delayed, went over budget and/or lost focus
because of restrictions imposed by non relevant people. How many times did meetings drag on because
you had to involve everyone in the department? How many times where you misinformed because you
could not find the correct answer or talk to the right person. Being a pig (If you excuse the expression) not
only empowers you, it also clears unnecessary obstacles, and gives you direct access to other pigs.

The scrum meeting

The scrum meeting is the hallmark of Agile project management. It is a daily meeting, capped at 15
minutes, held in the same place, only pigs are allowed to talk, chickens can listen. Each person (Pig) tells
the team what he/she has done yesterday, what he/she are intending to do today and finally mention any
impediments they have.

Tip1: Have the scrum meeting standing up and away from your desks. This ensures people focus on the
meeting, and prevents the quick peak to the old email inbox
Tip2: Pigs stand in a circle, chicken outside the circle. This enforces the separation between committed
and non committed resources
Tip3: Establish a fine for those who are late for the scrum meeting. Mine is a pack of Bahlsen Choco
Leibniz dark chocolate biscuits (http://www.bahlsen.co.uk/), or two if you someone has been really
naughty!! Most people though tend to gather the money and later donate it to a charity of their choice.
Tip4: The daily scrum time is very important. If you make it too early, you might have people stuck in traffic
and start having problems with people missing very often. Having your scrum meeting too late, say at 10
am or 10:30 am, people will tend to do whatever (not work related) waiting for the scrum to begin. Scrum
time of 9:30 am is on average a good time.

Semi-committed pigs*
Sometimes you require resources for a short time only, or have to share them with other projects. I call
these Semi committed pigs. The first problem with having semi committed pigs is time management and
estimation. They will often say: “I have to work on this other project, so you can have 40% of my time.”
This is very difficult to estimate because they will not always give 40% every day. The second problem is
that you always try to have the team in the same room. Semi committed pigs will generally not be able to
do that because of their other engagements. This introduces plenty communication issues.

* The term “Semi-committed pigs” is truly my term and you will probably not find it in any of the books. If
you do find an equivalent please let me know.

Why use Agile for integration projects?

So far I have been involved in four (and counting) BizTalk project using Agile. I am a certified Scrum
Master. I had the privilege of working on the first integration scrum project ever. The following is why I
think Agile is very well suited for Integration and BizTalk Projects.

Communication
Having daily scrums with relevant involved stakeholders is the biggest advantage Agile projects have over
other Traditional approaches. In addition, there is a benefit of delivering functionality early to business
users:
- The feedback is much more relevant to what has been delivered so far.
Therefore if the delivered functionality is not completely correct, it
could be easily altered as you did not build any additional functionality
on top of it, and the development knowledge is still fresh in developers’
minds
- The feedback is immediate, and updates the team’s knowledge of the
system. This is very important especially if in the next sprints there will
be functionality that either uses this feature or extends it.
- Reprioritising functionality: Very often stakeholders will re-evaluate
their priorities because they have identified additional benefits in what
has been delivered making other products on the backlog either
redundant or less attractive. This is key as, in traditional projects, we see
large budgets being spent on features that are rarely used. This is
because the business did not get to see the solution until development
was completed for all the features of the system.

Project Inception vs. Scrum Sprint 0

In traditional waterfall projects business analysts, technical architects, and project managers spend
considerable amount of time identifying the business requirements, business processes (Existing and
changing) existing legacy applications to integrate with, human interaction, etc. They try to define this for
the entire product they are supposed to deliver at the end of the long term project. Not only is this a very
complicated task, but by the time you reach, say, user acceptance, this is often outdated because
everyone would have gained much better understanding of what actually needs to be done. Also the
complexity of the systems we usually build forces us down routes that were not anticipated at the
beginning of the project. The end result is a change request process that usually drags on, as well as out
of date documentation. The other problem with the waterfall approach, and I had this recently, the
inception process is very loose and undefined by definition. The room for error is very large especially that
the waterfall model does not enforce the concept of daily update meetings. The result is that business
analysts, Technical architects, PMs would have initial project meetings with the stakeholders, go away for
few weeks and think about it, and come back with a design and a project plan that they feel would solve
the problem. Doing this, the stakeholders are led down a road that the “Experts” thought through, and they
will trust them to have considered all the options. The point is, that there was no daily involvements from
the stakeholders on the decisions being made during this very important phase of the project. The end of
inception is usually one or very few options that are presented and discussed. In my experience, we
started a project using scrum after the inception phase produced an invalid solution. I was lucky because I
was critical and questioned everything before I started development. It took some time to discuss the
shortfalls of the proposed solution and convince my client to adopt what I called “Sprint 0” or Sprint Zero.
Because it was agile, we had daily scrum meetings with a representative from the stakeholders, a product
owner. Every morning we discussed our options, and made decisions together because we were on the
same level. At the end of sprint 0, we presented our alternatives to the rest of the stakeholders and really
felt that by that time we have reached a decision together. Everyone was much more involved just
because they were part of the decision making process.

So what should be in Sprint 0?

Basically this should include everything to get the project started with Sprint 1. Therefore, this sprint could
include everything from identifying the business case for the solution or project, identifying business
benefits, revenue potential, and cost savings, present the different cost options, rollout strategies, etc This
sprint could also include decisions regarding vendor selection, pricing models for hosting systems, leasing
hardware, evaluating Components Off the Shelf (COTS), identifying the tasks that could potentially be on
the product backlog, and the team structure that could be required and their skill sets. In one project we
talked about peeling the onion, i.e. identifying the layers of complexity and costs with the knowledge that
we still have to be agile and account for changes and modifications in later sprints. At the end of sprint 0
you should end up with a working product backlog, or at least one that will let you start the first sprint. Also
consider that you might finish the project without doing all the items of the backlog, as well as being
prepared to stop the project after each sprint review and deploy your application to production.

Sprints and Sprint Duration

The standard sprint duration is 30 days. Sprint durations should be equal. I used 10 day sprints for my first
agile BizTalk project. The advantage is that it fits with the average time you need to build a demonstrable
BizTalk Project, or part of it. In that project I had to write a custom BizTalk adapter that allows us to
integrate with Chip and Pin devices. That was easily done “Production quality” in a ten day sprint. On the
other side of the interface was integration with the AS400 system using Attunity. Again this was easily done
in a ten day sprint. I passed this by Ken Schwaber (The father of Agile) and he was fine with. He said: As
long as you are fine with it I’m fine with it! The disadvantage of a ten day sprint was that external vendors
usually fund our pace overwhelming and struggled to keep up with our demands. We were too agile for
then and I think they were not used to that. The second disadvantage was with Semi committed pigs. The
short sprints allowed little room for error and if the semi committed resource is stuck with another non
project related task you could not compensate easily.

Architecting SOA the Agile Way

It is now agreed that SOA increase the flexibility and the reusability of software components. As enterprise
applications evolved and the adoption of SOA increased, integration among all the services became
critical. By definition, each service is an independent component, has its own schema(s), unit tests,
architectural views, etc. Architecting SOA the agile way is about incremental delivery of production quality
services. This could mean that in certain sprints the development team will deliver production quality
incomplete services but they are complete in the context of the sprint. In other words, the service would do
the functionality required for this sprint and subsequent sprint will extend that service further. The result is
a business targeted delivery rather than an IT based approach. At the end of each sprint, it is the decision
of the stakeholders if they would want to move this service to production and reap the rewards of that
functionality. This is a very strange concept to developers as they are used to dealing with complete
architecture, and technical specifications when they start coding. In fact, if the agile process is
implemented, the development team should have an incremental architecture with every sprint, which
should be much more accurate and up to date in comparison with what would have been produced in a
waterfall planning session. Architecting SOA the Agile way is really about implementing corrective
incremental steps towards the end result. In golf talk, in a waterfall context the architect is expected to
score a “hole in on” every time, while Agile is all about taking steps in the right direction.

Coping with Change

If I had a pound for every time someone changed his/her mind on a project I would have been a very rich
man indeed. Actually, it is more likely that business analysts, stakeholders, project managers, developers,
and admit it even architects, will change their minds during the duration of the project. In fact, the longer
the project, the more likely change will be introduced. Last year I worked on a large project integrating
SAP, Siebel, and AS400 legacy applications. Change was the only thing that hasn’t changed throughout
the project. Even during “Code freeze” periods, we where often asked to implement the changes in the
middleware interfaces because of the code freeze in SAP and Siebel. Modular delivery does help cope
with change, for a start you don’t have to design everything up front and change it later. In addition, you
don’t have to spend a lot of time designing everything only to implement part of it, or even none of it.
Delivering incrementally will put the team in the right frame of mind required for the change because you
only plan for this sprint, and if things change then you will plan for them in the next sprint and so on. Of
course if things change for this sprint then the team will have a problem, but I think this is where a shorter
sprint duration comes in handy. In integration projects things can change all the time. Even when we were
doing the SAP and Siebel interfaces we knew exactly needed to be delivered, but testing was an issue; not
everything could be tested the same way or by the same testers. Therefore planning for what takes place
when and what to work on next is a challenge. Shorter sprint durations, say ten days, might be an
advantage during some integration projects, especially if you want to deliver small modules and reach you
goal quicker coping with change along the way. On another project I had to share some resources, and my
project was not the most critical. With shorter sprints I could re-organise the sprints to cope with resource
availability, a situation not ideal but will be the difference between frustrated demoralised project team, and
well, not!

And Finally
These were few points from my personal experience, and every time a new agile project starts and
finishes there are loads to learn. If you are still reading, I hope you have found this useful, and I look
forward to your comments and questions