Agile 2008 Conference

Dependency Management in a Large Agile Environment

Eric Babinet salesforce.com ebabinet@salesforce.com Abstract
Salesforce.coms R&D organization has over 30 Scrum teams working simultaneously in a single release code branch. This report highlights practices that salesforce.com has been using successfully to scale Scrum and to manage inter-team dependencies.

Rajani Ramanathan salesforce.com rajani@salesforce.com

2. Team Structure
Product development at salesforce.com is organized into three business units: Applications, Platform, and Core Infrastructure. There are 30 plus Scrum teams distributed across these three business units. Each Scrum team typically has dedicated developers, quality assurance engineers and a Product Owner. Each team also has a ScrumMaster, who is usually a program manager, development manager, or QA manager. Depending on the complexity of the features being developed, the Scrum team may also have dedicated or part-time team members from other functional teams such as System Testing, Documentation, UI Design, Usability, Technology Operations, and Release Engineering. Typically all Scrum teams are working on the same major release and in the same codeline/branch. Given the integrated nature of our products, the features implemented by these various teams can be very tightly interwoven, both functionally and technically. From a technical implementation perspective many teams may be using common shared code. From a functional perspective, teams working on features targeting the same end user need to coordinate closely to provide a consistent and coherent user experience.

1. Introduction
In October 2006, salesforce.com's R&D organization embarked on a large scale transformation from a waterfall software development approach to an agile approach based on Scrum. It had been 10 months since the last major release, the planned release date of the next release had moved five times, and many people were frustrated by the inability to deliver frequently and on schedule. Instead of waiting until the release was completed, we reconfigured existing teams into Scrum teams and used the Scrum process to deliver the in-progress release to our customers in February 2007. Since then we have delivered five major releases (at 3-4 month intervals) of our Software-as-a-Service (SaaS) application suite and Force.com platform using our new agile approach. Every release has gone into production exactly on the pre-planned release date. During the transition, Scrum offered significant guidance for individual teams, but not a lot with respect to coordinating between teams. We organized teams to minimize dependencies, but the code hadnt changed overnight and there were still significant dependencies between teams. We implemented scrumof-scrums meetings early and they were helpful for discussing issues and status, but not sufficient. Over the last five releases we've tried out and refined a number of additional practices to improve coordination between teams. This report highlights challenges weve encountered with respect to dependency management in a large agile environment and how we overcame them.

3. Why is Difficult?

Dependency

Management

With so many teams working together on the same release and on shared features and code, it is impossible to anticipate all the issues, surprises, changes, failures and successes that teams will encounter during software development. That unpredictability is one reason that dependency management is difficult. This section highlights some other reasons why dependency management is difficult. Any solution must address these underlying tensions and challenges.

978-0-7695-3321-6/08 $25.00 2008 IEEE DOI 10.1109/Agile.2008.58

401

3.1. System Complexity

The first step in managing dependencies is to identify them. Like most mature software, salesforce.com's systems are complex enough that one person or team cannot possibly know about all dependencies. Increasingly we must rely on the collective knowledge of many people to identify and understand dependencies. Pooling this collective knowledge and connecting those who have the knowledge with those that need the knowledge is essential. Experts with broad knowledge of the system are highly valued and play a key role in identifying the dependencies and impact of proposed changes. However, as the system grows in complexity it is hard to avoid more specialization and knowledge fragmentation. Given the volume of change it is also no longer practical for these experts to review the detail of every feature being developed. We need a method to identify changes with the highest likelihood of impacting or depending on other teams, so we can focus our attention on those areas. Of course this is not easy, as deceptively small changes can have a big impact.

The ability of teams to change priorities each sprint is a benefit of Scrum, however, it can make managing dependencies more difficult. Dependency identification and analysis cannot just happen once at the beginning of the release, but rather must be an ongoing process. Over the course of a release dependencies come and go as teams refine what they can deliver, and any process for managing those dependencies needs to be equally dynamic.

3.4. Short and Overlapping Release Cycles

Short release cycles mean that dependencies and impact must be understood early as there is less time to coordinate with other teams. Salesforce.coms release cycles are designed with some overlap, meaning that the planning process for the next release starts before the prior release is completely finished. This is intentional as some teams are completely finished and need to be able to move on to the next release. However, teams that are not finished may fall behind in their release planning and be less available to discuss next release dependencies with other teams. This is problematic given the importance of collective knowledge in understanding dependencies.

3.2. Conflicting Priorities

Dependencies can lead to conflict between teams. If one team needs another team to do something, the product owners for those teams must negotiate the relative priority of that work. If they negotiate effectively and reach a common understanding for when the work can be done, then managing that dependency should be relatively easy. If they dont reach a clear agreement, or if the work is still a lower priority for the other team, there will be greater uncertainty and risk around that dependency. Another type of conflict occurs when one team makes a change that imposes work on another team. A common example is when one team does architectural refactoring that requires supporting changes by other teams. The impacted teams may get no short-term benefit from these changes, and may resent having to make them, especially if the need for them is unexpected. We attempt to identify these types of changes early in the release, but inevitably there are some that show up later in the release, either because they arise opportunistically or due to a change in priorities.

4. Specific Practices
This section describes the specific practices we are using to manage dependencies and address the above challenges. Some common strategies incorporated into these practices include: Create visibility: for release plans, dependencies, designs, build status, etc. Provide forums to stimulate collaboration and knowledge sharing between teams. Promote self-organization and decentralized dependency management. Automation, automation, and more automation.

4.1. Release Kickoff

About one month prior to the current major release going live in production, we hold a release kickoff meeting for the next major release. This is an all-hands one hour meeting in which the Senior Vice President for each business unit gives a high level presentation of the features that each team is planning to build in the next release. This meeting has a number of benefits with respect to dependency management. First and foremost, it motivates all teams to complete their initial release plan by a specific date. If a team has leftover work from the current release they can fall behind in planning for the next release, but the Release Kickoff

3.3. Dynamic Scope

402

helps teams stay on track with that planning. It is difficult to identify dependencies or negotiate commitments with other teams if those teams don't yet know what they're doing in the release. The Release Kickoff meeting acts as an important synchronization point for teams, and helps ensure more productive discussions around inter-team dependencies. The second major benefit results from the visibility that the meeting creates around team release plans. Our goal is to generate a high level of awareness of what teams are doing in the release, as we believe that this visibility will naturally cause people to think about and discuss dependencies. In order to encourage attendance and focus attention on the release plans, the meeting is given a high-profile and all R&D executives attend. The high level of participation helps align expectations and create broad awareness of what is planned for the release. Of course, team release plans can and do change throughout the release. We recently started reviewing an updated version of the Release Kickoff presentation during the monthly sprint reviews. The updated presentation highlights features that have been dropped or added to the release, and has proven to be very effective at maintaining release plan visibility throughout the release.

each Scrum team in a room with a large white board. Usually the representative is the ScrumMaster or Product Owner from the team, but it can be anyone. The first part of the meeting involves all team representatives going up to the whiteboard simultaneously and creating two diagrams representing the dependencies between teams. One diagram captures teams that need another team to do something for them. The second diagram captures teams that are doing something that will impact other teams. For these diagrams we use a simple notation: A circle represents a team An arrow between two circles represents a dependency. On the first diagram the arrow points from the team doing the work to the team that needs the work done. On the second diagram the arrow points towards the team that is impacted. A label on the arrow describes the dependency If a dependency is committed (i.e. the other team has agreed to do the work) we put a box around the dependency label. At first there is a commotion as everyone draws their dependencies on the whiteboard and onlookers start asking questions and pointing out dependencies that are missing. After about 20 minutes the diagram stabilizes and comments subside. At that point we ask each team representative to briefly describe their dependencies. In a very short amount of time we generate a comprehensive view of the dependencies in the release. Figure 1 shows the typical output of this meeting after it has been copied into a more readable format. Hotspots (i.e. teams with a lot of dependencies) are immediately visible. Teams that have not yet obtained commitments for their dependencies are also visible. These teams have a higher risk of not delivering on their release plans until the dependencies are committed.

4.2. Dependency Identification Exercise

Once teams have their initial release plans, they can have a more meaningful discussion about dependencies. There is a lot of informal discussion between teams to negotiate dependencies, but we have found it very valuable to facilitate a more formal Dependency Identification exercise. This is a highly interactive and collaborative event that involves representatives from all teams. We have found it most effective to do the exercise shortly before the Release Kickoff as it improves the quality of the release plans presented at the Release Kickoff. The actual logistics of the Dependency Identification exercise are simple. We gather representatives from

403

Figure 1. Example team dependency diagram

4.3. Release Open Space

We recently started holding an open space style meeting during the week after the Release Kickoff. The purpose of this open space is to create a forum for discussing questions or concerns regarding team release plans. We ask that at least one representative from each Scrum team attend, but otherwise the meeting is optional. In standard open space style, attendees propose topics at the beginning of the meeting. Then we have 45 minutes for breakout discussions and 30 minutes for reports from the breakout groups. Popular topics include wanting to know more about functionality that is new and which can benefit other teams, and wanting to know more about changes that will impact other teams. Attendees have reported the open space to be educational and have found it interesting to be in discussions with people with whom they do not normally associate. Since the open space meeting format is new to the

organization we are experimenting with variations to see what works best, including:

some

Generating topic ideas prior to the open space Encouraging senior management participation Changing the number and length of breakout sessions

4.4. Functional Design Reviews

The goal of the functional design review meetings is to improve the overall quality and usability of our products by: leveraging the collective wisdom and creativity of our product teams improving design coherence across products creating synergies through cross-team knowledge sharing

404

These are cross-functional and cross-scrum team meetings that focus on the functional and user interaction design of features deemed to be higher risk, higher complexity, or higher impact. These meetings are intended to break down team silos and surface design inconsistencies and dependencies of which the Scrum team may be unaware. There are two different parts to the functional design review meetings. Early in the release cycle the discussions focus on design concepts and aim at achieving functional design coherence and synergy between existing and new features. These discussions are usually attended by the product owners of each Scrum team and UI designers, and there is little representation from other functional teams such as development or QA. Starting near the middle of the release cycle the participants change to primarily include QA and some development team members, and the focus is on reviewing implementation details. These discussions provide insight and clarity around: Additional dependencies between teams Additional integration testing that needs to be added to the test plan The release risk of a particular feature, and perhaps the need to restrict availability of that feature Deployment specific tasks

least 20% of their time every release towards implementing changes recommended by the VAT. The VAT meets for two hours twice a week to review the technical implementation of products and features being built by the Scrum teams. The teams building the most complex features in the release are asked to present to the VAT. The group provides valuable feedback to the Scrum team on how their technical design will impact or be impacted by other areas. The VAT focuses primarily on the technical implementation, especially scalability and performance considerations. Teams asked to make significant changes must present again in the same release cycle and provide details on how they modified their design.

4.6. Continuous Integration

We have a web-based automated build, test and triage infrastructure that provides a continuously integrated build system and allows us to track the health of any given codeline. This infrastructure is critical for enabling all developers and QA engineers to work in a shared clean codebase. The primary test suite is built using an extension of the JUnit framework and provides a mechanism for implementing functional tests through the Force.com API. We also have a UI testing framework using Selenium for automating test cases that need to be executed using the UI. Some fundamental tenets guiding our automation approach are: 1. Provide fast feedback to developers so they understand the effects of their changes. Each checkin triggers a build and a subset of tests to be run. Build failure notifications are sent to the responsible engineers immediately. The basic test suite runs within half an hour and the developer is notified of the results of their change. Extended test suites run periodically throughout the day. 2. Fix build and test failures quickly. To ensure this, we have a rotating build master role that changes each week. This role is typically filled by two people, a dev manager and a senior developer, who coordinate closely. The build master is responsible for monitoring build results, tracking test history, tracking failures across different code lines, and assigning bugs to fix failures. The test results are logged in a database and metrics are collected on a central dashboard (% of tests passing, build times, number of failures, etc.) The build master also uses this data to send out status emails. 3. High automated test coverage. Scrum teams typically aim to have over 70-80% of code covered by automated tests. Our large collection of automated tests

4.5. Virtual Architecture Team

The Virtual Architecture Team (VAT) is "virtual" as it is made up of developers from every Scrum team. Members work on the VAT in addition to being on a Scrum team from the Application, Platform or Core Infrastructure business units. The VAT owns maintaining and extending our industry-leading software architecture. They do this by defining the architectural road map, by reviewing architecturally significant changes to the code, and by defining coding standards to ensure consistent and maintainable code. The VAT maintains a backlog of major architectural projects or refactoring changes that need to be implemented. As the product keeps evolving, we need to redesign features and remove old code that is no longer optimal. Developers are sometimes reluctant to tamper with programs that already work, even when they know it would be better if coded differently. Product Owners share their reluctance as they would rather see new features developed than have something that already works be rewritten. To counteract these tendencies, the Application, Platform, and Core Infrastructure business units are asked to allocate at

405

is one of our biggest assets and a key enabler for delivering high quality on-time releases every 3-4 months.

5. Conclusion
Salesforce.com has demonstrated that it is possible to scale Scrum and we feel very confident in our ability to continue scaling as the organization grows. Dependency management and cross-team coordination are a significant challenge as the number of teams increases. Having a robust build and automated test infrastructure is absolutely critical. Increasing visibility and alignment through practices such as the release kickoff, the dependency identification exercise, scrumof-scrums, and lightweight status reports is very helpful. Ultimately it comes down to effective communication, collaboration, and knowledge sharing between teams. Practices such as the virtual architecture team, the functional design reviews, the release open space, and the scrum-of-scrums are all designed to open communication and promote collaboration.

4.7. Scrum-of-Scrums
Each business unit has a weekly scrum-of-scrums meeting and there is also a weekly scrum-of-scrum-ofscrums. Occasionally we will form additional scrumof-scrums if there is a cluster of teams working closely together on a common goal. We have experimented with a few different formats for the scrum-of-scrums. We started out using the standard "4 question" format where teams report on what they've done, what they're going to do, what is blocking them, and if they are doing anything that will impact another team. That worked fine initially, but became a bit tedious due to the large number of teams. We have since moved to a more open and self-organizing format, in which attendees propose discussion topics by writing them on the whiteboard at the beginning of the meeting. This approach leads attendees to take responsibility for the content of the meeting and has resulted in more productive and collaborative discussions. The topic of cross team dependencies does come up during the scrum-of-scrums, especially early in the release cycle. The Dependency Identification Exercise is held during a scrum-of-scrums meeting and for the next two to three weeks after that we will typically discuss changes to dependencies during the scrum-of-scrums.

4.8. Status Reports

Written status reports are often discarded when a team moves to Scrum as visibility into team status is provided by other means (e.g. sprint review, sprint burndown chart, daily standup). When we adopted Scrum we decided to keep a lightweight status report written weekly by each ScrumMaster. When we were using the "4 question" format for the scrum-of-scrums there was definitely some redundancy between the team's scrum-of-scrums update and the team's status report. However, now that we use the open format scrum-of-scrums, the weekly status report is actually an important complement to the scrum-of-scrums. We don't discuss individual team status in the scrum-ofscrums unless specifically raised as a discussion topic, however, the status is always available in the weekly report. Dependencies, blockers, and risks are all highlighted in the report. While the report is a little extra work for ScrumMasters we feel that the visibility it provides is worth the cost.

406