Research report released June 2012

$250

Measuring Developer Program Performance 2012

Table of Contents
Table of Contents ...................................................................................................................... 1 Table of Illustrations ............................................................................................................... 2 Preface .......................................................................................................................................... 3 Recent trends ............................................................................................................................. 4 Background................................................................................................................................. 6 Findings ........................................................................................................................................ 7
Measurement? Crickets. .................................................................................................................. 4 Genesis.................................................................................................................................................... 6 Respondents ......................................................................................................................................... 7 Observations ..................................................................................................................................... 11 Measuring program performance ............................................................................................. 11 Measuring performance of developer program staff ......................................................... 14 Developer relations in their organization .............................................................................. 17

Recommendations................................................................................................................. 19 About Developer Think Tank ............................................................................................ 22

Measuring Developer Program Performance 2012 Developer Think Tank LLC

Table of Illustrations
Figure 1: Company or organization type ............................................................................ 7 Figure 2: Country ................................................................................................................ 8 Figure 3: Number of employees in entire organization...................................................... 8 Figure 4: Number of employees and contractors in developer group ............................... 9 Figure 5: Age of formal developer program ....................................................................... 9 Figure 6: Recruiting developers to use and contribute to a platform or build applications ................................................................................................................................... 10 Figure 7: Time spent working with developer program ................................................... 10 Figure 8: Measuring program performance ..................................................................... 12 Figure 9: Program performance: Applications or code .................................................... 12 Figure 10: Program performance: APIs or tools ............................................................... 13 Figure 11: Program performance: Site engagement or conference or workshop attendance ................................................................................................................ 13 Figure 12: Program performance: Revenue generated or expenses reduced ................. 14 Figure 13: Program staff performance ............................................................................. 14 Figure 14: Comparing the metrics used to measure program performance and personal performance ............................................................................................................. 15 Figure 15: Program staff performance: Specific measures of applications or code ........ 15 Figure 16: Program staff performance: Site engagement or attendance at conferences and workshops .......................................................................................................... 16 Figure 17: Program staff performance: APIs or tools ....................................................... 16 Figure 18: Revenue generated or expenses reduced ....................................................... 17 Figure 19: Department...................................................................................................... 17 Figure 20: Senior management's view of program........................................................... 18 Figure 21: Expected change in program headcount ......................................................... 18 Figure 22: API Growth over time (courtesy of Programmable Web, February 6, 2012) .. 20

Measuring Developer Program Performance 2012 Developer Think Tank LLC

Preface
Developer Think Tank is the outgrowth of years of working with developer programs. Fundamentally, it is based on our belief that more and more companies will take a platform approach and introduce APIs, either as an offensive move to create some competitive advantage, or as a defensive move as the actions of other companies and changes in consumer behavior put pressure on revenue and margins of firms with traditional business models. From our earliest days of working with developer programs in 2000 we have seen organizations treat their developer programs as strategic, yet have little sense of the way in which to measure them, save for the most obvious: Number of members. While this is appealing in some respectsits easy to measure and its usually more generous to programs than other choicesit is probably the furthest possible from a useful metric to link an organizations efforts with developer program results. In fact, measuring members incents behaviors that generate new members, which is usually an expensive undertaking. Without looking at whether members use the companys API, the main focus of the developer program remains far apart from activities that will help the company advance their business. Just as important, when the number of members is the primary measure, spending is evaluated based on the number of members (i.e., anyone who signs up) added from a given activity, not the number of target market developers recruited (i.e., developers who are going to use the API to build things that advance the efforts of the company). Its with this as a backdrop that we took a close look at the way in which organizations are measuring program performance today.

Measuring Developer Program Performance 2012 Developer Think Tank LLC

Recent trends
Today launching an API and a developer program requires very little in terms of staff time, infrastructure investment, or developer communications. This fact combined with rapid technology adoption that can result from making an API available is one of the reasons we see companies as small as three-person start-ups releasing an API and announcing a developer program. Formerly Staff Building an API Deploying an API Publicizing the availability of an API Recruiting developers Supporting developers Tracking Today Part of an engineers time Tools speed development Platforms offered as a service handle this Social media; sites that list APIs available; evangelists Build a useful API and publicize its availability GitHub, vBulletin, GetSatisfaction, Zendesk, member-to-member support Via deployment platform

Measurement? Crickets.
A few months ago we realized that, despite all of the changes over the last 10 years, little has changed in the area of measurement. Someone at a large developer program posted a question about measurement to a group of developer evangelists and there was zero discussion. Crickets. Given the focus of our work at Developer Think Tank, we realized that this was an important area that was ripe for study. However, we didnt want to run into the problem where only a handful of people have a well-thought-out way to measure their developer programs. As a result, we added questions not just about the way they measured programs, but also program staff. Over a five-week period last fall we heard from the members of developer program across a wide range of programs, by size and age of program, company size, and industry.

Measuring Developer Program Performance 2012 Developer Think Tank LLC

The results confirmed some of what we believed when we undertook the research: The number of members remains widely used as a way to measure the program, but most organizations have added at least one other important measure. Most individuals are measured on activities that are more specific than simply number of members. Company management is viewing these programs as more strategic and is investing more resources in them. The report takes a close look at these and other key trends. As always, we welcome your comments: Research@DeveloperThinkTank.com. Lee Wright and Kevin Sharp Founders Developer Think Tank June 14, 2012

Measuring Developer Program Performance 2012 Developer Think Tank LLC

Background
Genesis
From the outset, one of the areas of interest for Developer Think Tank has been measurement, both of the program and of the key staff within the program. The topic was raised on a LinkedIn group of developer evangelists. Seeing no posted responses, we contacted the person who posted the question to find out if she had received any responses directly. She hadnt, which was a particular surprise given that the question was asked of key individuals engaged in each developer program. We concluded that this was a bigger issue than even we had believed, and launched this research a few weeks later

Process
The research was conducted using an online survey available from July 30 to October 7. Individuals were made aware of it through networking, blog posts, Tweets, and posts to the appropriate LinkedIn groups. (This exercise confirmed again our sense of the value of Developer Think Tank since there is no site that serves as a community for developer program managers, evangelists, and others involved with developer programs.) All of the survey questions and other communications were in English. It was made clear that no personally-identifiable information would be released. Participants could enter their information anonymously. A drawing for $100 certificate was offered to participants, who could provide a personal e-mail address if they wished. Each respondent was examined to spot individuals outside of the target group. In addition, the responses to each question were examined for outliers, which were validated or removed.

Measuring Developer Program Performance 2012 Developer Think Tank LLC

Figure 4: Number of employees and contractors in developer group

More than half of respondents were in fairly new developer programs, one-third of respondents in developer programs that were less than one year old (31.4%), and another one quarter had developer programs that had been around one to two years (28.6%). About one-quarter had been in existence for 8 or more years.

Figure 5: Age of formal developer program

The titles of participants covered a wide range and included CEO, API Evangelist, Application Development Director, Developer Advocate, Developer Engager, Developer Evangelist, Developer Marketing Manager, Developer Program Product Manager, Director of Strategic Marketing, Director of Developer Evangelism, DirectorOpen Platform, Director of Worldwide Developer Technical Support, Lead Developer
Measuring Developer Program Performance 2012 Developer Think Tank LLC 9