Transcript of BriefingsDirect Podcast on Developer Productivity Metrics

Edited transcript of BriefingsDirect[TM] podcast on developer productivity with host Dana Gardner, recorded Nov. 29, 2006.

Listen to the podcast here. Podcast sponsor: 6th Sense Analytics.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today, a sponsored podcast discussion about developer productivity. Interestingly enough, some 9.5 million developers around the world are toiling away, and their bosses -- either individually or collectively -- have rather small insight and limited information about what they do and how they do it.

We see requirements go in one end. We see code come out the other end. And there’s often a great veil, cloud, or fog over what detailed activities develop the solutions and the processes that create good code. To discuss this need for greater insight, analytics, and data around developer productivity -- joining us for the first time -- we have Greg Burnell. He is the chairman, co-founder and CEO of 6th Sense Analytics. Hi, Greg.

Greg Burnell: Thanks Dana, glad to be here.

Gardner: I’ve painted a portrait here about lack of insight. You’ve been in this business for going on 20 years. Tell us a little bit about the history of how development came about. Even though we’ve had tremendous productivity benefits and improvements, it’s still somewhat of a "dark art," if you will. Give us some context on the history of how we got to where we are today?

Burnell: I might even roll back to some of your opening comments, Dana. You characterized it from a management perspective. I'd suggest, too, that individual developers don’t have a lot of insight into the day-to-day conduct of their position. All stakeholders in the lifecycle have an equal visibility problem.

Application development is a knowledge-worker-centric thing. You’ve got very bright, very talented people, but they work in an isolated format, and in many instances there aren't a lot of tangible, visible outcomes to their work. We’ve always kept our hands off of application development. We’ve felt like it’s an area that we don’t want to manage too heavily or monitor -- or even measure, which is an absolutely forbidden concept.

We've lived in this world for many, many years, introducing tools, technologies and processes to try to help people do this better. The fact is that while software drives a lot of the growth and innovation throughout the world we do a very inadequate job of bringing visibility to this whole process. I think it’s because people work in such an insulated manner, and they feel that it’s not possible to measure it. We’re on a mission to change that.

Gardner: I suppose that the insight needs to happen on an individual basis for what the individual developer is up to, but also in a context of a collaborative environment -- with teams in one locale, and increasingly teams distributed all over the world.

Burnell: Absolutely. There are a couple of phenomena. You talked about globally distributed development. We all think we’re aware of the global nature of work that occurs today. Application development and IT support resources are being managed all over the world, but I think the challenge is that application teams have never really focused on providing the same data and visibility to all people in the process. I wrote something yesterday and said if the data isn’t available and accessible to all the people that are part of it, then it really doesn’t have any meaning.

Data that the individual doesn’t see is relatively useless, and data that individuals have -- but that never really makes its way to the manager in an aggregated way -- doesn’t have a lot of value. What we’re trying to do is bring to everyone the same level of visibility, so people can talk about the same things and understand themselves all in the context of the execution of the development project.

Gardner: This seems to be a long time in coming. I mentioned that there are 9.5 million developers around the world. That number's growing. Even the definition of a developer is growing -- business analysts can now be considered developers, given the level of tools and how processes are being brought together.

There have been tremendous insights brought to other types of activities. We’ve applied sciences and statistics, economics and theory to a lot of different activities in the last 10 or 15 years. Why do you think it is that development, while important in growing part of the economy, has not been looked at through a scientific lens?

Burnell: It’s a great question. I’m not sure I can answer why it hasn’t happened, to be honest with you.

I can tell you a couple of trends that we saw, that we think lent itself to the creation of a company and an opportunity like 6th Sense. In years past, there was just an enormous number of vendors out there providing application tooling -- tools for infrastructure to build software.

One thing that happened in the last decade is that there’s been a lot of consolidation in that industry. Also you have companies like Serena, IBM, Rational, Compuware, and Borland. They’ve kind of assembled what we call an SDLC, a Software Development Life Cycle -- all of the tooling for the end-to-end lifecycle app-dev project.

So, these consolidations help, because now you just don’t have a ridiculous number of technologies from which you try to cobble together the whole best-of-breed solution. You go out and buy some type of a lifecycle from somebody. The other thing that’s happened is that people have had to plug their own tools together. In order to do that they’ve had to build these robust connectors or APIs -- ways in which these tools can work together.

That migration and consolidation in the industry and the creation of all of this open infrastructure are what created the opportunity for 6th Sense. That’s why we can participate with those tools. We have access to those toolings, and we don’t have to worry about supporting 10,000 tools. We have to worry about supporting probably in the neighborhood of 100 to 200 tools, which is much more doable.

That transition in the industry has really created opportunity to do it in the way we do it. There have been other attempts, manual processes. I don’t think that people are reluctant to do it. I think they’ve tried to do it in many ways, but they’ve always found that, when push comes to shove, and the project delivery’s in jeopardy or the schedule is slipping, that the first thing that gets thrown out the widow is any type of management oversight. With the technology we put together, we’re unobtrusive. We don’t take away from the process. We empower the process, and I think that’s important.

Gardner: I see. Previously, many tools were involved, and it was difficult to get into each one and have insight into the individual idiosyncrasies that could then be relayed back to give a general view of productivity, habits, and work. At the same time, there was a great deal of dynamics and change in the marketplace, so the tool that was popular three years ago might not be very popular now.

It might not have made business sense for someone to go in and make an investment to have insight into a tool that actually might fall from favor in a fairly short amount of time. I bring this up, because as a hosted service, 6th Sense Analytics takes advantage of this software-as-a-service (SaaS) trend that also gives you insight into activities, but without being really embedded into the technology. Is that right?

Burnell: Absolutely. The software-as-a-service model reflects the fact that people are building software all over the place, and teams are distributed all over the place. Organizations have come to an awareness that they don’t want to continue to build out this IT infrastructure for no value received, when they can go to a vendor. We have security. We have all the permissioning and all the facility skill. We have a level of security equivalent to what they have in their IT shops. They’re looking using the software-as-a-service model, because it has a much lower cost infrastructure.

The other day, I was reading an article that was saying, “Is Eclipse the IDE Killer?” Eclipse is a phenomenon in our industry. It walked in and now dominates development at the IDE level. Few people saw that coming, and all of a sudden here it is. IBM certainly saw it, and that’s a huge trend that we get to leverage.

The two major tools in the industry are Microsoft Visual Studio and Eclipse, and everybody else is a distant number two in those spaces. Software-as-a-service is an interesting opportunity for organizations to have through the web, and through all the of the infrastructure that is in place from a high-speed perspective. Now, we can really distribute applications very quickly, and we can evolve our application very rapidly to benefit all of our customers at the same time.

Gardner: I suppose the development process has also been sort of a whipping boy, if you will. People like to point to low success rates when it comes to completion time and when it comes to full quality and requirements criteria that were set out. So, people often come away with this impression that development is a haphazard and losing-proposition of some kind.

I think that this propels the notion of development as a cost center -- rather than an innovation center -- and is really problematic. Part of the reason for that is just a lack of transparency into what’s going on. Before we get into the notion, can you tell us how your solution works? Help me better understand why development remains with sort of a tarnish to it. From what you’ve done to date with these analytics, do you really think that when we get in there that a different story emerges?

Burnell: We’re very optimistic about application development. I want to be very clear about that. We believe a lot of the perception of failure is because expectations were set incorrectly. If the expectation’s wrong or overly optimistic -- or even overly negative, to be honest with you -- the outcome is never going to meet the expectation. If you build a car, and your expectation was it’s going to have four wheels and it shows up with three, it’s very clear that it didn’t meet your expectation. With application software it’s not so easy to focus on one particular attribute of it.

One of the things we believe is that we’re going to start helping people to understand a level of effort and to set expectations in a more discrete manner. They’ll have a better idea that they’ve applied this level of effort and the outcomes that they have. They’ll have a better opportunity to view themselves successfully in the context of the outcome. If you start off on the wrong road, or if you start off too optimistically, you’re never going to get back on track.

This is a highly unstructured process involving knowledge workers. These are very bright, very talented, very creative people, and you’re trying to corral this energy and create something. It’s a unique management challenge, and requires unique approaches.

That’s why we’ve always approached it from the perspective that if we can be part of the process and not sit external to it like a process tool or a portfolio management tool -- if we can resonate with the process as it unfolds -- then we have a better opportunity for people to understand where they are relative to where they’re going. They have a better opportunity to set an expectation, based upon this effort to produce an outcome. So when they’re done, they can sit down and make a much better assessment of how they’ve done against their expectation. And, they can revise accordingly for the next time.

Gardner: Let’s help our readers out there understand what it is that we’re discussing, we’ve talked a little bit about the “why,” and what the problem set is and the issues. This is BriefingsDirect. I’m the analyst. You tell me, Greg, what is it that you bring to the table as a hosted service that gives insight into developer behavior?

Burnell: What we’re trying to do is say that software development can be measured. The singular or the collective effort of a development team can have associated with it: measurement, metrics, and analytics that the team, the individual, the project, the executive can rely upon. What we’ve done, as we talked about earlier, is create a technology that resides with the development technologies that you use: IDEs, SCM systems, defect trackers, test tools, requirement management systems. These are tools that you use inside of the development process.

As you utilize these tools it’s kind of like a hammer in construction. You have to use a hammer to build a house. In order to build software, you have to use technologies to do it. We sit there with these technologies -- plugins, addins, whatever kind of technology is most appropriate for the tool -- and as you take advantage of these tools, we simply collect data about the process, and then we give it back to you.

I like to say we don’t impart the process -- we reveal the process. We show you what you’ve done.

One of the neat things that’s important to know is that when a knowledge worker is most successful and there’s a concept called flow. I don’t think we’ll get to that in this call, but when knowledge workers are highly successful and highly focused on the task at hand, time becomes an irrelevant concept to them. They want to achieve the objective. One of the things we’re trying to do is to help people understand the relationship of time and effort to outcome, and in order to do that you have to be transparent behind the scenes, and you can’t take away from their efforts.

We plug in with these technologies -- Eclipse, Visual Studio, as I mentioned earlier -- and as the developers use these tools we collect data and we send it back to our host, the centralized server. We run it through an exciting analytics engine that takes this raw data and translates it into a picture of what has occurred in the software development process. Through a Web 2.0-style Web interface, utilizing Ajax and the like, we serve back to all the stake-holders in the process a very rich analytics engine for them to get a multidimensional view of the application-development process.

Gardner: Is this qualitative or quantitative? Determinations can be made, but are they specific, team-wide, or project-wide? Are they specific to a day? Can you give me some sense of what the data shows?

Burnell: There’s a lot of quantitative measurement. The base unit of measurement is called “active time.” It’s our unique offering. “Active time” is the amount of time you’re actively developing software.

We take that “active time,” that base unit of measurement, and we can roll it up. We know things like what your process is, your mix of edit time, view time, debug time, test time, and design time. We know that by your use of the tools, by the functions that you’re using inside the tool.

That’s very quantitative, but that’s also qualitative, because it reflects back on your process. We understand the tools and the technologies that you’re taking advantage of. We know the amount of time spent on tasks. At the end of the day, instead of just trying to sit down and say “I spent four hours on this and three hours on this,” we answer those questions for you.

It’s a rich set of both qualitative and quantitative metrics, and certainly the quantitative aspects of it are important. We measure the same way yesterday, today, and tomorrow. That’s important. We now have the opportunity, when change is imparted and we’ve changed either a process or a tool or a person, to sense that, because we understand what the dynamic is in the next day. We see it measured the same way and we can now start seeing the outcomes.

One of the things with software development is that we make a lot of changes as we go through the process. People come up with great ideas, read new books, and buy new tools, but no one has an idea whether they, in fact, improved the process or if they’ve had a beneficial effect from that. We think that we'll be able to help bring a lot of visibility to that.

Gardner: So, you’re leveling the playing field, not only within an individual activity, but perhaps across a multitude of activities. You can draw industry-wide inferences and perhaps start saying which tools are shelfware, what aspects of development are hang-ups that need to be focused on. That might bring knowledge to other vendors as to where they should devote their energy. This could be not only an individual, project, or even a company-wide benefit, but also an industry-wide benefit.

Burnell: That’s the other reason that we do it as a software-as-a-service solution, Dana. We gain the approval of our users to participate in our community of data. We do have that kind of visibility. We understand what the utilization of Eclipse is on a global basis for an installed community. We become a representative. Instead of looking at surveys and samples and calling a CIO, I know in my organization that 68.8 percent of the time spent in the last year was spent inside of Eclipse. I know that answer. I also know what my breakdown of my other tooling is.

You can look at it on a global basis. Application development is global, so you not only can look inside of your organization, but you should look outside of your organization and see what’s happening in the industry at large that you should be aware of and attuned to in your own track. Are you taking advantage of the things that are out there right now? We’re going to provide a lot of visibility to that.

This is all about decision making, and having the right things at your fingertips to make the critical decisions to ensure a successful outcome. The only thing anybody wants is software that’s on time, on budget, and meets the expectation. We’re going after those same challenges. That’s all we want to accomplish too. We’re going to help people get there faster and help people get there with a higher degree of confidence that they’re getting done what they need to get done.

Gardner: Those industry-wide metrics are extremely interesting to me. I think we’ve been fairly underserved, even in the analyst business, by the quality of the data we can get. We do surveys. We don’t know exactly who’s taking them, and how honest they are about the responses. People who are the busiest, the most productive, will be the least likely to take the survey. So, to me it’s very exciting to be able to get that visibility.

Let’s move into where this is going to be most relevant. You’ve mentioned offshoring and global issues quite a bit. Talk about lack of transparency and visibility. When I send off a set of requirements across the globe, and things come back from a variety of sources, how could I start measuring so that I know if I’m doing the right thing as a business person? What will your solution bring to the table that will help address this lack of insight into what’s going on in these offshore projects?

Burnell: My co-founder, Todd Olson, really is the architect of the solution we’re discussing. He said early on that the one commonality across application development worldwide is the desktop. People are working with the same tools. People are using Eclipse worldwide. They’re using Visual Studio worldwide.

So, we can basically isolate ourselves from all the cultural differences, the management differences, and the process differences. We can just eliminate those, and we can measure things the same way. What that means is that whether a person is managing a distributed team, an outsource situation or a captive situation doesn’t matter. They know that they’re looking at data that’s highly comparable between the locations. It speaks the same language. That’s the great thing about it.

The challenge with globalization is that you physically can’t be there all the time. So, you need to find ways to measure things. It goes well beyond just collaboration tools, meetings and phone calls. You’ve got to have some measurement that you rely upon. That gives you confidence that people are applying effort to the places that you want effort applied to, that the teams are conducting themselves in the manner that you wish them to. That’s what we do.

We basically give a high-degree of confidence that you can look at this information that we provide and ensure that your teams are all working together in the manner in which they’re supposed to. You can ensure that people are buying into the goals and the objectives, and they’re putting their time against it. That’s a key thing. People outsource for cost. I don’t want to take a radical view of outsourcing, but they do it for cost reasons. It’s not because they just have this desire to go global. They go out and try to find out lower-cost-basis resource. That does not diminish the talent and the genius of the educational systems and the people on a worldwide basis. They’re unbelievable.

We found that out after the fact, though. We went there for the cost and then we found out, hey, they’re very good. They’re extraordinary, actually. So now we said, let’s manage this as an opportunity. And, you have to have a measurement approach that is consistent and independent of all the other variables that exist in a global environment.

Gardner: So, we’ve got some value here: small teams, big teams, industry wide. We can address some of the issues about globalization. How about verticals? Are there specific sweet spots in the development world -- the type of applications, the type of environment -- perhaps it’s ones where the time-to-value or time-to-market is a really important element. Where’s the sweet spot for this stuff first?

Burnell: Anybody who makes their living from software should be worried about this. People who make their living on time to market, quality, and cost concerns should be all over this today. We’re going to tell them that you can now measure it, but beyond that, we really think that this is an opportunity for anyone to get a better grip on the overall process.

Organizations of all sizes are outsourcing now. Most boards of directors are looking at their companies and asking, “What’s your outsourcing strategy?” They just assume that there is some intrinsic benefit to outsourcing. We’re not so sure that’s true, but that’s not really the point. If your life depends upon your execution of your software process, this should be something that you would look at very closely. Beyond that, if you still have a need to manage a big team, manage people all over the world, manage across borders and times zones and language, this is also another thing you should be concerned about.

The interesting thing is, Dana, I’ve got eight or 10 people sitting in my development shop right now. We’re a relatively modest size organization from the development team point of view. I will tell you that as CEO of this company I have better visibility into the execution of my IT development resources than any other CEO on the planet. I know what we’re doing, and that’s an empowering position for any executive to be in from a decision-making perspective.

Gardner: You’ve got a development dashboard as a business person.

Burnell: I’ve got more than a dashboard -- I can see the alignment of the objectives that we set in our managements meetings to the execution of my development activities on a weekly and monthly basis. I know where my technologies are. I know where I need to go buy technologies.

If you’d called me up a year ago and said “Greg, you’re going to be spending 48 percent of your time building JavaScript,” I’d have told you that you were crazy. But that’s what we do here. I didn’t realize, and we didn’t realize going into this, the amount of investment we would have in the Web-side technologies like Java Scripting and Ajax.

We need tooling to support that. And I have to tell you, there’s not a lot out there. Now, we’re now talking to other people to build the tools. Instead of being reactive, we’re basically being proactive, looking for technologies that line up with our development requirements. And that’s a new place to be.

Gardner: Because you’re a hosted service, explain how the cost works. Do you charge per developer seat, per organization, per line of code? What’s the way that you can charge for this?

Burnell: It’s on a subscription basis, and we do it per subscriber; individual subscribers. Certainly, we understand enterprise pricing and the dynamics of organizations. As people come to us, we’re willing to sit down and look at the organization, because we believe this is the technology that needs to be implemented for all people in the process. It is a per-user basis at this time, but in any of our discussions with large corporate clients, we’re looking at the organization, the needs of the organization, and we can put a annual pricing in place for those organizations, if needed.

Gardner: Now let’s extrapolate a little bit and look into the future. If you can help define what that "flow" is -- when people are at their most productive and when they’ve used the tools that have been made available to them; when they’ve used their own creative juices -- you've found the efficiencies that they need to work smart, not necessarily work hard.

If you can quantify and qualify that for development, it seems to me that you could perhaps define that flow and apply your metrics and ability to gather information to make inferences about other creative activities. Help me understand that. Is this pie in the sky or do you think there’s something there?

Burnell: There is the old adage that you know who your good developers are. When we start showing you what the good developers do, what the most efficient developers do, you can now start looking at them and start prototyping activities and events and technologies against people that produce outcomes, and try to understand them a lot better.

To answer the second part of your question, this really is a platform for collection of data related to the knowledge-work process that involves a desktop technology. It even goes outside of application development. If you are managing a process, and the worker is using technology to do it, you can call it a human-based process.

I’m not really crazy about that term, because last time I checked, there aren’t any animals involved in the process. Nonetheless, a human being is sitting there interacting with a desktop technology to produce an outcome. So, we can extend it beyond software development, and we can look at other processes. What we’re talking about is trying to understand the dynamics of human interaction to a desktop technology to produce an outcome.

You mentioned manufacturing earlier. If Henry Ford didn’t figure out the assembly line, we’d still be building one-off cars at a $100,000 apiece. We have the same opportunity to really dig into and understand these human processes related to technology, and start evolving on a much faster basis our productivity and our output.

Gardner: That’s very interesting, Greg. Thanks for sharing all that. We’ve been talking about development productivity and a new approach from 6th Sense Analytics to gather data and provide analytics into how developers are the most productive, and how people can help them further that trend toward greater productivity.

Joining us for the call has been Greg Burnell, he's the chairman, co-founder and CEO of 6th Sense Analytics. And 6th Sense has also been the sponsor of this podcast. We appreciate their support. Thanks for joining, Greg.

Burnell: Dana, it’s my pleasure, and it’s great to talk you.

Listen to the podcast here.

Podcast sponsor: 6th Sense Analytics.

Transcript of Dana Gardner’s BriefingsDirect podcast on developer productivity metrics. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.