Transcript of BriefingsDirect Podcast on Borland Gauntlet and Enabling Development Process Management

Edited transcript of BriefingsDirect[TM] podcast with Dana Gardner, recorded Dec. 19, 2006.

Podcast sponsor: Borland Software.

Listen to the podcast here.

Dana Gardner: Hi, this is Dana Gardner, and you’re listening to BriefingsDirect. Today, a sponsored podcast on developer productivity, on looking at an holistic approach to development that takes into consideration policy-based approaches -- managing process and people -- and not just the technology. We'll also talk about seeking a feedback approach, where testing is continuous -- and visibility into performance becomes an imperative.

To help us sift through some of these issues today we’re joined by Rob Cheng. He’s a Director of Developer Solutions at Borland Software. Thanks for joining us, Rob.

Rob Cheng: Thanks for having me, Dana.

Gardner: Rob, we want to get into a little bit of a historical context here on developer productivity. Why don’t we start out with understanding some of the basic principles around Agile development? I think that’s a good place to start, as we get into some of the approaches that Borland’s taking, and some of the newer benefits that are available in the field now. Give us a little primer and a little history, if you will, on Agile?

Cheng: Certainly. Agile essentially arose as a reaction against what was perceived as more heavyweight methods like Rational Unified Process (RUP). They were considered to be somewhat inflexible and inefficient, and imposed too high of an overhead on the development organization. The term "Agile" actually covers quite a few different specific methodologies, including extreme programming and Scrum, but they all share some basic principles, things like customer focus and rapid iterations on working software.

Gardner: I suppose if you do have rapidity, and you’re remaining lightweight, you also might lose visibility. So, managing this becomes a bit more important.

Cheng: Absolutely. I don’t know if it’s true that you lose visibility, but you have to be more cognizant of what you need to do to maintain visibility. A rapidly iterating process gives you more opportunities to measure things. You could conceivably end up with better higher fidelity measurements of visibility. That’s really where we’re going from the Borland Gauntlet perspective.

Gardner: And obviously it gives us an opportunity to gather more data at more points in the process.

Cheng: That’s right. In terms of data or metrics, rapid iterations really give you more opportunity to sanity check what you’re building, either from a code perspective or, ideally, with the customer/end user of the product.

Gardner: When one gains more data and input, then one can also make comparisons, and have, in a sense, a feedback opportunity.

Cheng: That’s exactly right. One thing that isn’t talked about enough in this area is the context. You get around to looking at projects in relationship to each other in context and also in relationship to time, looking at how things have changed from previous versions or over the last few months of the development process.

Gardner: I suppose the emphasis, until fairly recently, has been on managing the code, managing the development itself, in bits and bytes, checking in and checking out, and so forth, but with less attention paid to the overall process of how to take a project from beginning to end with a lifecycle approach.

Cheng: I think that’s actually an important point around Agile methodologies. If you look at the Agile Manifesto, or any other documentation around it, it really does try to emphasize that there’s a customer focus in all of this. And that customer focus helps drive some of this end-to-end thinking that we’re talking about.

You can’t have customer focus when you’re ignoring requirements, or when you’re ignoring use cases. Your iteration is not going to be completely successful, unless, at the end of it, there is a validation.

Gardner: And that leads of course to the real Nirvana here, which is to align business goals and interest with the development process, and development in general.

Cheng: That’s right. When you talked about aligning to customer needs, ultimately what you’re doing when you’re aligning the customer needs is aligning with the business needs.

Gardner: Now, what's at stake in getting this right or wrong? How big a deal is this, when we look at different vertical industries? And when we look at competencies within organizations, as they compete with one another, and the impact of globalization, and the need for getting to market quickly. What are the stakes here?

Cheng: The stakes are ultimately the success or failure of your software initiative. When you look at reports like the [Standish Group’s] CHAOS Report, which specified some ridiculously high number -- over 60 percent of software projects failing -- that entails a huge cost to the business. There’s a huge cost to software failure, software project failure, and there’s also a significant cost within projects of having to deal a lot of late stage rework.

Getting this right during the process means that it will eliminate a lot of those costs upfront. We can find out early whether there are issues, and the scope and size of those issues. The second part isn’t addressed quite as often. It’s the high business risk involved in simply not knowing. This goes back to the visibility issue. If you’re heading up a business unit, and you ask your accountants for some financial analysis, and they tell you, “Come back in a few months and then maybe we’ll give educated guesses,” what would you do?

Gardner: I’d be concerned.

Cheng: You bet, and you’d be right to be concerned. That’s not how businesses are run, but this is how software development has been running for decades. The businesses that have been trying to manage these, development organizations, have taken it for granted that this is how development works. This is black art. And, it goes to the heart of problem that we at Borland are trying to help the customers solve.

We’re trying to help them make software delivery more of a managed business process that’s based on metrics and objective measurements, not just guess work and anecdotal evidence. When we’re talking about more agile or rapidly iterative process, this is where it really gets to the heart of it. You can contribute to more frequent and improved measurement, not just from code, but, as we were talking about earlier, from an end-to-end user acceptance perspective.

Gardner: So, I shouldn’t need to wait until the project’s done to know, how well it’s doing.

Cheng: That’s a recipe for failure -- waiting until the product’s done to know whether it’s successful. You could take any other example in the business industry and you would be laughed out of any conference room, if you suggested, “We’ll wait until we’ve delivered our product or come off the manufacturing line to decide whether it’s okay.”

Gardner: This makes good sense. It’s logical and it’s rational. We can compare this to other initiatives in the industry in business over the past 150 years, whether it was manufacturing or the Industrial Revolution. So, there seems to be a natural evolution. But, how do we actually put it into practice? How do we go from theory to execution? You’ve gone out and decided that this is an important and a good business to be in. And, you have done some acquisitions to bolster your approach to this. Tell us a little bit about how you got from theory to execution?

Cheng: The technology that we’re talking about today around Gauntlet arose from an acquisition of some technology from a start-up that was run by several of BEA’s WebLogic Workshop veterans. They saw the problems and the challenges within their previous roles in very high definition and contrast.

They felt this was a serious problem that really impacts every development organization on earth. Borland, at that time, was also looking at how we could help extend some of the best practices and process improvements across the lifecycle -- and earlier into development. You might have noticed that we’d also made recent acquisitions around the testing in Quality Assurance (QA) space.

Gardner: Right.

Cheng: One was with Segue Software. One of the challenges that vendors like Segue or Mercury have had is that, although they’ve been quite effective at doing some automation and improvement around the QA process, they had no visibility into what’s happening in development. So, we still had this problem that you’re testing more effectively, but you’re still doing it very late in the game. After it comes off the assembly line, you have very little impact, and you can make very few changes effectively and cost effectively at that time.

Gardner: When I was doing some research into Gauntlet, I learned that the word Gauntlet comes from the idea that you’re going to put their code, the process, the people, through this gauntlet, making sure that it passes all the tests along the way, which is a good visual idea. It reinforces what’s going on here. Can you give us a bit more detail? What do you mean by test-before-the-test-phase and continuous test?

Cheng: That definitely is the metaphor of Gauntlet. What we mean by testing continuously is essentially that we are marrying quality control with existing version-control processes. Specifically, what’s happening is, as developers are committing their changes into their SCM or version control system, Gauntlet works behind the scenes to test, analyze, and measure those changes.

We can also intercept problems. For example, a very common problem is that you make some change, and because you weren’t careful, you break the build. It causes everybody else not be able to work, because you did something, perhaps some trivial change to a configuration file, that stopped it from working for every one else. Well, Gauntlet can actually interpose itself into this process, do the automated build analysis, and find that either it doesn’t build or fails some important test. Gauntlet won’t allow it to progress. It actually won’t promote the code and integrate it into main lines.

Other developers won’t see that and won’t be impacted by that change. When we talk about continuous test, we’re really talking about an approach to software delivery that emphasizes that quality really should be approached, and should be considered in every role and every stage in the life cycle. It’s not just the responsibility of QA. Unless you’ve made that mental acceptance that this is QA’s job, and you’re already in that place where it’s after the fact. You’re doing all this work but it’s already too late to make effective changes.

Gardner: Of course, the earlier you intercept the problem, the less costly and time-consuming it is to fix it, and the sooner you can bring the rest of the team in and on it.

Cheng: Actually, that’s true for number of reasons. Testing frequently does give you this notion of early detection. If you’re testing constantly and continuously, that means the time between when you create a problem or defect and the time when you find out about it is relatively short. That has a couple of consequences. One is that, it doesn’t have time to grow. It’s a bit much to talk about these defects as viral, but they can impact other projects and other development efforts.

Software is classically complex and interdependent. The longer you take between introducing a problem and discovering it, the longer you have that problem, and the longer you have other components and technologies -- or other software modules -- that depend on that particular flawed implementation. Unrolling that becomes much more expensive when you find out about it much later.

Gardner: I’ve heard some folks say, “Gee! You want us to have quality, and you want us to test continually, but you also want it done fast. Now which is it going to be?” I think there’s been some mentality that testing slows things down, and you’re bringing this in however as an overlay. This isn’t something that people need to do terribly differently. It’s not a rip-and-replace approach. You’re letting people use their same software configuration management (SCM), the same tests, and you’re just providing an organizational matrix on that. Is that correct?

Cheng: Yes that’s right. In fact, we do lower the overhead to some extent. When we talk about testing often, those are manual tests. Those are tests that the developer or QA engineer has to remember to do. They have to do them manually, have to take time out of their implementation to go and run these. Build it locally, test it locally. Presumably you’re doing something with that output as well, and all of that is overhead.

Gauntlet is trying to help streamline this process by automating all this stuff. The building and the testing actually all happens on the server. So, once the developer checks it in, it’s fire and forget. They can go on to the next task and check with the dashboard to see what the result of that was later.

Gardner: So, a rundown Rob, if you will, about what Gauntlet is? How it’s priced? What’s its architecture? Give us just a quick elevator description of what it is we’re actually talking about here.

Cheng: Sure. Gauntlet, as we talked about earlier, is a continuous build-and-test automation solution. Essentially, it’s a server-based product, server software.

It’s charged by the seat. It does interact with the version control systems that work with StarTeam, and we are constantly looking at also increasing the range of version control systems or SCM systems that we support.

Gardner: What have the results been so far? Are you going to come out with a new version or major release in early Q1 of 2007?

Cheng: That’s right.

Gardner: What have been some of the results so far? How has the field reacted to this?

Cheng: Right. The release in early Q1 of 2007 is actually the first official production release. We acquired Gauntlet in February, and since then we have doing a number of internal customer field trials as well as make a public early access release available. The feedback has been quite good.

I’ve been surprised by how quickly people have been able to get up to speed in using it and seeing benefit. The thing that’s really interesting about is that there are organizations, customers, and evaluators that are at all different phases and all different parts of the spectrum, when it comes to their occurrence, build, and development processes.

We have some that barely have unit tests. It was a feat to get them to actually have a build script that they can kick off in an automated fashion. Gauntlet gives them a framework, the central dashboard if you will, to monitor their progress, because it is one of their initiatives to improve. Part of what you need to do to build improve is to measure. You only get what you measure.

Gauntlet not only does the measurement, but it also does the visualization of it. So you can see from day to day, week to week, and project to project how things are progressing. You can see how often you’re able to build successfully, how much testing coverage you have from different projects, and how that is changing overtime. Are you increasing the coverage of testing that you have for your the different classes or different projects?

Then, we have other customers who are way on the other end of the spectrum, where they’re already very rapidly iterative. They are very agile and they see Gauntlet as a way to push that even further. They take their best practices and multiply the impact of them by offloading a lot of manual effort, doing it more on an automated way, and to giving the entire organization much more real-time feedback and better visibility about what’s going on?

Gardner: So, from their perspective of using this as a tool, they can find utility in it across the spectrum of maturation, in terms of where they are in their development and where they hope to go? That’s always a good descriptor of a long-term value. You mentioned something about visualizing a dashboard. Are these things that are designed for the architects or the project manager level, or is it some thing you can talk to the business people and say, “Here’s where we are. No guess work. We’re on track?” Is it designed for business people to engage with this as well?

Cheng: The Gauntlet dashboard currently is designed for development managers, project managers, and individual practitioners: the developers, the QA engineers, QA management, of course. The consequence of Gauntlet is that those managers, those project leads, the project managers, have the objective information to tell the business people what the actual information is.

They can tell what the status of the project is, what the risk is involved, how close they are to being able to actually ship, and if they know what the real stability is. They know what the confidence level is. A really good way of characterizing is that you’re giving developmental IT management much more confidence in how and what they say to business management?

Gardner: They can report to these business side folks with authority, because they do actually know where things are?

Cheng: Exactly. It’s fact based. It’s not just based on educated guesses.

Gardner: That notion of hope as a business strategy doesn’t have to kick in here.

Cheng: Actually, it’s not business strategy; it’s a recipe for failure and disaster. There often is too much -- what was one of the term that [former Federal Reserve Chairman Alan] Greenspan used -- an over abundance of optimism?

Gardner: Exuberance, perhaps?

Cheng: Perhaps that was it, but the overabundance of exuberance is really something that characterizes a lot of development in organizations sometimes. Gauntlet is a nice way to run a sanity check on that.

Gardner: One of the things you mentioned a moment ago about this spectrum of different organizations being able to apply this well gets to the heart of an area that’s always fascinating to me. That’s how to manage behavior, cultural shifts, individuals and in larger groups, getting them to change the way they think and move beyond a certain pattern of accomplishment, even though they feel it has probably been the only way up to a certain point.

Can you tell us little bit about how Gauntlet might be brought to bear on this issue of managed business processes, and how to get people to change the behavior? Can we really think about this as a way of orchestrating teams and behaviors? Or is that biting off a bit too much to chew?

Cheng: Yes and no. There is definitely something required as a forcing function, and it allows the adoption of more and more agile development. There’s usually a big failure or big problem.

One of our beta customers, for example, who is very aggressively adopting this technology, had what they called a brand-damaging, embarrassing failure event on their external customer-facing Website. They felt as if this was a symptom of the process failure that they had, and that was what really initiated their need to drive a change in process and in culture.

So, often there’ll be issues that people know they need to change, but they’re kind of set in their ways. They need something to spur them to action. Gauntlet does help in a sense that once somebody -- it doesn’t even have to be the entire company, the entire organization -- gets it in their head that they need to try this or they want to experiment with the more iterative, more agile approach, Gauntlet really helps them climb that learning curve. They climb that ladder of progressing incrementally towards getting more effective in their processes.

It can be just simple things like tracking builds, making sure that you can automate your builds, and that you’re building successfully. Then, tracking your unit tests. Do you have enough unit tests? Are they succeeding? How much of the code base are your tests actually covering? A lot of those things you could figure out, but it would be a lot of extra overhead to manage that process manually.

Gardner: So, for those organizations that have decided that they need to move from a tactical ad-hoc approach to a more strategic processes-driven approach, this gives them a ramp and a tool to start moving swiftly in that direction.

Cheng: That’s right. When you talk about strategic, we talked earlier about the notion of software delivery as a managed business process. One of the things that characterize other business processes in organizations is that you can do real sophisticated analytics. You can do business intelligence. That’s one of things that Gauntlet is trying to bring to the table, because of being able to continuously test, measure, and collect all the data.

Gauntlet actually gives you this great wealth of information that you can then data mine. When you talk about strategy and process, you also have to think about how to optimize the process. How do organizations continuously improve processes, not just code or technology? How are they are doing the work itself? A lot of that comes down to being able to mine to do ad-hoc analytical queries against the data that’s been collected about how the organization is actually operating. That’s one of the things that Gauntlet really provides -- that data warehouse of activity and metrics around actual development actions.

Gardner: So, I’ll get more data, I‘ll be able to start analyzing it. And then I can start setting policies, and then be able to revisit those policies and, on an iterative basis, improve upon them. What can we tell people about this policy-based approach or policy enforcement up and down the process and up and down the development lifecycle?

Cheng: That’s a very good question. The issue of process is important. A lot of times, process decisions are made for very good reasons at a higher level, in the process office, in the CTO, or the CIO’s office. They’re made in conjunction and collaboration with the management of software development. It often doesn’t get down to the individual practitioners, or it’s hard to enforce, because it’s just something that you’re telling people you’re managing that they have to do. How do you discipline it, and how do you enforce it?

One thing that Gauntlet allows you to do is make policy decisions at the point of insertion, at the point of code entering the system. Earlier, I talked about how we can essentially gate the promotion of code, the integration of code changes into different controls, based on whether they’ll be all that successful or whether test passed? Well, this is also something like you use much more generically as a framework, so that you can embed policy into that point as well. We can make decisions about requirements around maintainability, the complexity of the code, and the readability of the code.

If we have tools, they’ll help us analyze security, vulnerabilities or even such things as license compliance. Are you introducing unapproved third party’s libraries as open source libraries and packages? All these types of issues have the same characteristic cost curve, which is the later you find out about problems, the much more expensive it is to remediate.

Gardner: It also sounds like you can capture knowledge and competencies and then extend them, reuse them in a sense. This leads us into my next set of questions. What’s coming down in the future around these sorts of benefits, as we think about things like managed services and services-oriented architecture, and continuous ecology development of services, where there really isn’t just a cut-off date? As people consume these services, you’re going to adjust them, and they’re going to be working under service-level agreements. Is the Gauntlet approach something that’s going to become necessary as we move into that new arena of constantly dynamic development?

Cheng: Absolutely. In fact, a couple of words that you said were really resonating with me. One of them is “competencies.” From the competency perspective, one of the things that really breaks the back of development organizations sometimes -- and I’ve heard this a number of times from development directors – is that there may be dependencies in your system that you’re not really aware of. You don’t really think about them until they’re broken.

One of those is the human element, where there may be one person in the entire organization who knows how to put the software together. You only have one person who knows how to build the software, and that person leaves the company. You’re kind of hosed. You may have weeks or months worth of catch-up to get that knowledge back, but it ought to be institutional knowledge.

Gardner: Right.

Cheng: That shouldn’t be somebody’s personal expertise. The company or the development organization is running on that knowledge. Gauntlet helps to capture that in a centralized way and in an automated way, and share it with everyone.

This is also true of the situations when you have maybe niche platforms that you are trying to support or niche environments. There’s only one guy doing it, but you feel like it’s okay. Then, later there’s a customer escalation, because they’re that one big customer that happens to use that platform. And, by the way, the institutional knowledge is now gone. Again, the idea that we could capture that in a system, which actually is doubling as a means of collaboration, is an important one.

The second part about being able to have more services-oriented approaches, more dynamic development processes, also speaks to the issue of globally distributed and outsourced teams. One of the things that Gauntlet can bring to the table there is measurement. A big problem with outsourcing is how do we know what the cost benefit is? I know what the cost is, but we don’t know what the benefit is.

Gardner: Right.

Cheng: We don’t know how effective it is? We have no way to measure productivity or quality of the deliverables. Gauntlet, with frequent objective fact-based measurement, can start giving you answers to those kinds of questions. On the other hand, we also have the ability to do that process enforcement we’re talking about or the policy enforcement. Once I made a determination about the relative effectiveness of different teams, I can put different policies on the different teams.

Gardner: That strikes me as essential, if you want to go from the perception of IT as a cost center to a value center, moving the yardstick up in terms of getting more discretionary spending and more innovative types of projects that are going to be brought to the table and funded.

Cheng: That’s absolutely true. When you talk about that perspective and also the SOA arguments, all of it is predicated on the ability of someone in the IT organization convincing those with the purse strings that they can deliver predictably and they can deliver to target in a managed way.

Gardner: Give us a little bit of a road map if you could, as to what the, purview of Gauntlet is. I think that it’s mostly focused on Enterprise Java development at this time. Where do you see its boundaries or how do you expect it to evolve?

Cheng: The great thing about Gauntlet is that from a framework perspective, it’s as agnostic -- particularly of your language and your development --as version control itself. You can store anything. It doesn’t even have to be source code. And, you can do documentation and other types of content management essentially through version control.

Gauntlet does provide that same framework to be able to invoke and then analyze the results of tests and other measurements that you do. But, you’re quite right that in the first release we are very focused on Enterprise Java. That’s where the ecosystem of the test and analytic capabilities is really focused, but we will be aggressively moving over the next year to support additional languages, environments and systems with .NET development, with C, C++ development.

These are all areas in which customers really want to be able to leverage Gauntlet as well. So, we’ll definitely be moving to that direction and we’ll also be expanding, as I mentioned earlier, the version control on SCM support that we’ll have.

Gardner: And this is already something that’s usually used in conjunction with open source and the Eclipse-based development.

Cheng: Absolutely. This acquisition was from a start up that was almost a 100 percent open-source stack, and it does have extremely effective networking with the sub version that was the original SCM version control system. It leverages NAnt and JUnit as tools for building and unit testing. We can configure it to work with a number of other open-source tools and analytical packages.

At a higher level, it’s also a way bringing some of the best practices of open-source development to commercial customers. One of the things about open source development is that it’s very transparent. I can go to the Website or the public repository and see what’s happening. I can see who did what, when, who broke what, who fixed what?

Earlier we talked about how we start nudging cultural change. One great way to do it is having the kind of transparency that promotes peer-to-peer accountability, as opposed to top-down management. You don’t want to be the one who’s embarrassed by breaking everything. And, on the flip side, you want to be the one who’s recognized as making the most contributions. That’s true in open source, and we think that it could be just as true in improving commercial proprietary software development.

Gardner: Great. Well, thanks. I think that wraps it up for our time allotment today. We’ve been discussing productivity in development and bringing more visibility and accountability in data, fact-based development, if you will.

To work through this discussion, we’ve been joined by Rob Cheng. He’s the Director of Developer Solutions at Borland Software. We’ve been discussing a product called Gauntlet that will be coming to market in full maturity in Q1 of 2007. Thank you for joining, Rob.

Cheng: Thank you very much, Dana.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions. You’ve been listening to a sponsored BriefingsDirect podcast. Thanks for listening.

Listen to the podcast here.

Podcast sponsor: Borland Software, Inc.

Transcript of Dana Gardner’s BriefingsDirect podcast on development process management. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.