Pulse Provides Novel Training and Tools Configuration Resource to Aid in Developer Education, Preparedness

Transcript of BriefingsDirect podcast on Java training and education with Genuitec Pulse for Java and Eclipse.

Listen to the podcast. Sponsor: Genuitec.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect.

Today, a sponsored podcast discussion about the complexity around Java training and education. The development toolset, the plug-ins, the community are all very fast-moving targets. It's difficult for trainers, educators -- not to mention the students and budding developers themselves -- to get a full grasp of what's expected of them, and then to find resources that are up-to-date and timely.

We're going to be discussing with some experts how better to organize Java training and education. We're going to look at the Eclipse profiles that can be organized and coordinated using Pulse, a Genuitec-organized tools configuration network function. We're going to learn about how organizations can organize their training, so that students can better anticipate what's expected of them in the real world.

To help us understand some of these issues and work toward some solutions, we're joined by Michael Cote, an analyst with RedMonk. Welcome to the show, Michael.

Michael Cote: Hello, everybody.

Gardner: We're also joined by Todd Williams, vice president of technology at Genuitec. Welcome to the show, Todd.

Todd Williams: Thanks very much, Dana.

Gardner: Lastly, we're joined by Ken Kousen, an independent technical trainer and president of Kousen IT, Inc. He's also an adjunct professor at Rensselaer Polytechnic Institute. Welcome, Ken.

Ken Kousen: Hi, Dana, glad to be here.

Gardner: As I mentioned, complexity and moving targets are part of the problem, but it also seems that there is a disconnect between book knowledge or training knowledge that one gathers about development and what happens in the real world, what goes on with code being checked in and checked out, and how teams are organized. It seems difficult for someone to anticipate what's really going happen.

Let's go to Ken first. Ken, what is the gap, from your perspective, between what students and budding developers get through training and in university settings, and then what's often expected of them in the real world?

Kousen: It's interesting. The gap between what's taught in academia and what's taught in the real world is very large, actually. The classes I teach tend to be in a master's level program, and I teach a couple of classes in developing enterprise applications that are specifically constructed to address this gap.

Academia will talk about abstractions of data structures, algorithms, and different techniques for doing things. Then, when people get into the real world, they have no idea what Spring, Hibernate, or any of the other issues really are.

It's also interesting that a lot of developments in this field tend to flow from the working professionals toward academia, rather than the other way around, which is what you would find in engineering, when I used to be in that area.

Gardner: Todd, when you're doing hiring or are doing development for business, and you're talking to your customers -- folks that use MyEclipse and your services for training and consulting -- are you seeing a worsening situation in terms of how to acquire qualified labor, or do people have a pretty good sense of where to go to find good Java developers?

Williams: Finding quality employees is always a challenge, and probably always will be. Part of what I see as being difficult, especially in the Java and Enterprise Java market, is the huge number of technologies that are being employed at different levels. Each company picks its own type of stack.

Ken mentioned Spring and Hibernate. There is also Java transaction API (JTA), Java server faces (JSF), and Struts, Web framework and persistence technologies, and application servers that are in use. Finding employees that fit with what you are trying to do today, with an eye toward being able to mature them into where you are going tomorrow, is probably going to always be the concern.

Gardner: Now, what's been going on with development, not just the function, but teams, the collaboration, agile types of activities, Scrum? It used to be that people could specialize, stay in one little niche, but now the "master of all trades" seems to be more in demand.

Let's go to Michael. Michael, is development fundamentally changing? When we think of developers, do we need to recast how we imagine them or conceive of them?

Cote: Yes. I think it's fair even to go to the extreme and say absolutely. You look at the employment patterns that most developers find themselves in, and they are not really working at some place three, five, ten, even twenty years. It's not realistic. So, specializing in some technology that essentially binds you to a job isn't really an effective way to make sure you can pay your bills for the rest of your life.

You have to be able to pick up quickly any given technology or any stack, whether it’s new or old. Every company has their own stack that they are developing. You also have to remember that there is plenty of old existing software out there that no one really talks about anymore. People need to maintain and take care of it.

So, whether you are learning a new technology or an old technology, the role of the developer now, much more so in the past, is to be more of a generalist who can quickly learn anything without support from their employer.

You're not going to get a lot of slack to learn things in a given time, paid training, and things like that. You're pretty much left on your own, or there are always cheaper alternatives to go to.

So the heat is really on developers to be Type A people who are always seeking out the best option.

Gardner: Alright. Well, now that we have scared anyone from ever wanting to be a developer, Ken, help us get a little bit closer to earth. What can students do, what can professors or instructors do, to help get more of this real-world perspective into what they do in these courses and in this preparation?

Kousen: It's interesting that while the various tools and technologies evolve, some of the basic principles always hold pretty fast. I've taught this class several times and I have to say that every time I've taught it, it's been very, very different, but the overall architectural issues are pretty constant.

Plus, what seems to follow in the industry are various trends, like an increased emphasis on testing, for example, the recent rise in dynamic languages, and things like that. The idea of continually trying to follow what's going on in the marketplace and seeing what's interesting seems to be very helpful.

I also emphasize to the students that a good source of information is to find some of the better open-source projects, and not necessarily join them, use them, or do anything with them, but follow what they do and see those projects as the communal efforts of some of the best developers in the world.

So, if they all say, "Oh yeah, we obviously have to have this source-control mechanism," then maybe that's an interesting thing that should be looked at, or this particular bug reporting tool, or whatever. I often emphasis that particular direction as well.

Gardner: How about that, Todd? Are these open-source communities, these chat rooms, these forums, the real, practical lab that the students and developer should be looking towards?

Williams: I think to a degree that it's certainly a practical lab that students have easy access to. Obviously, in open source, whether it’s something like the Eclipse Foundation, Apache, or what have you, they make a very explicit effort to communicate what they are doing through either bug reports, mail lists, and discussion groups. So, it's an easy way to get involved as just a monitor of what's going on. I think you could learn quite a bit from just seeing how the interactions play out.

That's not exactly the same type of environment they would see inside closed-wall corporate development, simply because the goals are different. Less emphasis is put on external communications and more emphasis is put on getting quality software out the door extremely quickly. But, there are a lot of very good techniques and communication patterns to be learned in the open-source communities.

Gardner: Now, when we go to community, that also means choice, which is a good thing. But, there is also a downside to choice. There are a lot of variables, many different things to look at. Tell us a little bit about the importance of profiling, and when you have got many new plug-ins to choose from, and you've got lots of commentary and social media being generated about what to use and what not to use.

Give us, Todd, if you could, some idea of the problem set that you saw in the marketplace a couple of years ago when you were thinking about Pulse.

Williams: Let me take a step back and quickly explain what Pulse is for those who aren't familiar with it. We built a general-purpose software provisioning system that right now we are targeting at the Eclipse market, specifically Eclipse developers.

For our initial release last November, we focused on providing a simple, intuitive way that you could install, update, and share custom configurations with Eclipse-based tools.

In Pulse 2, which is our current release, we have extended those capabilities to address what we like to call team-synchronization problems. That includes not only customized tool stacks, but also things like workspace project configurations and common preference settings.

Now you can have a team that stays effectively in lock step with both their tools and their workspaces and preferences.

What drove us to build something like this were a number of things. If you look at the Eclipse market, where we have been for a number of years, there are literally thousands of products and plug-ins for Eclipse. If you just want to go out and take a survey of them, or try some of them, it's a very daunting process for most people.

It starts out when you download Eclipse, go find some plug-ins, possibly looking into Eclipse Plug-in Central, find those update sites, type them in, download the plug-ins, and try them. This pattern repeats for quite some time, while the developer goes out and tries to figure out which of the plug-ins are good and which ones aren't.

With Pulse, we put these very popular, well-researched plug-ins into a catalog, so that you can configure these types of tool stacks with drag-and-drop. So, it's very easy to try new things. We also bring in some of the social aspects; pulling in the rankings and descriptions from other sources like Eclipse Plug-in Central and those types of things.

So, within Pulse, you have a very easy way to start out with some base technology stacks for certain kinds of development and you can easily augment them over time and then share them with others.

Gardner: Ken, help us understand how this can be used in the training and/or academic setting? What is it about Pulse that brings in more of the real world, and anticipates what choices developers are going to have once they get into the nitty-gritty of coding?

Kousen: Looking at academic and training settings, they are a little bit different. In a training setting, one of the real challenges the training classes face every time is getting the initial classroom set up correct. That is often very involved and complicated, because a lot of the tools involved are somewhat dependent on each other and dependent on environment variables and things like that.

So, trying to set up standard Pulse configurations and then being able to set up a classroom using those shared deployments is a very interesting opportunity. I haven't had the chance to do it yet, but I have definitely been talking to some training providers about giving that a shot.

I did try it in a classroom, and it's rather interesting, because one of the students that I had recently this year was coming from the Microsoft environment. I get a very common experience with Microsoft people, in that they are always overwhelmed by the fact, as Todd said, there are so many choices for everything. For Microsoft, there is always exactly one choice, and that choice costs $400.

I tried to tell them that here we have many, many choices, and the correct choice, or the most popular choice changes all the time. It can be very time consuming and overwhelming for them to try to decide which ones to use in which circumstances.

So, I set up a couple of configurations that I was able to share with the students. Once they were able to register and download them, they were able to get everything in a self-contained environment.

We found that pretty helpful, although I've got to say that this year the class size was sufficiently small, so that I don't know that we really got the same benefit we would get in a large classroom, where there would be many, many setup issues to deal with.

Gardner: So, almost mimicking a collaboration activity in a development setting, but in the classroom.

Kousen: Exactly.

Gardner: Are there any particular things that you learned from this exercise that those who might be evaluating and thinking about using Pulse could benefit from?

Kousen: It was pretty straightforward for everybody to use. We had to make sure that people using it had fast download speeds, but that had nothing to do with Pulse. That had to do with the size of Eclipse.

Of course, whenever you get students downloading configurations, they have this inevitable urge to start experimenting, trying to add in plug-ins, and replacing things. I did have one case where the configuration got pretty corrupted, not due to anything that they did in Pulse, but because of plug-ins they added externally. We just basically scrapped that one and started over and it came out very nicely. So, that was very helpful in that case.

Gardner: Michael, as you are listening to this, is there anything that jumps out at you in terms of understanding of Eclipse and its popularity, and then dealing with complexity that you could share?

Cote: I like the comparison of the Eclipse development world, versus visual studio, versus getting the one thing, because it is very accurate. That's sort of the ethos of Java -- maximum "choosability," if you will. It's one of these things in development that takes a long time to accept, but having lots of options is often more expensive and burdensome than having fewer options. Now that said, you want to make sure that you have good fewer options.

In every development team I have been involved with in my previous lives, as it were, anytime someone new comes onto the team, it’s always an extremely difficult issue just to get their tool chain setup correctly.

Having something wrong in the tool chain, the shared tools that the whole team uses, can really be quite disruptive. That's because the way that you assume your team members are going about solving problems is slightly wrong, and so they may not have the fully optimized way that your project is based around.

I guess you could call that the commercial application of that tediousness of setting up the configuration in more of an educational or a training environment. It's difficult to just sort of give someone a print out and say, go setup your stuff like this, because you are always missing little bits, and there is a lot of nuance in how things are exactly setup in the tool chains.

Gardner: Back to you, Todd at Genuitec. Have there been any surprises since you brought Pulse to market in how it’s being used? Are there unanticipated consequences that you would like to share -- the good ones anyway?

Williams: It's been interesting. We have seen a good number of people using Pulse, the way we anticipated it, sharing their tool stacks, and publishing them for their teams.

There seems to be a lot of people that use it privately. They don't share it with anyone, but they use it to manage multiple development profiles. So they might do C++ development one day and Java development the next, or what have you, and they like to keep custom tool stacks just for those things.

Even though they are kind of an island, and we made Pulse to share amongst teams, they find a lot of value in it, just to keep everything tidy.

Cote: If I can add to that, I personally haven't seen people using Pulse like this, because I haven't stuck my head in a developer shop when Pulse has been around. We would typically have a problem where -- across different versions of the project you are working on -- you would have your ID or your tools set up differently.

So, if you wanted to very quickly switch between those different versions, for example, to support or do some debugging in an old version, if there was some support issue, that switching cost between the two setups is a big part of going to fix an older version of something.

Nowadays, you have a lot of virtualization, so you can make this step a little easier, but you end up doing absurd things, like just having machines dedicated to specific versions of the software that you are working on.

If you can more easily switch between the profiles and the configurations that you have, then you can hopefully make it easier and less tedious to support these older products that you tend to have a lot of requests to support.

Gardner: Ken, did you see some advice that you might offer to those, either in academia or in the training field, things that they might want to consider as they are evaluating such things as Pulse?

Kousen: I agree with what the others were saying about the idea of setting up a series of alternative profiles that match the environment you are going to be working in.

I realized, as Michael and Todd were saying that, that I actually do that myself. I have a J2EE profile or Java EE profile, and I also have a regular Java profile, when I am working on different things, because there are certain shortcuts that won't conflict with anything in Java EE, if I use it in Java.

Eventually, I hope when you wind up adding Grails support or Groovy and Grails support to Pulse, it will probably have a configuration environment for that as well. The idea of having a variety of profiles that could each be used in its given time is very helpful.

I know that in a training environment we will definitely try to do that. We will be setting up alternative profiles that can be shared in a particular training class.

Academically, I like to leave things a bit more free form, although I agree that the initial setup is very helpful, because if the students don't have any feel for the environment at all, getting them over that initial hurdle is very, very helpful. After that, letting them experiment is always very, very useful. So that's good.

Gardner: Todd, Ken mentioned support for Ruby, dynamic languages, Groovy. Can you tip your hand a little bit and let us know what you've got in mind in that regard?

Williams: Actually, all of those things are in the Pulse catalog right now. Sometimes they are hard to find, because it's kind of big, but we added search to it to help you run them down. But, there are actually multiple Ruby solutions; I know Groovy is in there.

If a particular solution that you like isn't in there though, it's relatively straightforward to add it, not to the catalog, but you can still add it very, very easily to any of your profiles, either locally or shared.

So, the catalog is like a really good starting point that we try to keep up to date with what our users ask us to put into it. On the other hand, if it contains everything in the world, it gets a bit unwieldy as well.

Kousen: Dana, can I comment on that? I did speak very quickly on that issue. There is a Groovy plug-in in there. I was actually very pleased to see that, because I was concerned.

I've been using the Groovy plug-in for a while, and I wasn't sure whether that was going to be in the catalog at all. It did take me a while to find it, because it was filed under an area that I wasn't expecting, but once I put it in the search box, then it showed up immediately.

The only thing about Grails is that there isn't really a dedicated Grails plug-in yet, and the Groovy plug-in is really moving towards something like that. So, when that becomes available, I'm sure it will be incorporated into Pulse.

By the way, another issue that is very useful is that when I am browsing inside Pulse, just looking around to see what sort of components have been added, it's interesting to see what turns out to be popular; things that I hadn't really anticipated.

For example, I have been using Subclipse for the Subversion plug-in for a couple of years now. In browsing into the Version Control category I see that there are various other Subversion plug-ins as well and also others coming down the line. So that was another capability that I didn't anticipate and found rather interesting.

Gardner: Todd, looking forward a little bit, it seems that this profile information, while useful in a tactical sense, might actually have some strategic value too.

I'm thinking about the metadata that might be available through a profile and a definition of what a developer wants to do from an activity or behavioral or a pattern base. Then, applying that to when they do a search; perhaps refining the search based on their profile at that time, or perhaps using the profile in regard to when they do testing, bills, other aspects of lifecycle management for development.

Have you taken this a step further, where we could take these profiles and use them in a more strategic fashion, or is that something you are looking at?

Williams: That's a great question, Dana. Of course, we have a very large product plan for Pulse. We've only had it out since November, but you're right. We do have a lot of profile information, so if we chose to mine that data, we could find some correlations between the tools that people use, like some of the buying websites do.

People who buy this product also like this one, and we could make ad hoc recommendations, for example. It seems like most people that use Subversion also use Ruby or something, and you just point them to new things in the catalog. It's kind of a low-level way to add some value. So there are certainly some things under consideration.

Gardner: Michael, what do you think of that, taking more profile information; metadata about behaviors, uses, pattern of work, and then applying that to some of the other larger lifecycle activities in development?

Cote: Things like that work out really well, when you have the proliferation of choice that we were talking about earlier, where the systems can always be gained and everything.

This thing is a small enough subset that it doesn't happen, but just seeing sheer quantity-wise and rating-wise what people are using, helps you evaluate. I am probably making this figure up, but if there are 10 different unified modeling language (UML) plug-ins for Eclipse, then you need to somehow narrow down to the ones that are going to work out well for you.

The mixture of the fast and best way to get to that is really just to see which one is being used the most, because chances are, if people are still using it actively, its going to be a popular one. People are pretty fast to dump plug-ins that don't work well for them.

There is a place to capture the metadata or the usage data that's going around with things. That's the kind of thing that usually developers only get a chance to figure out when they are face to face with someone at a conference or other sort of events that don't happen as frequently as you might want to, to simply figure out which plug-in you might want to use.

Gardner: Any time you can take personalization information and automate that or refine searches and activities is certainly a productivity improvement, and Pulse really strikes me as setting up the opportunity to do that.

Cote: Absolutely.

Gardner: Alright. Let's stat wrapping up a little bit. Ken, any last thoughts as a technical trainer about where you would like to see this sort of capability go?

Kousen: I'm not exactly sure where I will be able to take advantage of it. Let me rephrase that. I think the current Pulse configuration is already very useful, and I'm not sure what else I need in order to start trying to incorporate it into an environment.

The only other issue that I wind up having in a training environment is setting things like environment variables onto the operating system. If there is some way we can get that into Eclipse for example, or rather into Pulse, rather than having to do it on the operating system itself -- maybe through the tools or whatever -- then that would be helpful. But I don't know. Right now, I think the situation is pretty good. I can't think of anything else concrete that I would want to add right there.

Gardner: Okay. Todd, thoughts about what educators and trainers should be considering as they look at something like Pulse, and how to exploit it and leverage it.

Williams: One thing that came to my mind, from a student's perspective, is the integrated development environments (IDEs) that are available right now; even the various configurations of Eclipse, are really made for professionals. When you take something like MyEclipse, there is just so much in it.

We need the ability to actually strip down the IDE to only what is needed for a particular exercise. For example, you could set up a profile for the first exercise of the class with just a limited set of tools that a new student would need to get their hands on. It limits the confusion factor. When you do the next exercise, you could easily update the profile; add a few additional tools to it.

So, you have kind of a selected discovery of additional tools and capabilities that coincide with the level of expertise the students are developing, as they are going up the learning curve in a particular course. I was just wondering. Is that the kind of thing that now we have enabled through having a technology like Pulse, that makes delivery of that straightforward, versus what had to be done before.

Gardner: Just for those interested in perhaps getting started, Pulse uses its network. How do people access that, how do they find it, how do you get started?

Williams: Sure. The Pulse website is www.poweredbypulse.com. There is a little 5 MB installer that you download and start running. If anyone is out in academia, and they want to use Pulse in a setting for a course, please fill out the contact page on the Website. Let us know, and we will be glad to help you with that. We really want to see usage in academia grow. We think it’s very useful. It's a free service, so please let us know, and we will be glad to help.

Gardner: Terrific. I want to thank our panelists for helping us dig a little bit into training issues, and some of the solutions that are welling up in the market to address them. We have been talking with Michael Cote, he is an analyst at RedMonk. Thank you Michael.

Cote: Absolutely.

Gardner: Todd Williams, vice president of technology, Genuitec. Appreciate your input, Todd.

Williams: Thanks, Dana, I have enjoyed it.

Gardner: Ken Kousen, independent technical trainer, president of Kousen IT, Inc., and adjunct professor at Rensselaer. Appreciate your experience and input, Ken.

Kousen: Oh, you are welcome, no problem.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions. You have been listening to a sponsored BriefingsDirect podcast. Thanks for listening, and come back next time.

Listen to the podcast. Sponsor: Genuitec.

Transcript of BriefingsDirect podcast on Java training and education with Genuitec Pulse for Java and Eclipse. Copyright Interarbor Solutions, LLC, 2005-2008. All rights reserved.

Borland's Own ‘Journey' to Agile Development Forms Real-World Foundation for New Software Delivery Management Solutions

Transcript of BriefingsDirect podcast on Agile Development principles and practices with Borland Software.

Listen to the podcast. Sponsor: Borland Software.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today we present a sponsored podcast discussion about Agile software development.

We're going to be talking to a software executive from Borland Software about Borland's own Agile “journey.” They deployed Agile practices and enjoyed benefits from that, as well as gained many lessons learned, as they built out their latest application lifecycle management (ALM) products. [See product and solution rundowns.]

We're going to talk with Pete Morowski, the senior vice president of research and development (R&D) at Borland Software. Welcome to the show, Pete.

Peter Morowski: Thank you, Dana. It's good to be here.

Gardner: Before you get into Borland Software's journey, I want to get a level-set about Agile Development practices in general. Why is Agile development a good idea now? What is it about the atmosphere in the evolution of development that makes this timely?

Morowski: From the standpoint of software development, it's a realization that development is an empirical process, a process of discovery. Look at the late delivery cycles that traditional waterfall methodologies have brought upon us. Products have been delivered and end up on the shelf. The principles behind Agile right now allow teams to deliver on a much more frequent cycle and also to deliver more focused releases.

Gardner: There are also, I suppose, technical and business drivers: better quality, faster turnaround, more complexity, and, of course, distributed teams. What is it about the combination? Why is this important now in terms of some of these other technical business and even economic imperatives?

Morowski: With the advent of Web applications, businesses really expect a quicker turnaround time. In addition, when you look at cost structures, the time spent on features not used and other things are critical business inhibitors at this point.

Gardner: Let's help out some folks out who might not be that familiar with Agile and its associated process called Scrum. Tell us a little bit from an elevator-pitch perspective. What is Agile and what is Scrum?

Morowski: Agile really is a set of principles, and these principles are based on things like self-directed teams, using working code as a measure of progress, and also looking at software development in terms of iteration. What we mean by that is that when you look at traditional software development, we talked about things like design, code, and testing as actual phases in a development lifecycle. Within Agile, in an iteration, these are just activities that occur in each iteration.

Now, when you talk about Scrum, that is more of a process and a methodology. This is actually taking those Agile principles and then being more prescriptive on how to apply them to a software-development cycle.

In the case of Scrum, it's based upon a concept called a sprint, which is a two-to-four week iteration that the team plans for and then executes. In that two-to-four weeks, whatever they get done is considered completed during that sprint, and what work hadn't been completed goes into what they call "product backlog" for prioritization on what is done in the next sprint. You chain these several iterations together for a release.

The beauty of this is that now you have a way to induce change on the borders of those iterations. So, one of the things that's really advantageous to Agile is its ability to adapt the changing requirements.

Gardner: When I try to explain Agile to people, some of them come away thinking that it's an oxymoron or is conflicted because they say, "Okay, your goal is to do things better and faster, but you are telling people use fewer rules, use less structure, and have your teams be self-selecting." People see a conflict here. Why isn't that a conflict?

Morowski: I think it's a misnomer that self-directed teams and that type of thing mean that we can do whatever we want. What it's really about is that teams begin to take ownership for delivering the product. What happens is that, by allowing these teams to become self-directed, they own the schedule for delivery.

What happens is that you see some things like traditional breakdowns of roles, where they are looking at what work needs to be finished in a sprint, versus "Well, I am a developer. I don't do testing," or "I am a doc writer, and I can't contribute on requirements," and those types of things. It really builds a team, which makes it a much more efficient use of resources and processes, and you end up with better results than you do in a traditional methodology.

Gardner: It almost sounds like we're using market forces, whereby entrepreneurs or small startups tend to be more energized and focused than teams within a larger, centralized organization. Is that a fair characterization?

Morowski: Yeah, I think it is very fair.

Gardner: And, given that we're looking for this empirical learn-as-you-go, do what's right for you, I suppose that also means that one size does not fit all. So, Agile would probably look very different from organization to organization.

Morowski: It could. One thing we chose to do, though, was to really to set a benchmark process. So, when Borland first started developing in Agile, we had multiple locations, and each site was, in essence, developing its own culture around Agile. What I found was that we were getting into discussions about whose Agile was more pure and things like that, and so I decided to develop a Borland Agile culture. [See case study on Borland and Agile.]

We broke that up on geographic bases, where we started with one site, had one "ScrumMaster" and we built what we call the reference process. As we've grown, and our projects are getting more complex, the fact that we evolve from site-to-site based on the same process and the same terminology has allowed us to now choose more complex agile techniques like Scrum of Scrums or work across organizations, and have a common vocabulary, and that kind of common way of working.

Gardner: It also sounds like you are taking the best of what a centralized approach offers and the best of what a decentralized approach offers, in terms of incentive; take charge, and local ownership, and then making them co-exist.

Morowski: That's correct.

Gardner: All right, let's get specifically into Borland's situation. What is it about the way that Borland has been developing software, which is of course a core competency for a large independent software vendor (ISV) like yourselves, and it has been for 15-plus years … How difficult was it for you to come into this established organization and shake things up?

Morowski: Initially, it wasn't an issue because, like most organizations, when we went through and looked at it, there were a couple of grassroots efforts underway. From an Agile perspective, one of the things we did was to begin to leverage that activity and the successes that it had to use as a benchmark with other teams. As we grew and moved into other organizations that were not necessarily grassroots efforts, there were some challenges.

Gardner: So, it might be quite possible that lot of organizations that do development have people who are Agile-minded and perhaps even followers of Agile doing this already. Perhaps they should look for those and start there.

Morowski: I would recommend that you start with your grassroots efforts, establish your benchmark process, and then begin to move out from there.

One thing we clearly did was, once that we saw the benefits of doing this, we had a lot of executive sponsorship around this. I made it one of the goals for the year to expand our use of Agile within the organization, so that teams knew it was safe to go ahead and begin to look at it. In addition, because we had a reference implementation of it, it also gave teams a starting point to begin their experimentation. We also paid for our teams to undergo training and those types of things. We created an environment that encouraged transformation.

Gardner: Let's learn a little bit more about you, Pete. Tell us a little bit about your background and how you came into development and then into Agile?

Morowski: I've been in this business a little over 25 years now. I started in the defense and aerospace industries and then moved into commercial ISVs later in my career. I've been an executive at Novell. I've also been a CTO at IBM Tivoli, and prior to Borland, was the vice president of software at Dell.

Gardner: You've taken on this Agile project at Borland, and you've written a paper on the “Borland Agile Journey.” I've had the pleasure of reading it. I think it's a really nice read and I commend for you it.

Morowski: Oh, thank you.

Gardner: Tell us about this particular product set [Borland Software Delivery Management information] that Borland is coming out with. It's a product set about helping people develop software. Is there a commonality between some of the lessons you learned and then what you may have actually visited in terms of requirements for your products? [See demo and see launch video.]

Morowski: Oh, absolutely. One of the interesting things about the products that we are delivering is that one of them is a product for managing Agile development, especially in distributed teams and managing the requirements. So, we had the advantage of actually using the tools as we were developing them.

Now, we were also very cautious because you can get myopic about that type of thing, where we also using Agile principles, and we involved our customers in the process, as well. So we were getting kind of the best of both worlds.

Gardner: What makes software development different? In reading your paper, I was thinking about how these principles about self-empowerment and working quickly and then setting these boundaries -- "Okay, we're going to just work and do this for three weeks and then will revisit any changes," -- that might be something it would apply to almost any creative activity where a team is involved.

Is Agile something you think applies to any creative activity, a complex team-based activity, or is there something about it that really is specific and germane to software development?

Morowski: If you look at Agile principles, conceptually, they do apply to a lot of things. Anything in which you are going into a period of discovery, one of the key things is knowing what your goal or mission is. In the case of software, that's a requirement, and what you want the product to be.

But in any kind of empirically based endeavor, this would be something that you could apply. Now, when you get down to the actual Scrum process itself, it's the terminology, the measures, the metrics, and all those types of things are really tailored for software development.

Gardner: When I read your paper, I also came away with some interesting observations. You say, there is a difference between how development is supposed to work and how it actually works. It's sounds like many companies are living in denial or a certain level of dysfunction that they are not necessarily facing.

Morowski: It's one of the issues with laying a manufacturing process over something that's inherently an empirical process. In the end, all software R&D organizations or IT shops responsible for applications are responsible to the business for delivering results. And, in doing so, we all try to measure those things.

What I have observed over my career was the fact that there really existed two worlds. There is what I call the "management plane," and this is a plane of milestones, of phase transitions and a very orderly process and progress through the software development lifecycle.

Underneath it though, in reality, is a world of chaos. It's a world of rework, a world of discovery, in which the engineers, testers and frontline managers live. We traditionally use Gantt as a measure that is task-based. It requires a translation from the implementation world to the management world to show indications of progress. Any time you do a translation, there can be a loss of information, and that's why today software is such an experienced-based endeavor.

Gardner: And it's often been perceived as sort of a dark art. People don't appreciate or understand how it's done, and that those who do it should say, "Hey, leave me alone, get away from me. I'll come back with the results in three months."

Morowski: Exactly.

Gardner: But that doesn't necessarily or hasn't historically been the best approach.

Morowski: Absolutely not.

Gardner: Also, at times, you see them downplay process and say that doing good hiring probably is the biggest issue here. What's the relationship between hiring and what people, not always affectionately, refer to as human resources? What's the relationship between HR and Agile?

Morowski: Well, first of all, just getting back to a little bit on hiring thing. Hiring is important, regardless of what methodology you use, and I tend to stress that. I do contend there are different kinds of personalities and skill sets you are going to be looking for when you are building Agile teams, and it's important to highlight those.

It's very important that whoever comes onboard in Agile team is collaborative in nature. In traditional software environments, there are two roles that traditionally you may struggle with, and you have to look at them closely. One is the manager. If a manager is a micromanager-type, that's not going to work in an Agile environment.

And, the other thing, interestingly enough, is the chief architect role. What's interesting about that is that you would think you would fit in Agile very easily, but in a lot of traditional software organizations, all decisions of a technical nature on a project go through the chief architect. In an Agile world, it's much more collaborative and everybody contributes. So for some personalities, this would be a difficult change for them.

Gardner: So there is that grassroots element, and you have to be open to it.

Morowski: Right.

Gardner: What is it about the structures here? Again, for folks who might not be that familiar with Agile, tell us a little bit about some of the hierarchy.

Morowski: There are really two key roles. There is the ScrumMaster and the ScrumMaster runs what they call the daily stand-up. This is basically a meeting, where everybody on the team gets together on a daily basis and they answer three questions. "What did I get accomplished yesterday?" "What am I going to do today?" And "What's blocking me?"

Everybody goes around the room. It's a 15- minute meeting. You solve any particular problems, but you log things. The role of ScrumMaster is to run that meeting and to remove blocks to the team, and it's a very key role.

The second major role within Scrum is really the product owner, and this is the individual that's responsible for prioritizing the requirements or what we call the product backlog -- what is what is going to be done during the sprint, which features are going to be completed. Those are the two primary roles, and then from there everybody is pretty much a team member.

Gardner: When you decided to bring this into play at Borland, a very large, distributed organization, you didn't try to bite off too much. You didn't say, "We are going to transform the entire company and organization." You did this on more of an iterative basis. It seems that most people, when they do Agile, will probably follow similar path. They'll take a project basis and then say, "Now we need to expand this and make it holistic."

Many organizations, however, across all kinds of different management activities, can stumble at that transition from the project, or the tactical, into the holistic, or general, across an organization. What did you learn in making this transition from small to large scale at Borland?

Morowski: A couple things. One is that, as we rolled it out, let's say starting by site-by-site, we grew from teams-to-teams. The ScrumMasters worked very collaboratively to help each other out, because, in the end, they were responsible for delivering at the end of those sprints. That was a very positive effect.

As we moved out to distributed teams, there were a number of challenges, things like the daily stand-up, or if I have people in Singapore that are supporting a particular sprint, say, from the system testing point, that made things difficult. But, what I found is the team was pretty creative in involving those individuals, whether they recorded sprints, whether they shifted time zones, and they did this all on their own.

That was the absolute positive, one of the things that surprised me. It was an interesting discovery.

As we started to be more broad with the interaction with the non-Agile parts of the organization, this was a little bit more of a challenge, and I learned a couple of things. In doing any kind outsourcing, if you try to match a traditional, contractual base -- the statement of work (SOW)-type outsourcer -- with an agile team, that's going to present problems. The outsourcer is expecting very detailed specifications as a statement of work and that's just not produced during an agile or sprint/Scrum type of development activity.

The other thing is internally, and what I would say at the end of the pipe and at the beginning of the pipe, working with marketing and our new product introduction processes and support and getting sales out. One of the things that we found is that we started to have a capacity to release more often, but the organization, as a whole, had to adjust now to: A) provide market requirements to us in a different manner, and B) we had to adjust our process at the end to be able to accept more rapid releases.

Gardner: So in order to get the most out of Agile, it sounds like, for those organizations where software development is core competencies, important to their successes as a company, or as a government organization, or a public not-for-profit, that the edges of Agile start to blend into other departments. The whole competency around their organization can perhaps borrow some of these principles from development and extend them into the entire lifecycle.

Morowski: Yes, we no longer look at it as strictly an R&D thing anymore, just because of that. And, it's interesting. You know you are making progress from a development team perspective, when you are starting to output more than the organization can accept.

Gardner: Interesting. So, adjustments along the way, and that's again a principle of the approach.

All right. In this age of Agile and your Agile journey, you came away with three basic observations about the benefits. One was around self-directing teams; second around being able to manage change well; and, third, about how to do the relationship with the customer, in this case the customer being the folks who are interested in getting the software. Tell us about these three benefits and what you have learned?

Morowski: Well, we touched on the self-directing teams, and the key to that is one of the most important things as an executive is that you really have to take the lead and let your teams go and develop -- let them truly own their projects. There will be mistakes along the way, but once they do, it's an extremely powerful concept.

One of the great things about agile is that it's a very open and very visible methodology. During daily stand-ups, I can attend any daily stand-up and sit there and listen to what's going on. I can't contribute in those meetings, because that's run by the ScrumMaster. But, one of the times I was attending the daily stand-up, I knew the teams had progressed a great deal.

When they were looking at their remaining work backlog that they had for that particular sprint, and there were a couple of tests that need to be run that there was nobody assigned to. One of the developers had time, looked at that, and picked it up.

Now, normally, that would never happen, because we behave in a silo fashion. "I am an engineer." "I am a tester." It's an "I am a …" type of thing. But, when you really have a self-directing team, the team owns that schedule and it's very interested in making sure that they meet their commitments.

Gardner: I suppose that also fosters willingness of people to move in and out of role, without just saying, "Well, that's not my job …", but taking more group responsibility, and even as an individual.

Morowski: Absolutely correct, and that to me has been one of the more powerful things that I have personally observed.

Gardner: Change management has often been something that drives developers crazy. They hate when people come in and start changing requirements when they are in the middle of doing code or test. On the other hand, things don't stay the same, and change is part of everything in life and business, perhaps more so today than ever. How do you reconcile those two?

Morowski: Well, I think the reality is that there is going to be change during these development cycles, and so the question is what's the best way to handle it? If you look in a traditional waterfall methodology, you march along phase transitions. Even if you have iteration in place, if you discover a design or coding defect late in the game, you have to go backwards to a different phase and start going into the design or fixing the code. Then, you repeat the process again, and you continue to move along your space transition line.

The thing that's interesting is that with Agile you have an orderly way of injecting change. In other words, as a sprint completes and you've demonstrated the code -- and you demonstrate it after that three-week iteration -- if something has changed and you need to change the prioritization, you have a way to inject that change along that boundary, and then let the team go forward. That's what I always like to say, "We're always going forward in Agile."

Gardner: And how do the teams adjust to that?

Morowski: It's part of the process. The changes go into the backlog. The product owner looks at them and then prioritizes it based upon the complexity of the work and the timing and so on and so forth, and just how important that is. If it's important enough, it will go into the next iteration. The teams are used to doing that, because you are not, in essence, disrupting at a random point. They have already finished what work they were working on, and now there is a cleaner opportunity to inject that change.

Gardner: So, boundaries allow for those who want change to get it done without having to wait for a particularly long period of time or until the project is done. But, for those involved in the project, they have these sections where it's not going to become chaotic and they are not going to lose track of their overall process, because of this injection of change.

Morowski: No, as a matter of fact, the process encourages it.

Gardner: How about this, what you call customer relationships? It sound to me as thought it's just being transparent.

Morowski: It is. It's a different approach, in the sense that you are actually bringing in the customer as what I would call a partner in the development. They participate in sprint reviews, and sprint reviews at the end of a sprint, where you show the working code, what you have completed and so. Those are done on an every-three-week basis, and we involve our customers.

They also take early drops of the code and provide input into the product backlog on requests that they want, and things like that. It's proven to be very beneficial for us. The one thing is that, when you choose these customers to participate, it's important for them to be Agile, as well, and understand that, and they need to approach this as a partnership not just an opportunity to get their particular features or requirements in.

Gardner: And, that must also help keep expectations in line, right?

Morowski: Absolutely. What I have found is that the customers we have involved want to get used to our cycles and our delivery rhythm. They are less adamant about getting every feature on a list in a particular release, because they know it's a relatively short time before the next one comes around.

Gardner: When we describe these customers, would that, in many organizations, include bringing the marketing people in, and the salespeople. Can they get involved so that this becomes something that will enter the market as an agile activity, rather than having Agile happening on the development side, and then falling back into a waterfall mentality when it comes to the go-to-market activities?

Morowski: Yes, we do, and the transparency that's there actually helps build confidence in the rest of the organization on what we are delivering, because they see it as we progress along. It's not something that mysteriously shows up on their doorstep.

Gardner: It certainly sounds great in theory, and it sounds like you've been able to accomplish quite a bit in practice, but what about metrics of success? How have you been able to say, "it works?" Has Borland cut their cost, their time to development? Do they have better products? All of the above? How do we know we are succeeding?

Morowski: I'd say it's combination of all the above. The first thing is that by putting these teams together, they are much smaller teams than in traditional organizations. So, if you look at it, my teams are almost 30 percent smaller on the Agile side than they are on the traditional side.

Gardner: And what's accounting for that change?

Morowski: I think one, is the ownerships of the teams, and two, the breakdown of very specific roles.

Gardner: Would I be going out on a limb in saying you have eliminated the middle management factor?

Morowski: There is absolutely that as well. The other thing is the fact that we're delivering working code and involving with customers. We are developing fewer superfluous features. When a product goes out the door, it generally has the most important features that were entailed for this release. So, it really helps the prioritization process.

Gardner: Not too many cooks in the kitchen?

Morowski: Exactly.

Gardner: Cool! Tell us a little bit about what surprised you the most about this Agile journey of Borland.

Morowski: I think the power of the daily stand-up. I mean, yes, we got a lot of benefits, and yes, we had a number of successes, we were able to transition code from locations and things like that, but I owe that a lot to the daily stand-up. The thing that surprised me is how powerful it is each morning when everybody gets around the table and actually goes through what they've done, basically saying, "Have I lived up to my commitments? What I am committing to the team today? And then is there anything blocking?"

Generally speaking, a lot of developers tend to be quiet and not the most social. What this did is that it just didn't allow the few people who were social to have input on what was going on. This daily stand-up had people, everybody on the team, contributing, and it really changed the relationships in the team. It was just a very, very powerful thing.

Gardner: It sounds like balance among personality types, but that balance directed toward the real activity that is developing code.

Morowski: Absolutely.

Gardner: Interesting! Well, congratulations. I enjoyed reading your paper, and this certainly sounds like the future of development, I know that's what many people in the business think. We've been talking about Agile development practices and principles and how Borland Software has been undertaking an Agile journey itself, in a development project around development process tools and application lifecycle management products.

Back to those products. Is there anything about the synergy between doing it this way and then presenting products into the field that you think will help other people engage with Agile benefits?

Morowski: Are you talking about the products themselves?

Gardner: Yes.

Morowski: The products themselves, absolutely. We have a product coming out called Team Analytics. The key to this is that, while we talked about self-directed teams, we still have responsibilities to reporting to the business and how we are progressing.

Team Analytics gives us a view into the process, gives us the ability to go ahead and look at how the team is progressing, and those types of things, what features have been included or dropped, without having to go into the team and request that information. So that's a very powerful thing.

Gardner: Right. So, it's one thing to agree that visibility and transparency are good, but it's another to actually accomplish it in terms of complexity in large teams and hierarchy.

Morowski: Absolutely. This allows us to move to what I call a "monitored" from a "reported" kind of methodology of metrics. What I mean by that is, typically, at the senior vice president or vice president level, you really get to look at the state of your products once a month, in the sense that you have operations reviews or some kind of review cycle where all your teams come in and then they report the progress of what's going on.

With Team Analytics, you are able to actually look at that on a daily basis and see if anything’s changed over time. That way, you know where you need to spend your time and that's why we call it monitored, at this point.

Gardner: Super! Well, thank you for sharing your insights. I think there is a lot to be taken away here and learned.

We have been talking with Pete Morowski, the senior vice president of research and development for Borland Software. We were looking at Agile principles in the context of Borland's Agile journey.

Thanks, Pete.

Morowski: Thank you, Dana.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions, and you’ve been listening to a sponsored BriefingsDirect podcast.

Thanks for joining us and come back next time.

Listen to the podcast. Sponsor: Borland Software.

Transcript of BriefingsDirect podcast on Agile development principles with Borland Software. Copyright Interarbor Solutions, LLC, 2005-2008. All rights reserved.

WSO2 Data Services Provide Catalyst for SOA and Set Stage for New Cloud-Based Data Models

Transcript of BriefingsDirect podcast on data services, SOA and cloud-based data hosting models.

Listen to the podcast. Sponsor: WSO2.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect.

Today, a sponsored podcast discussion about data services, how services-oriented architecture (SOA) is enabling an expansive reach for data, particularly enterprise data, how this will relate to the development of cloud infrastructures and services.

We’ll also examine what technologies and approaches organizations will need to federate and integrate these data sources and services and hosts, but without additional risk. That is to say, to free up data, give it more legs, but without losing control or increasing security risk.

We're also going to look at how open-source software relates to this, and how organizations are bridging the risk reduction and larger availability of data using open-source software.

To help us in this discussion, we are joined by distinguished a panel. First, Paul Fremantle, the chief technology officer at WSO2. Welcome, Paul.

Paul Fremantle: Hi, nice to be here.

Gardner: We are also joined by Brad Svee, the IT manager of development at Concur Technologies, a travel and expense management company in Redmond, Wash. Welcome to the show, Brad.

Brad Svee: Glad to be here.

Gardner: We are also joined by James Governor, a principal analyst and founder at RedMonk. Good to have you back, James.

James Governor: Thank you very much. Hello, everyone.

Gardner: Let's set this up by looking at the problem. I suppose the good news of looking into the past is that data has been secure and controlled. There are lots of relational databases with many, many bells and whistles applied over the years. There has also been a lot of development, middleware, and system's administration work to manage that data, keep it available, but also secure. That's the good news.

The bad news is that it's been limited in some respects by that firewall of personnel, technologies, and standards around it. I want to go to first to Paul Fremantle. Can you tell us a little bit about why the old model is not going to be sustainable in the world of services and mixed hosting environment?

Fremantle: It's a very interesting question. There are two key issues around the old model. The first is the just-in-time nature of data systems that are being bought today. Typically, customers are coming onto Websites and expecting to see the status of the world as it is right now.

They don't want to know what happened yesterday. They don't want to know what happened a week ago. They want to know exactly what happened right now. So it's very important that we move away from batch-oriented systems and file dumping and move to a real, live connected world, which is what people expect out of the Internet.

That live, connected world needs to be managed properly and it's very, very difficult to build that as a single monolithic system. So, it's really essential to move the ownership of that data to the people who really know it, create it, and own it, and that really plays to SOA. This is the model of keeping the data where it belongs and yet making it available to the rest of the world. That's the first point.

The second point is, of course, the danger inherent in getting it wrong. I have two stories, which I think will shed some interesting light on this. One is, I was working with a government organization and they were involved in a situation where every day one of the employees has to FTP a data file from a remote system and back load into their local system.

The employee went ill, and, of course, this didn't happen. They had a whole process to find out who had the password, who could do this and solve this problem. The employees had no one there to back load this data. As I was investigating, it turned out that data in the remote system, from the other organization, was actually coming from within their own organization.

There was another employee uploading the data from the main system to the remote system every day, and they had no clue about this. They didn't realize that this process had built up, where the data from organization A, was being sent to organization B, and then re-downloaded to organization A again, every single day, taking up two employee's time to do that.

Gardner: This is sort of the medieval approach to data transfer.

Fremantle: This is the medieval approach to data transfer. This was not back in 1963 that this is happening. This was actually happening in 2007.

Governor: Medieval or not, the simple fact is that there are vast amounts of exactly that kind of stuff going on out there. Another lovely story told by Martin Fowler talks about a customer -- I believe he was in the U.K. NHS, but I should be a little bit careful there. I should say it was a large organization, and they were freaking out. They said, "We've got to get off Java, because the printer driver is just no good."

He said, "What exactly are you trying to do? Let's have a chat about the situation." "We got to get off Java. We will just try and work it out." He looked at the work was that got involved. Basically, they were getting a document, printing it out, taking it across the room, and then typing it into another system on the other side of the room. He had to tell them, "Well, maybe there is another way of doing it that won't require printer drivers."

Gardner: One of the motivators, it seems, is if nothing dramatic requires you to change your patterns, then you stay with them. It's sort of inertia with people's behavior, and that includes IT. What we're seeing now is an impetus, or an acceleration and automation in services, because they have to, because there are outside organizations involved. A business process is not strictly internal, from one side of the room to the other, but could be across the globe and span many different companies. Does that sound correct, Paul?

Fremantle: Absolutely. I just want to give you a second example, which has been very well published in the U.K. where I live, but maybe it hasn't been so well known outside of U.K. The revenue and the customs in the U.K. had a significant problem recently, where they sent a CD containing 20 million records, including the birth dates, names, national insurance numbers, and bank account details of the 20 million people to another government department.

And, they lost it. They sent it again, and they lost it again. It would not be too far to say this had significant ramifications on the government and their ability to stay in government. The payoff of this was, they had policemen out searching rubbish dumps. They had to send a personal letter to each of the 20 million people. Banks had to update their security procedures.

The overall cost of this mistake, I imagine, must be in the millions of pounds. Now, the interesting question is, firstly, they didn't encrypt that data properly, but even if they had, there is a huge difference between encrypting an online system and encrypting a CD. If a hacker gets hold of the CD, he can spend as long as it takes to decrypt that system. If it takes him two years of computing power to do that, he can sit there for two years and break it.

If you have an encrypted online system and someone keeps trying to break it, the administrator sees log messages, knows that something is happening, and can deal with that. So it's not just the lack of encryption and the bulk dumping of data from one department to other, that's the problem. The model of sticking it on a CD hugely increases the dangers.

Governor: Well, people should be imprisoned for that, or at least lose the right to trade. Obviously, being government organizations, it's difficult to make that stick, but the U.K. government loves the use of phrase "fit for purpose." Quite frankly, there has been evidence that they are not fit for purpose.

Interestingly enough, one of the things about the importance of data and managing it more effectively, is thinking about data in a more rigorous way. I was going to talk on this call about "leaky abstractions." One of the problems with SOA is the notion that, "Oh, we can we can just take the system as it is and make it available as a service."

Actually, you do want to do some thinking and modeling about your data, your data structures, and how it can be accessible, and so on, because of this notion of leaky abstractions. You can push something in one place and something else is going to pop out in another by just taking a service as it is and making it online. You may not be doing the work required to use it more effectively.

I think that's the kind of thing that Paul is talking about there. What better example of the leaky abstraction is there than somebody sending a disk and not tracking where it goes? Again, the fact that there wasn't any cryptography used is really shocking, but frankly, this is business as usual.

Fremantle: In fact just to completely confirm what you are saying there, the government department that wanted this data did not want the bank account details, the national insurance numbers, or the ages. They didn't want all that data. What actually happened was the revenue and customs team were not sufficiently IT aware to be able to export just the data that was wanted, so they dumped everything onto the disk.

I think that exactly confirms what you are talking about the leaky abstraction. They just pushed out everything, because that was the simplest possible thing to do, when it wasn't exactly what's required, which is what should have been done.

Gardner: So, it does seem clear that the status quo is not sustainable. That there is inherent risk in the current system and that simply retrofitting existing data in turning it on as a service is not sufficient. Either you need to rationalize, think about the data, and generate the ability to slice it and dice it a little better, so that in the case of this disk of vast amounts of information, there was only a small portion of that that was actually required.

Let's look at this also through the lens of, "If we need to change, how do we best do best do that?" Let's look at an example of how someone who needs to view data in a different sense, in a more modern sense, how they are adjusting? Let's go to Brad at Concur. Your organization is involved with helping to improve the efficiency and productivity of travel and management inside of organizations.

Your data is going to come from a variety of areas that probably could be sensitive data in many organizations. Certainly, people are not happy about having their travel information easily available around the organization or certainly outside of it. And, of course there are government and tax implications, compliance, and implications as well. Can you give us a little bit of sense of what your data problem set is and if it's different from what we have heard on the "medieval" front? What sort of approaches you would like to take and have been taking?

Svee: First, I would like to clarify the distinct separation between our research and development team, which actually works on our product that we sell the clients, and my team, which works internally with our internal data.

I would like to draw a distinct clarification between those two. I am only able to speak to the internal data, but what we have found is exactly that that. Our data is trapped in these silos, where each department owns the data, and there is a manual paper process to request a report.

Requesting a customer report takes a long time, and what we have been able to do is try to expose that data through Web services using mashup type UI technology and data services to keep the data in the place that it belongs, without having a flat file flying between FTP servers, as you talked about, and start to show people data that they haven't seen before in an instant, consumable way.

Gardner: So, not even taking that further step of how this data might be used in an extended enterprise environment or across or departmental organization boundaries, just inside your organization, as you are trying to modernize and free up the data, you are looking at this through the lens of availability, real time, lower cost and clip, print, and touch from IT personnel. What sort of technologies and approaches have you been taking in order to try to achieve that?

Svee: For the last year or so, we have been pushing an SOA initiative and we have been evaluating the WSO2 product line, since, maybe November. We have been trying to free up our data, as well as rethink the way all our current systems are integrated. We are growing fairly rapidly and as we expand globally it is becoming more and more difficult to expose that data to the teams across the globe. So we have to jump in and rethink the complete architecture of our internal systems.

Gardner: What is it about the architecture that has a bearing on these flexibility and agility you are looking for, but that also protects your sense of reduced risk, security privacy access control?

Svee: Most of the data that we are dealing with is fairly sensitive, and therefore almost all of it has a need for at least per-user access basis, as well as, when we are transporting data, we will have to make sure that it's encrypted or at least digitally signed.

Gardner: Now, it seems to me that this data will need to be available through a browser-based portal or application to the end users, but that the data is also going to play a role with back office system, ledger, and different accounting activities, as this travel and expense content needs to be rectified across the company's books.

Svee: The browser becomes the ubiquitous consumption point for this data, and we are able to mash up the data, providing a view into several different systems. Before, that was not possible, and the additional piece of moving the file between financial systems, for example, we are able to not have to pull files, but actually use Web services to send only the data that has changed, as opposed to a complete dump of the data, which really decreases our network bandwidth usage.

Governor: There's even potentially a green argument in there. I mean, all of this batch is just kind of crazy and unnecessary. We see a lot of it. There is so much data duplicated everywhere. It seems like we, as an industry, are very good at just replicating and getting ridiculous redundancy, and not so good at synchronizing and really thinking about what data does need to be transported and working with that accordingly.

That sort of makes a lot of sense to me. It's very good to hear you are taking that approach. I think sometimes we miss-call things SOA, when in fact what you are doing is kind of "suck and play." You take this thing, suck old things out, and then work on the new thing, as opposed to actually thinking about the data structures you need to enable the data to be useful and fit you.

Gardner: Let's go to Paul. Now, here is an instance where the organization has, I think, its feet in both camps. In the old style, there is accounting, the ledgers, and data extension across application sets from a common repository, and how to batch that in such a way that the data is all on the same page, so to speak, across these applications in a time frame.

We also need to take this out through Web applications to individuals and also across applications that are Web services enabled. So, it sounds like what we have here is a situation where the data needs to do many different tricks, not just a couple of old basic tricks.

What is it that WSO2 has done recognizing this kind of need in the market and is able to satisfy this need?

Fremantle: What we have built is what we call WSO2 Data Services, which is a component of our application server. The WSO2 Data Services component allows you to take any data source that is accessible through JDBC, MySQL databases, Oracle databases, or DB2, but, in addition, we also have support for Excel, CSV files, and various other formats and very simply expose it as XML

Now this isn't just exposed to, for example, Web Services. In fact, it can also be exposed by REST interfaces. It can be exposed through XML over HTTP, can even be exposed as JSON. JavaScript Object Notation makes it very easy to build Ajax interfaces. It can also support it over JMS, and messaging system.

So the fundamental idea here is that the database can be exposed through a simple mapping file into multiple formats and multiple different protocols, without having to write new code or without having to build new systems to do that. What we're really replacing there is, for example, where you might take your database and build an object relational map and then you use multiple different programming toolkits -- one Web services toolkit, one REST toolkit, one JMS toolkit -- to then expose those objects.

We take all that pain away, and say, "All you have to do is a single definition of what your data looks like in a very simple way, and then we can expose that to the rest of the world through multiple formats."

Gardner: When that data changes on the core database, those changes are then reflected across all of these different avenues, channels, and approaches it's sharing. Is that correct?

Fremantle: Absolutely, because it's being accessed on demand and then exposing them as needed through whichever format they ask for. So, it's not storing those data formats in it's own system,

Governor: One of the things that I really like about this story is that we went through a period where there was a view that everything needed to be done with the WS stack, and the only way to do SOA, the only way to data integration, was to use these large-scale Web standards. But they're not critical in all cases, and it really depends on your requirements for the security and so on. Do you really need SOAP and some of the heavier weight protocols and technology?

I think that the approaches that say, "Let's understand is this behind the firewall? What are the levels of protection that are required?" "Can we do this in a simpler fashion?" are very valuable. The point about JSON, for UI related stuff, certainly REST kind of interfaces, but at the end of the day it's a question of, do you have developers that are available out there in your shop or to hire that are going to be able to do the work that's required and some good examples that came out of the Web world?

If you look at eBay, they had a SOAP API, but nobody used it. A great number, or 80 percent plus, of the calls were using RESTful styles. Understanding the nature of your problem and having more flexibility is very, very important.

Gardner: One of the things that I really like about this is that, almost like Metcalfe's Law. The more participants there are on the network, the more valuable it is. The more people and systems and approaches to distributing data, the more valuable the data becomes. What's been nice is that we've elevated this distribution value with data, at the same time that open source and community-based development have become much more prominent.

That means that the ways in which the data is shared and transferred is not just going to be dependent upon a commercial vendor's decision about which standards to support, but we can open this up to a community where even very esoteric users can get a community involvement to write and create the means for sharing and transferring.

The data can take on many more different integration points, and standards can evolve in new and different ways. Let's discuss a little bit, first with Paul, about the role of open source, community, and opening up the value of data.

Fremantle: I am just a fanatic about open source and community. I think that open source is absolutely vital to making this work, because fundamentally what we're talking about is breaking down the barriers between different systems. As you say, every time you're pushing the proprietary software solution that isn't based on open standards, doesn't have open APIs, and doesn't have the ability to improve it and contribute back, you're putting in another barrier.

Everyone has woken up to this idea of collaboration through Web 2.0 websites, whether through Flickr or FaceParty or whatever. What the rest of the world is waking up to is what open-source developers have been discovering over the last five to ten years. Open source is Web 2.0 for developers. It's how do you collaborate, how do I put my input, my piece of the pie? It's user-generated content for developers, and that power is unbelievable. I think we're going to see that grow even more over the next few years.

Governor: I fundamentally agree with it. Open source was an application of a pattern. Open source was the first real use case for this need for a distributed way of working, and we're certainly seeing that broadened out. People are getting a much, much better understanding of some of the values and virtues of open approaches of exposing data to new sources.

Very often, you will not get the insight, but someone else will, and that sort of openness and transparency, and that's one of the key challenges -- actually just getting organizations to understand some of the value of opening up their data.

I think that is one thing to have to tools to see that. Another is that we all now are beginning to see organizations kind of get it. Certainly, "How do we syndicate our information?" is a really key question. We are seeing media companies ask themselves exactly that. "Do we have an API? How do we build an API? Where do we get an API, so that people can syndicate the information that we have?”

I suppose I'm just double-clicking on what Paul said -- that passion is something that is becoming more and much better understood. Reuters is realizing it has to have an API. The Guardian, which is a British newspaper -- and those Americans certainly of the leftward persuasion are very familiar with it -- now has a team that is also presenting at Web conferences and talking about the API. We've got to think about how to make data more available, and open source will just be the first community to really understand this

Gardner: I'd like to bounce this off of Brad at Concur. Do you feel a little bit less anxious, or more at ease, knowing that whatever data needs that you have for the future, you don't have to wait for a vendor to come up with the solution? You might be able to go and explore what's available in a community, or if it's not available, perhaps write it yourself or have it written and contribute it back to the community. It seems to me that this would be something that would make you sleep better at night -- that an open-source and community-based approach to data services deliverability gives you more options.

Svee: I personally love open source. I think that it is the movement that's going to fix software and all these proprietary systems. I think that my small team, four developers and myself, would not be able to produce the kind of quality products internally that we're essentially asked to do, without being able to stand on the shoulders of a lot of these geniuses out there who are writing amazing code.

Gardner: Do you agree that there is this sense that you can almost future-proof yourself by recognizing, as you embrace open source, that you're not going to get locked in, that you're going to have flexibility and opportunity in the future

Svee: Exactly. I find that there are a few products that we have that we've been locked into for quite some time. It's very difficult to try to move forward and evaluate anything new, when we're locked into something that's proprietary and maybe not even supported anymore. With the open-source community out there, we're finding that the answers we get on forums and from mailing lists are every year getting faster and better. More people are collaborating, and we're trying to contribute as much as we can as well.

Gardner: And, of course, over the past several years, we've seen a tremendous uptake in the use of open-source databases and sources from MySQL, Ingres, Postgres, and there are others. Let's bounce this back now to the WSO2 product set. What is it about, when you are developing your products, Paul, that open source becomes an enabler, as well as, in a sense, a channel into the market?

Fremantle: What was interesting about us developing this data services solution was the fact of what we built on top. The data service's component that we built actually took us very little time to get to its first incarnation, and obviously we are constantly improving it and adding new capabilities.

We were working on that and it didn't take time, but the very first prototype of this was just a piece of work by one of our team who went out and did this. What enabled that really was the whole framework on which it was built, the access to framework, the app server that we built, and that framework built on the work of literally hundreds of people around the world worked on it.

For example, if we talk about the JMS support, that was a contribution by a developer to that project. The JSON support was a contribution by another developer and relied on the JSON library written by someone else. The fact that we can choose the level of encryption and security from HTTPS all the way up to full digital signatures relies on the works of the Apache XML security guys who have written XML security libraries. That's an incredible, complex piece of work and it's really the pulling together of all these different components to provide a simple useful facility.

I think it's so amazing, because you really stand on the shoulders of giants. That's the only way you can put it. What I like about this is to hear Brad say that he is doing the same, we are doing the same, and all around there is a value change of people doing small contributions that, when put together, add up to something fantastic. That's just an amazing story.

Gardner: Given that there are many approaches that Brad, as a user organization, is undertaking, and they dovetail somewhat with what you are doing as a supplier, we also have other suppliers that are embracing open source increasingly and building out product sets that have originated from technology that was contributed or project format or license. How do these pieces come together, if we have a number of different open-source infrastructure projects and the products? I'm thinking about perhaps an ESB, and your data solution, and some integration middleware. What's the whole that's greater than the sum of the parts?

Governor: I certainly have some pretty strong opinions here. I think we can learn a lot from the ecosystems as well. One of the absolutely key skills in open source, as a business, is packaging. Packaging is very, very important to open source, and pulling things together and then offering support and service is a very powerful model.

It's really nothing new. If we look at personal computers, you go out and you can buy yourself chips from AMD or Intel, you can buy an OEM version of Windows or choose to do with Linux, you can buy RAM from another company, you can buy storage disks from another company, and kind of glom it all together.

But, as that industry has shown us, it really makes a lot more sense to buy it from a specialist packager. That might be Dell, HP, or others. I think that open-source software has really got some similar dynamics. So, if you want an Eclipse IDE, you are likely to be buying it from an IBM or a Genuitec or CodeGear, and a couple of those are our clients. I should disclose that.

In this space we've got the same dynamics. If you are, for example, a Web company, and you don't want to be paying these third parties to do that packaging for you, fine. But, for the great mass of enterprises, it really doesn't make that much sense to be spending all your time there with glue guns, worrying about how pieces fit together, even in Eclipse, where it is a very pluggable architecture.

It makes a great deal of sense to outsource that to a third party, because otherwise it's really a recipe for more confusion, I would argue. So yes, you can do it yourself, but that doesn't necessarily mean, you should. The PC example, yes, for a hobbyist or someone who wants to learn about the thing, absolutely, build your own, roll your own. But, for getting on with things in business, it does make sense to work with the packager that's going to offer you full service and support.

Fremantle: I've got to jump in here and say that's exactly our model. Though we don't just offer the data services, we offer, an ESB, a mashup server, and SOA registry, and we make sure all those things work together. The reality is that there are a lot of programmers out there who are hobbyists, so there are a lot of people who do like to take individual components and pieces and put them together, and we support both of those equally, but I think your analogy of the PC market and that plug and play model is absolutely like open source and specifically open-source SOA. We all focus very much on interoperability will make sure that our products work together.

Open source drives this market of components, and it's exactly the same thing that happened in the PC market. As soon as there was an open buy off that wasn't owned by a single company, the world opened up to people being able to make those components, work in peace and harmony, and compete on a level playing field. That's exactly where the open-source market is today.

Gardner: So how about that, Brad? Do you really like the idea that you can have a package approach, but you can also shake and bake it your own way?

Svee: That's exactly the sweet part in my opinion. I can shake and bake, I can code up a bunch of stuff, I can prototype stuff rapidly, and then my boss can sleep well at night, when he knows that he can also buy some support, in case whatever I cook up doesn't quite come out of the oven. I see there's a kind of new model in open source that I think is going to be successful because of that.

Gardner: Okay, now we have seen some very good success with this model: have it your way, if you will, on the infrastructure level. We are moving up into data services now. It seems to me that this also sets us up to move an abstraction higher into the realm of data portability. Just as we are seeing the need in social networks, where the end user wants to be able to take their data from one supplier of a social networking function to another, I think we are going to start to see more of that in business ecologies as well.

A business will not want to be locked into a technology, but it also doesn't want to be locked into a relationship with another supplier, another business. They want to be able to walk away from that when the time is right and take their data with it. So, maybe we'll close out our discussion with a little blue-sky discussion about this model taking a step further out into the cloud. Any thoughts about that, Paul?

Fremantle: I think that's a really interesting discussion. I was at a conference with Tim O'Reilly about two years ago and we were having exactly this discussion, which is that openness of services needs to be matched by openness of data. We are definitely seeing that in the Web marketplace through back-end systems like Amazon S3 storage, and we are beginning to see a lot of other people start to jump on this and start to build open accessible databases.

I think that's an absolutely fantastic usage for this kind of data service, which is to say, "It's my data. I don't just want to host things in an open fashion. I don't want to write code in an open fashion. I want open services and open data, so I can get it, move it, protect it myself, and relocate it."

So, I think there's a really interesting idea behind this, which is, once we get to the point where your data is no longer tied to a specific system and no longer has to be co-located with a particular MySQL database, we start to free up that processing. If you look at what Amazon did with the Elastic Cloud Service and their storage system, the storage system came first. The data services were a precursor to having an effective cloud-computing platform. So, it's really a precursor. You have to have data services, before you can start to migrate your processing and scale it up in this fashion.

Gardner: What do you think, James? Is this something that will be in great demand in the market, and there is also a green angle here?

Governor: Yeah, I think undoubtedly it will. Simon Phipps from Sun talks about the freedom to leave. We had a big example recently, Comcast buying Plaxo. They have lost a lot of the users. A lot of Plaxo users just closed up their account there. Interestingly enough, Plaxo had a nice function to do that -- very good for closing the account, not so good for exporting the data. I am not so sure the problems are primarily technical. I think there are a great deal of policy and social problems that we are going to have to deal with.

It's very interesting to me that we call people heroes that are trying to break Facebook terms of service, in some cases with the recent data portability example. We've got some really key challenges about what does data ownership mean. From my perspective, as I said earlier, I think it's very important that we have the mechanisms whereby we have access to data without necessarily allowing replication of it all over the place.

If it is your data, then yes, by all means, you should have permission to take a copy of it. What about if you're on a network and you want to take all the data and all of the surrounding metadata? Really, the discussion becomes about that metadata. Am I allowed to get anything back from Google about my behaviors and other people's behaviors?

It's really a social question, and we, as a society or a number of different societies, have got to think about this, and what we want from our data, what we want from privacy, and what we want we want from transparency. We can gain wonderful things, I mean wonderful advantages, but there is also the flip side, and I think it's very important that we keep that in mind.

So, it's going to be a wild ride. It's exciting, and I think that it is important that we get the tools in place, so that once we get the policies well understood, we can actually begin to do things more effectively. So, again, it's very exciting, but there are a lot of threats and lot of risks that we do need to take account of. Those risks are expanded, as I say, by what I sometime call "information bulimia." This notion that we just keep eating and swallowing more and more information and more data and we need more information, and if you do that, what you end up doing is puking it all up.

Gardner: Let's close here with that real-world perspective, Brad, aside from the visual image of puking, does this interest you in terms of the idea of third-party neutral cloud-based data and does that have any bearing on your real-world issues?

Svee: Well, I can give you an example what we were able to do with data services. Within a matter of weeks, not even months, we are able to use the data services in the application server from WSO2 to essentially give a complete client picture to the business by reaching into the ERP system, pointing out invoices and products, and then reaching into the CRM system to pull out open issues, as well as, sales manager, probably about 50 data points about each customer from the CRM, and then expose those services through a simple JSON-based UI with a smart type-ahead for the customer name. Quickly, we are able to show a picture of our clients that hadn't previously been available -- and within a matter of weeks actually.

Gardner: That data could have come from any number of different sources if, to James' point, you had the proper permissioning?

Svee: Yeah, and since we are IT and we own the systems, we are able to determine who is who, and we were able to use a Web service, another data service into our HR system, to pull out roles to see whether or not you could access that information.

Gardner: That's highly valuable from a productivity and planning perspective. If you are a business strategist, that's precisely the kind of information you want?

Svee: Exactly, and they were amazed that they've had been able to live their lives without it for so long.

Gardner: Paul, do you think much of this common view business, when it comes to data services?

Fremantle: Actually, we are working on another project with a health-care provider, which is providing a single patient view. So, it's exactly the same kind of scenario with significant security and encryption and data challenges to make sure that you don't provide the wrong information to the wrong person. Obviously, all the same issues need to be solved, and being able to pull together everything that is known about a patient from multiple different system into a single view once again has huge value to the organization.

Gardner: Well, this has to be our swan song on this particular podcast. We are out of time. I want to thank our guests for helping us get into a nice far-reaching discussion about data services, what the problem set has been, what the opportunity is, and how at least one organization, Concur, is making some good use of these technologies. We have been joined by Paul Fremantle, chief technology officer at WSO2. Thank you, Paul.

Fremantle: Thank you, it has been great fun.

Gardner: I also strongly appreciate your input Brad Svee, IT manager of development at the Redmond, Wash.- based Concur. Thank you, Brad.

Svee: Well, thank you.

Gardner: And always, thank you, James Governor from RedMonk for joining. We appreciate your input.

Governor: Thank you much. It has been an interesting discussion.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions. You have been listening to a sponsored BriefingsDirect Podcast. Thanks and come back next time.

Listen to the podcast. Sponsor: WSO2.

Transcript of BriefingsDirect podcast on data services, SOA and cloud-based data hosting models. Copyright Interarbor Solutions, LLC, 2005-2008. All rights reserved.