SaaS Providers Increasingly Require 'Ecology' Solutions from Infrastructure Vendors

Edited transcript of BriefingsDirect[TM] podcast with Progress Software's Colleen Smith on SaaS, recorded July 26, 2007.

Listen to the podcast here. If you'd like to learn more about BriefingsDirect B2B informational podcasts, or to become a sponsor of this or other B2B podcasts, contact Interarbor Solutions at 603-528-2435.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you're listening to BriefingsDirect. Today, a podcast discussion about Software as a Service (SaaS), the burgeoning marketplace for off-the-wire business and consumer applications, and the infrastructure that's required for those delivering these applications and services to thrive and prosper.

To help us sort out this market and the needs for infrastructure, we're joined by Colleen Smith, managing director of Software as a Service for Progress Software. Welcome to the show, Colleen.

Colleen Smith: Thanks, Dana.

Gardner: Tell us a little bit about how Progress Software, you, and how your research came to determine that the time is right for ramping up your offerings and their applicability to the SaaS provider market?

Smith: I was lucky. Progress had started to look at the application service provider (ASP) model back in the early 2000-2001 time frame to figure out whether there was an opportunity for some of the small ISVs who were using the Progress technology to become more of an application service provider. When I joined the company two years ago, I was basically asked to figure out how to build more of a SaaS partner program and look at ways in which we could work with our partners.

We basically stepped back and said, "All right, let’s look at a number of different areas," one being the technology enablement and how to build applications to go to market with SaaS. We also added a couple of other things, because we felt that one of the biggest challenges traditional software vendors had was around the business model, the go-to-market strategy, sales enablement, and figuring out ways in which we could actually help them to be more successful in this new business model. We were thinking of it more as a business model and not just as a technology.

Gardner: So, it's also providing the back-office applications and requisite integration for them to have a business go at this, not just the underlying guts of getting the scale and reliability for the applications themselves?

Smith: Right. We’re an infrastructure provider, but we also look very carefully at this channel, which happens to be ISVs that bring our infrastructure products to market. We wanted to make sure they could be successful with SaaS. Sure, there are the technical components of multi-tenancy, being able to have a Web-based access, and being able to drive policy configuration and personalization.

More importantly, we work with a lot with our partners or these ISVs to make sure they realize that this requires different marketing. It requires a different sales and business model, because clearly there are financial implications in terms of cash flows. There are also a lot of things they need to think about in terms of who is the target market.

We've helped them focus on looking at new markets and going down-market. Our partners have always focused very much on the mid-market, but SaaS has enabled them to target some very niche verticals and go down into the "S" of SMB (small and medium business).

Gardner: You mentioned application service providers (ASPs), and there was a lot of talk along the lines of potential for this market at least 10 years ago, maybe more depending on how the delivery was designed.

What's different between what was then conceived of as the ASP market and what we now call the SaaS market?

Smith: When I look back at the ASP market and what was going on, it was much more about the hosting. Everybody said that if you just take a business application and host it, you can be an application service provider. What they didn’t realize was that the folks who were trying to do the hosting really had no domain knowledge in terms of the business application. They didn’t understand how to focus on managing the business processes. They focused on getting the application up and running and hosting it.

There were a couple of problems with that. Number one, the applications were built for on premises. They weren’t built to be used by multiple customers. The other thing is that the people who had built those applications weren’t necessarily the ones who were doing the hosting. It was the hosting vendors who figured they could just load it up and run it.

SaaS and ASP in concept are still the same. The application is going to be housed and managed centrally, hosted somewhere, and run by different customers. The biggest difference is that the people who are now managing, building, and deploying those applications are more of the ISVs, who understand what it takes to run and manage that business process.

On the software side of it, there is much more of a focus on business-process automation, and the people who are building, deploying, and running those applications have a good, solid knowledge of the business itself. The second thing is that the applications are now architected specifically to be able to run for multiple customers, and it’s not a separate implementation for each customer.

The economy of scale is what killed a lot of hosting providers back in the ASP days and ran them out of business. They were just doing an implementation for every customer, as opposed to a single implementation that can now be used by multiple customers -- personalized and managed. The people who use the application run and use it differently, but the implementation is pretty much the same for all customers.

Gardner: Right. I want to get into a little bit of why the costs have come down and what the architectures are that make it more of a pure of-for-and-by the Web play. Before we do that, you mentioned that you've been at Progress for a couple of years. Why don’t you tell us a little bit about your background, and how you've come to be involved with the SaaS market?

Smith: Prior to coming to Progress about two years ago, I spent five years as an industry analyst. I was looking at the ASPs back in those days and understood what was going on around the market, but I also looked at the overall infrastructure market. Prior to that, I spent 15 years in the enterprise-application space.

What's interesting is my background. I started in the mainframe days, saw the transition to client-server, then saw the transition to the Internet. So, I watched the different transitions in how the software industry has grown over the last 20 years or so. I even started off my career at EDS doing service bureau and outsourcing. I've seen it come full circle.

When I joined Progress, I said, "Let’s look at this new business model, SaaS, and figure out how an infrastructure company can make a play in terms of being part of this new business model through using our technology, but at the same time working with a lot of ISVs who are out there trying to figure out how to make that transition.

Gardner: For those listeners who might not be that familiar with Progress. You have been involved with ISVs, first with tools, then with runtimes, and then increasingly with stack and infrastructure platform approaches. So, your core audience, your clientèle, are really ISVs. Isn't that the case?

Smith: Yes. We’ve been in business 25 years, and I would say for a 23 of those years our go-to-market has been to work with ISVs who take our technology, build applications on top of it, and bring it to market. We also have a direct arm, but a long-standing portion of what we’ve done over the years is to provide the software infrastructure for application partners or ISVs to be able to bring their applications to market, build them, and deploy them.

Gardner: One of my earlier questions was about the timing. Why is the timing right now. For a lot of reasons, these ISVs recognize that they can continue to support their licensed, on-premise businesses and, at the same time, pursue other market segments at a price point that makes it worthwhile. Would you agree with me that the timing is right for the ISVs, as they increase their business model directions, in terms of the price points? Why is this a good time for these ISVs to start looking at SaaS?

Smith: I do agree with you. I think the timing is right. There are a bunch of reasons why. Number one, the Web is finally viewed as a business platform. Seven or 10 years ago, the Web wasn't viewed as the way in which business applications were going to be run and managed. Because of that, as I mentioned before, a lot of traditional ISVs have been selling to the upper end of the mid-market, or to large enterprises. Those have been the folks buying business applications.

The "S" in SMB really couldn’t do a number of things. They couldn’t afford the dedicated IT staff to manage and maintain the applications. They didn’t necessarily have the infrastructure and the technology to run these business applications. A lot of business applications are much too complex and require too much manpower to manage and maintain the app.

A couple of things have happened. One, the price of computing has come down. People now have access via web browser to business applications.

The other thing, one we’ve all seen, is that ISVs realize there’s a whole new market. There’s that long tail, if you will, of the software market that allows them to be able to go after new people. In the past, software just wasn’t accessible to them, and now there’s a whole new market opportunity.

We stress to our ISVs, "You can continue to be in the traditional software business for your core market and the market that you’ve been going after, but there’s a whole new opportunity for you to look at new markets, whether they be the low-end of your current market, adjacent markets, or even new geographic territories."

Throughout South America, Africa, and Asia-Pacific, what we’re finding is tremendous growth opportunity for ISVs to look at these as new markets and to go into those new markets with a new business model. That new business model is SaaS.

Gardner: So, on the delivery side of things, that is to say on the direction in which they sell, there’s this long tail and globalization benefit that provides them new market opportunities. These areas are ripe for the need for productivity, yet they probably don’t want to set up their own data centers.

On the other side of the equation, on the supply side of how these ISVs can deliver, there’s a new support ecology available to them. They don’t have to create their own data centers themselves. They can find partners. We’ve heard a lot about Amazon, for example, and there are others, of course. These ISVs can focus on what they do well, which is their software, their logic, and then also take advantage of some hosting.

Tell us a little bit from your vantage point as a software infrastructure and tools provider how that ecology works when it comes to these hosting options?

Smith: You’re exactly right. Back in the ASP days, it was all about hosting. I’m not saying that in the SaaS world hosting isn’t important, because it absolutely is. What has changed over the last 7 to 10 years is that now you look at it in terms more of an ecosystem.

You’ve got your infrastructure providers, your application providers, and your hosting and managed-service providers. The biggest change that I have seen now is that each realizes they have a role to play, they have a core expertise, and that through building of this ecosystem and through partnerships you can be much more successful in being able to lower your deployment cost, but still being able to target and go after these new markets.

I look at our ISVs and our ecosystem within SaaS. We use partners like OpSource, for example, to be able to do some of the hosting and managed services. Our ISVs are the ones with the business and domain expertise, and know what their business application is. They know their particular vertical niche and they know how to best deploy, manage, and build out business processes to be able to support it. What we provide in the equation is the underlying infrastructure that helps them to develop, deploy, integrate, and manage and monitor their business applications.

Together, what we have is more of an ecosystem that’s able to go out and lower the overall total cost, because each one of us is playing our role in the system. Because it’s a partnership, the pricing and licensing is all done based upon what we call the shared risk/shared reward model.

Gardner: I think they also call that "pay as you drink," right?

Smith: Something like that.

Gardner: Now, we’re not just talking about big honking business apps anymore either. As more companies adopt Web services and look at methodologies such as SOA, we can perhaps get a little bit more granular as to what is hosted, and look to business process management from a hosted or SaaS view. I guess we could call them "components as a service." That requires quite a bit more heavy lifting on the infrastructure side. There’s more integration, managing data, and making data available in near real time across multiple services.

Can you address the opportunity for applications to be decomposed into services as a service, and also what new requirements there are for the infrastructure to support that?

Smith: Sure. What happens early on in a market is that we see lot of these niche, vertical, best-of-breed or single applications or components in the first wave of coming out to market with SaaS. So, whether it's in the legal sector, healthcare sector, or financial services, they say, "Here’s one specific business application -- mortgage applications, loan applications, or patient billing." What slowly happening is being able to start integrating business processes and offer them out to the community.

If you look at financial services, instead of just being able to offer loan applications, there’s now a whole suite of different types of business services or business components. As long as somebody who’s part of the financial services arena has the ability to integrate those different business processes and offer them out to their community, they basically have become more of a business service provider.

What we’re seeing is that you no longer are an application vendor like you were in the traditional business model. If you do this right and you use the underlying technology of governance, policy enforcement, integration, and development, then you can build out a whole service delivery environment or platform, where you can now offer multiple business services to a community that might be very vertical-based.

We see a lot of this happening in financial services, healthcare, the legal sector, and even in agriculture, which needs to now manage and maintain a lot of different business processes because of federal regulations, mad-cow disease, and all the other reasons people have to manage and monitor business processes a lot more thoroughly.

Gardner: Okay. So, where the traditional ISV might have had a monolithic application standalone, we’re seeing more decomposition into services. Their role becomes more of a services integrator off the wire, but at the same time they have more flexibility in how they deliver.

They also have more flexibility in how they create, in that they can exploit reuse more generally. They can also shop around for business services that might be on the market and available to them through the ecology. So, there seems again to be a two-prong benefit: one in how they can deliver, but also in how they can aggregate and create. Does that sound right?

Smith: Yes, and what that comes down to is the winner in all this is the end customer, who is looking for a single business-service provider who knows their business, whether they know healthcare, legal, or any business sector, and they’re able to provide a number of different business services for them. The big challenge that most large organizations had was integration, because they’d go to one vendor and buy a business app, then go to another vendor and buy a different business app.

In the SaaS world you could run into the same thing, if you’re going out to all these different SaaS providers. But, if you start to think in terms of those SaaS providers participating in this ecosystem that’s much more based upon who the end customer is, the end customer can end up benefiting. They’ll be able to go to a single business-service provider.

Maybe that business provider has built those services, or maybe they haven’t, but they’re able to pull in these multiple business services, Web services, or whatever technology they’ve been built in, and offer them out to the community. So, the end customer, who might be that smaller business user, can now have a single point where they go to access a number of different business services.

Gardner: Again, that points to the long tail. You can have a higher level of customization, but also manage your costs in dong so.

Smith: Exactly. The biggest challenge people have about going after that long tail is the fact that you’re really talking about millions of markets of a dozen. It’s very difficult to get your cost model to a point where you’re able to go after all of those millions.

But, if you really think about it and you build the business application that’s very specific to what they need to do, and you’ve built them based upon the small services, then the customer chooses which business services they need to run their business. You’re offering a spectrum of different services, because you understand what their marketplace looks like and what vertical they’re in.

Gardner: Now, when I speak to some enterprises and increasingly smaller businesses -- the S in the SMB -- I keep hearing the same concerns come up around data. They’re concerned about data ownership, managing iterations of data, and coordinating data between their on-premises locations, applications, data centers, what might be happening out in the cloud, and within the ecology, and then how that translates properly to the end user.

Now, Progress Software has made some acquisitions and has focused quite a bit of energy and investment on this data issue -- the real time, management, semantic issues around not just data availability, but about how things are termed, labeled, and classified, application by application, instance by instance, and even site by site. Help us understand how Progress is helping the ISVs in the SaaS environment deal with somebody’s complex data and semantic issues.

Smith: When people initially thought of integration, they thought about point to point, and it was more about at the business process level. We realized was that if you’re not addressing data-level integration semantic issues, then you’re not going to solve all of the problems that customers have.

We made an acquisition last year of a company called Pantero, and we have built a product that we’ve termed Data Xtend Semantic Integrator. It's all about looking at the semantics of the data and being able to match and manage that data from one system to another. So, it’s just another product that we’ve added to our infrastructure, and it allows customers or our ISVs to look at all different levels. We’ve got integration at a business-process level, at the enterprise service bus (ESB) level. We now have integration at the data level. We also have capabilities to govern and monitor and manage Web services.

Progress continues to add different technologies into our environment to support what’s going on in terms of all of the different integration and Web services challenges happening in the industry. Our basic focus is to continue to add software infrastructure components to support the needs of large enterprises, as well as to support the needs of business-service providers who are trying to offer these integrated business applications to their customers.

If you think about it, large enterprises can do a lot of this themselves and can buy the infrastructure to build, integrate, and manage. In a lot of ways, the SMB requires these services providers or these business services providers to be able to do all of that integration, and they expect that that integration will just be handled for them, and that’s what they’re really looking for.

So, that’s the big challenge of whether ISVs are going to become business-service providers or whether they’re going to partner with business-service providers. They almost become the manufacturer, if you will, of these small-business components that larger business-service providers will pull into their environment. So, you might see a different breed. We might move away from the traditional systems integrators and you might see more business-service providers focusing on supporting the customers. The ISVs are actually building some of the components that are used, but they aren’t necessarily going to be the service providers.

Gardner: So, it's still murky and up in the air as to how this supply chain works, who interfaces with the customer, and which type of organization does customization versus underlying OEM types of activities.

Colleen, I wonder if you could put on your analyst hat again for a minute and try to forecast how that market might shake out.

Smith: I think the SaaS market, in general, is really still in its nascency, and there are a lot of things that have yet to happen. But, the good news is this isn’t just a fad. We see a fundamental change in terms of the business model.

What I say a lot is that if we think in terms of the software industry over the last 20 years, we’ve come a long way in terms of building partnerships, and in terms of how systems integrators and service providers work with ISVs. What I see being the success of SaaS is that if we continue to enhance that model, it's going to be about hosting providers, working more closely with system integrators and ISVs. The only way that the end customer is going to win in this is if we get into a business model where there is that shared risk and shared reward, but the customer pays for only what they need to use.

It's going to come down to pricing models. It still has to come down to some building of ecosystems out there, where everybody knows their role and plays that role, but doesn’t necessarily try to do the other person’s role. There are still a lot of things happening.

I believe it’s going to be vertically focused. I don’t think this is going to be a horizontal play. We’ve seen a lot of success in vertical business expertise. There's going to be content, business applications, data, and services. If all of those can be offered in a single environment through a single service provider, the customer will end up winning.

Gardner: I suppose from the business point of view, software companies historically have been motivated to try to move towards the one-size-fits-all volume model. That’s been in their best interest. So, what we’re seeing here is the flipping of that model to, "You can do as well, or perhaps better, over time by focusing on the one-size-fits-small niche. Through reuse and efficiency in your infrastructure and tooling you can accommodate those small niches and still prosper." Is that the value proposition that you would like to stake out for Progress?

Smith: Absolutely. We’ve always worked with a lot of our partners and told them, "Figure out where your niche is, and, if you can be the best at your niche, you can be successful." We aren’t necessarily talking about creating the next SAP, but if you can be really successful within your specific niche area, then your customers are going to value your service.

In the SaaS model, there are two S's. One is the software part and the other part is the service. If you have business domain expertise, they’re going to look at you as a partner and they’re going to ask you to help to run, manage, and grow with their business. That’s the other part. If you’re focusing on SMB, you also have to help those small organizations figure out how they can scale to become large organizations. So, it’s your opportunity as well as your responsibility to make sure that you can scale with them.

Gardner: So, it’s domain expertise that becomes a differentiator that’s more valuable over time, rather than just the ability to write good software.

Smith: Yes. I often talk about it. You’re moving from develop-package-ship into develop-build-service-deploy. It’s much more about the ways in which you can deploy and service the customer, as opposed to just packaging, shipping, and saying, "Well, now it’s somebody else’s responsibility."

Gardner: I suppose it's being more of a partner with your customer, rather than just throwing software onto a disk, shipping it out, and saying good luck. Right?

Smith: Yes, because the switching costs are lower. It’s not necessarily happening in the industry today that everybody is dumping their SaaS application to go to another one, but you have to be able to service your customer. If they’re not getting the service that they require from you, they will look elsewhere.

Gardner: We’re getting towards the end of the show. I wonder if you have some examples that you could share with us, either by name or by definition, of ISVs that have focused in this vertical direction, have embraced SaaS, and are getting along okay.

Smith: I’ve got a couple of interesting examples in terms of ways in which you can think specifically about accessing the long tail, but being able to target a whole new set of users.

One of our partners is in the K-12 education area. They had a traditional business application that sold to faculty and to the student administration systems. They figured out that they could offer a business application that can now be accessed by parents over the Web. So, they re-architected their application for multiple school districts. They now allow parents to go in and track absenteeism, as well how students are doing in terms of grades, and things like that.

They've completely re-architected and re-thought the way in which they’re building and deploying business applications for K-12. That’s an interesting example of thinking about SaaS and thinking in terms of a new market, as opposed to just looking at large universities or schools that can afford a system.

They’re selling at the state level and saying, "Here is a state-wide student administration system that can now be used at all schools. Even if they don’t have a large IT staff, all they need is access via the Internet." That’s interesting in terms of the education. This is one of my small partners called Skyward, and they’re located in Wisconsin.

Another interesting area is in library management systems. They've been around for a while, and we’ve got a partner, Keystone, who has focused on a very small niche in terms of braille and applications for the visually impaired. What they’ve done is to rethink the way in which you can have access to books and to look up books available at libraries. And, it can all be done via the Web.

They’re actually coining themselves as the Netflix, if you will, of libraries, because what you can now do is use the Internet to look for availability and access to different books that might be available, not only in your town, but statewide, and have those books actually shipped to your house.

So, they've re-thought the way in which library systems are built and used, and are able to bring in access from the Internet, and, in this particular instance, can allow handicapped individuals access information right from their homes.

These are just two examples where you start to think about who the user of a system is. It’s a very traditional backend accounting and business management system, but it’s now being used and serviced to expand their market, as well as just to be able to have a service for some small end users, who, in the past, wouldn’t have had access to these types of technologies.

Gardner: So, it really provides for some creative business development by looking for these, as you say, entirely new types of customer bases.

Smith: It's looking at new markets, being able to target now at a statewide level, as opposed to a smaller individual school or individual library. So, it’s a new market opportunity, but they’ve been able to take something they’ve been doing for years with domain expertise and really expand their opportunity. What we see is that there are tremendous opportunities for ISVs out there, if they step back and think in terms of what they know, what's business domain expertise they have, and where they could provide a better level of service to consumers or customers.

Gardner: So, depending on the type of organization you are, you have the opportunity to scale up -- if that makes sense -- to scale down, perhaps to do both, and then, as we mentioned earlier vis-à-vis globalization, in a sense, scale sideways.

Smith: Exactly, we’ve had other partners who have looked at this as an opportunity. I've got a law firm software company in the UK that looked at this SaaS as an opportunity to go to both Australia and South Africa. They have expanded their territory, without having to build a lot of support in data centers and development centers in Africa or Australia.

They’re still running their main headquarters out of the UK, but they’ve been able to go into these two countries, because it just so happens that the legal systems in South Africa and Australia are similar to the one in the UK. So, the business processes were similar but they were able to expand into new geographies.

Gardner: Very interesting. Well, thanks. We’ve had a nice discussion about Software as a Service, and particularly on the business development opportunities for ISVs. We’ve been discussing this with Colleen Smith. She is a managing director of Software as a Service for Progress Software. Thank you, Colleen.

Smith: Thank you, Dana.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions, you've been listening to BriefingsDirect. Come back and listen next time. Thank you.

Listen to the podcast here. Produced as a courtesy of Interarbor Solutions: analysis, consulting and rich new-media content production.

If any of our listeners are interested in learning more about BriefingsDirect B2B informational podcasts or to become a sponsor of this or other B2B podcasts, please fill free to contact Interarbor Solutions at 603-528-2435.

Transcript of Dana Gardner’s Podcast on SaaS with Colleen Smith of Progress Software. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.

BriefingsDirect SOA Insights Analysts on Defining SOA and IBM's DataMirror Acquisition

Edited transcript of weekly BriefingsDirect[TM] SOA Insights Edition podcast, recorded July 20, 2007.

Listen to the podcast here. If you'd like to learn more about BriefingsDirect B2B informational podcasts, or to become a sponsor of this or other B2B podcasts, contact Interarbor Solutions at 603-528-2435.

Dana Gardner: Hello, and welcome to the latest BriefingsDirect SOA Insights Edition, Volume 22, a weekly discussion and dissection of Services Oriented Architecture (SOA) related news and events with a panel of industry analysts and guests. I’m your host and moderator, Dana Gardner, principal analyst at Interarbor Solutions. Our panel this week, and this is the week of July 16^th, 2007, consists of Joe McKendrick. Joe is a research consultant, columnist, and blogger. Welcome back, Joe.

Joe McKendrick: Hi, Dana, glad to be here.

Gardner: We’re also joined by Tony Baer. He's a principal at OnStrategies. Hey, Tony,

Tony Baer: Hey, Dana.

Gardner: We're joined once again by Neil Ward-Dutton, research director at Macehiter Ward-Dutton. Welcome back, Neil.

Neil Ward-Dutton: It's great to be here, Dana. Thanks for having me.

Gardner: Also, filling out our roster -- and we have a large group today -- Todd Biske. He is an Enterprise Architect at MomentumSI. Welcome back, Todd.

Todd Biske: Good morning, Dana.

Gardner: Also, Dave Linthicum, CEO of Linthicum Group. We’re glad you’re here Dave.

Dave Linthicum: Glad to be here, Dana.

Gardner: Last but not least, Brad Shimmin, principal analyst at Current Analysis. Good morning, Brad.

Brad Shimmin: Good morning, Dana.

Gardner: Our topics today are, on one hand, a large amorphous topic, which is to try to once again define and categorize SOA -- what it means and where it's going. Then second, a bit more refined, direct topic, and that is the recent acquisition by IBM of DataMirror, or perhaps the intention to acquire DataMirror, and what that means for services, the data-services layer, and the inclusion of more data for IBM in their Information Server Suite approach.

Let's start with the larger topic of the day. What's been coming up in some of the blogs and in some analyst reports is this notion of people overusing the term SOA categorically, and the suggestion that we should go back to the roots and get pure again about SOA. Some are saying that this other business around it being application types, application support, runtimes, and actual products and platforms needs to be muted a little bit.

At the same time, we’re also seeing discussions crop up around such things as platform as a service, particularly around the Salesforce.com ecology and approach, and also integration as a service. I recently did a sponsored podcast with Cape Clear Software on this topic. Then, of course, there's the the long-term discussion around software as a service (SaaS), and the ability to use Web services and a variety of application interfaces and APIs to avail yourself of hosted online applications and services.

So, let’s take this out to the group. Are we confusing topics here? There's this notion of things off the wire, outside of the domain of the enterprise. Should that remain separate and distinct, as a discussion point and as a direction for enterprises, from SOA, which is more of an architectural initiative and philosophy within the confines of their enterprise IT systems.

Let's go to Todd Biske first, because, Todd, you're an enterprise architect and you’re working with clients in the field. How do you come down on this? Is SOA inclusive of all of these "x as a service" or "blank as a service" or are they actually part and parcel of the same direction?

Biske: I really think it should be inclusive of all these external platforms. SOA, in my opinion, is really something that’s driven out of enterprise architecture. To go back to the simplest view, it's about the services. If you are able to get services from outside your firewall, those should be included as part of your SOA. You absolutely need to include the availability of external solutions as contributors to your overall SOA for your enterprise. I know Dave has talked about this a lot as well.

There's a lot of debate as to whether SOA is the whole kitchen sink. There's a Yahoo! mailing group, and they’ve had the same debate going on just recently on not just the technical aspects, but whether SOA should encompass all of the culture change that people are saying is so critical to success with SOA. Are we just muddying the term by trying to lump all of this in together?

If we look at it from what it takes to be successful, if you leave those pieces out and focus strictly on the technologies associated with it, you’re not going to get success. At that point we’re not achieving the goals we set out to do. So, it needs to include all of these concepts and have that factored into your overall enterprise approach.

Gardner: Dave Linthicum, what do you think? Do you think that we’re going to water down SOA by including all this stuff, or is there another category that we should put all of this stuff into, in which SOA is a component?

Linthicum: I agree with Todd. It's inclusive. In fact, look at my career in starting up Grand Central and Bridgewerx and all those companies, where it was all about creating SOA outside the enterprise and delivering it as a service. It's what Salesforce.com is doing with their Apex platform today, what they call Salesforce SOA.

So, it's an architecture, and ultimately lines are blurring between where the enterprise stops and the Internet starts. With mashups, outside-in services, all the stuff Google and Microsoft are working on, and all the services that are being consumed within the enterprise, it definitely is inclusive and it has to be considered in the context of the architecture, enterprise architecture, and architectural patterns like SOA.

Gardner: Tony Baer, do you think that this is begging too much of a typical enterprise IT department to both be managing transition and transformation to SOA internally, and, at the same time, start to work with these off-the-wire elements -- be it platform, integration, or services and applications?

Baer: I’m going to take this to another level, which is the business level. If you look at the way business has evolved, or supply chains have evolved, over the past 20 or 25 years, they're really evolving into an ecosystem that is very partner dependent. When you buy a product that is manufactured by a company or that’s branded by a specific company, chances are it is manufactured by somebody else.

So, SOA and enterprise architecture really need to echo or support the direction in which business is going. More and more enterprises are becoming virtual. As a result, if you're running your internal processes, the chances are you’re going to be interacting pretty intimately with one of your business partner’s external processes.

Gardner: So, this for you is really just the evolution of business in general, which is to outsource as best you can.

Baer: It's not just outsourcing. You’re working in coalitions. For example, look at how a car is put together today. There have always been suppliers, but suppliers are now taking higher-level roles. They're actually co-designing. When one of the automakers or OEMs designs a car, chances are they’re designing it with their partners. In the old days, they designed the car and then sent out parts orders to partners.

So, SOA in IT and enterprise architecture really needs to support the way the business is going. Business is becoming more virtual. It includes external processes and partners, and SOA must support that.

Gardner: Okay, but about five or eight years ago, a large number of companies really explored outsourcing, but then pulled back. There was something they didn’t seem to like about it. They seemed to do it for economic reasons, but there was this backlash against it. Do you think that was growing pains? What's up with that in terms of still wanting to retain a lot of control over IT internally?

Baer: Well, it's cyclical. I remember back around 1990, there was a grand announcement by Kodak, "Well, we’re outsourcing IT, because, guess what, IT is not part of our business." I’m not even going to fast-forward this 20 years. I’m going to fast forward it three years. Kodak realized, "You know something? IT is part of our business." They had to take large parts of those 10-year contracts with IBM and, at that time, Digital and, I believe, EDS, and after about three years those contracts essentially collapsed in. They dropped a number of provisions. Today, Kodak is saying, "We’re a digital business."

It's a cyclical thing. If you outsource without managing it, if you just throw something over the wall, you’re going to get what you paid for.

Gardner: Neil Ward-Dutton, this notion of the virtual enterprise, do you see that as being on the same wavelength with SOA activities. Do you agree so far with the discussion?

Ward-Dutton: I thought Tony made some really interesting points there. Certainly, one of the arguments that you could make for a long time about enterprise architecture was that it was locked into a way of thinking that was very last century. It was locked into this evolutionary level of thinking that started around the kind of information-engineering movement that happened in the 1970s and 1980s. It assumed that you were building everything yourself, in long-running waterfall projects, and things didn’t really touch other things. Everything was built in isolation.

That was a big problem for enterprise architecture, because, as Tony pointed out, businesses have been pursuing this kind of virtualization agenda for a long time. Think about EDI and value-added networks (VANs), all that kind of stuff. All of that was designed outside the purview of IT and was done by business teams. The enterprise designers or the architects had no real input into that whatsoever. They were focused on the stuff they build internally for the organization. It's long overdue that architecture practice starts to take a more holistic view of an ecosystem of technology provision.

This thinking is long overdue. SOA, shouldn’t be constrained, as Todd just said, by a particular hosting model, where service is provided from, or indeed a technical choice. It's all about the services and it's all about the architecture, but that shouldn’t be constrained by where that service happens to be delivered or who it happens to be delivered by. You can pursue SOA in an environment where the resources are widely distributed, where responsibility is federated. Indeed, more and more, that’s how people are going have to do it. This is all natural, and as Tony said, really, really needed.

Gardner: Brad Shimmin, we’re getting into this notion that IT, by being isolated, has been a handicap, as companies explore extended supply chains and VANs and reach out and capture clients through the Internet. Do you see it that way? Do you think that IT has been a handicap? And, on a related note, is SOA therefore a catalyst to greater virtualization of companies, greater efficiency, and firm productivity?

Shimmin: Well, yes to both. IT really has been isolated, and I think we can all look at the cataclysmic failure of the idea of the waterfall method of development, as a pointer to that. You really can’t work in a vacuum, just toss code over the wall, and hope that it runs somewhere. As we were saying with the failures in early investments and outsourcing, I think those were all built in that regard.

Nowadays, when you start to look at what SOA has done to the industry and how it has impacted IT, it has become an enabling notion that allows companies to employ methodologies like the agile development method, that are more kind in how they view the lifecycle of software and where software runs. It's a partnership with the business and perhaps with outsourced parties that might be hosting that software.

SOA, as a collection of standards, technologies, and ideas -- which is how I look at it -- is already playing a very active role in allowing that pendulum to swing back towards platform-as-a-service, integration-as-a-service, SaaS, infrastructure-as-a-service. It's actually enabling those.

If you look at the vendors that we cover all the time you will see that everybody from VAN vendors like GXS, Otics, and Sterling to the traditional platform vendors are all looking at how they can utilize SOA to better enable outsourced models. SAP, for one, has always offered managed services, but they offer hosted services for their software.

Microsoft is doing the same with their BizTalk Services and with their dynamic CRM, which is tied with BizTalk Services. I think we’re seeing everybody really jumping on this bandwagon.

The best example is one we've already alluded to, Salesforce.com. When I look at them, I see a hosted service that utilizes SOA ideas to facilitate a secured, reliable, manageable, business-savvy domestic environment between your backend systems and their backend systems, which just happen to be on somebody else’s servers.

Gardner: Joe McKendrick, you’ve written recently about Internet Service Bus (ISB). Tell us a little bit about what you mean by that and how that relates to this discussion.

McKendrick: ISB is something that Microsoft has out there. They calls it BizTalk Services, and it’s now in the community technology preview phase, where they’re offering the functionality of their BizTalk -- essentially their ESB -- on an online basis to companies that don’t necessarily want to invest the time, capital, money, or staff to build their own ESB or SOA.

Gardner: So, does this mean that you get the option of hanging your clothes on their clothesline, as it were, on their servers, and then you can coordinate applications or mix and match across this bus? Help me understand conceptually what Microsoft is getting at here.

McKendrick: That’s exactly what they’re hoping to accomplish. You see it happening across the breadth of their product line, with the whole Microsoft Live. You can see them moving to this notion of SaaS, platform-as-a-service. I think they see the handwriting on the wall.

One thing I always say is that Microsoft doesn’t make a move unless it sees a mass market available to pursue. The fact that they’re pursuing the ISB shows that they see it's reaching critical mass in the market. Their natural base is the small- to medium-sized business. It’s not the large corporations that IBM, Oracle, and Sun serve.

Gardner: Or even departments within large organizations, which can actually behave like small- and medium-sized businesses.

McKendrick: Exactly. One model that makes it easy for the business to understand SOA could be an internal SaaS that units of the company are making available to the rest of the enterprise.

What happens is you have this software and it’s something the business can understand. There is a division, maybe at the CIO’s office and maybe somewhere else in the enterprise, that’s deploying these service-oriented applications. The services are available for reuse across the enterprise, across the walls of the enterprise. The enterprise has the option of either picking up these services internally or picking up services externally from another partner. The enterprise becomes both a provider and a consumer of services. It’s two-way outsourcing.

Gardner: There is a tangential trend around ITIL and some of these other initiatives, whereby IT departments take on more of the service bureau or a shared services role, and move from being a cost center towards more of a services provider. Their budgets then become like a P&L. They’re motivated a little bit differently than they would have been as just "Tell us what you need and we’ll figure out how much money we can come up with at the end of year for you" type of approach.

Does anyone else have a reaction to this related area about IT departments shifting in their business models and how that relates to SOA? Todd Biske.

Biske: One of the things that I heard as we went through this conversation, and you hit on it with ITIL and becoming a service provider, is the notion of service management. When we’re looking at why there’s been an outsourcing trend and a backlash against it, a key factor is the degree of service management that’s provided.

It’s one thing to need this capability and go off and find it somewhere, but once you start using that, you realize that there’s this whole degree of transparency that’s needed to understand how that service is being used, where the results are coming from.

In the initial stages of outsourcing there was a lot of people saying, "We want to be a black box and we don’t want to expose any of our dirty laundry, because then our customers may go away." Information needs to be provided to give good service management, as well as to help the customers of those services understand how it’s being used and where they can make incremental improvements on it. It creates a feedback loop.

We can look at virtually anything, whether it’s blogging or podcasting. Before we all got on the call. we were talking about the metrics of how many people are listening to the podcast. By looking at that information we can make things better. We can target audiences or understand exactly who the audience is.

The same thing applies now in the service management domain. I need to have visibility into how the consumers are using the service, where they’re getting good experience, where they’re not, are there trends, and how it’s used. If I’m not getting that information, whether it’s from an internal service provider or from an external service provider, that’s going to start to have an impact over time.

That’s where it leads into culture change -- adopting things like ITIL and understanding service management and what it means to be a service provider. IT is used to building solutions. They put it out there and they go on to the next project. That’s not service management; that’s project management.

Gardner: We’re also seeing discussions -- and we’ve had them on this show, of course -- around governance, where you’re looking at internal governance of services. Then, you’ve got to multiply that in terms of how you would govern across platform as a service and integration as a service.

It raises the question of whether the complexity starts spinning out of control, but we could combine a couple of these observations today. We’ve talked about governance in terms of comparing it to a government, whether you do a democracy, federalized, local, or state. But, if we consider ITIL Service Bureau, we recognize that one of the best governance mechanisms for the most complex things -- business and the economy in general -- is capitalism, competition, or even more crassly, Darwinism.

Suppose we open up services and allow them to compete for the users and/or the lines of business managers to be able to choose among an external service, an internal service, a shared service, a supply chain service within an ecology. Does anyone think that a capitalistic or Darwinistic approach, if this follows its course, will allow for the best form of governance, which is, "Let the best service at the lowest cost win?"

Ward-Dutton: I say yes and no to that. Competition, or the threat of competition, is sometimes a very, very valuable thing to produce the mix. However, you've got to remember that if you’re talking about free market, one of the side effects of a free market is that an awful lot of companies go bust, and that’s fine. If you’re at the top, looking down, you’re just looking at the performance of a market or an economy overall. It’s fine if companies go bust. If it’s your company, though, that’s not so good.

If you’re looking at IT provisioning in this context, you've got to be careful. You've got to put some regulations in place. You don’t want your own internal IT provider spending a heap of money on stuff that isn’t going to be used, not going out of business, but essentially causing business problems, because they’re wasting a lot of money.

So, you’ve got to have layers of governance, i.e, rules and frameworks about how planning gets done, how priorities are set, what’s important, and what’s not. Those kind of things have to be decided, so that a company knows collectively where it wants to go, what it’s trying to do, and what that means for IT. When you’ve got that in place, then in constrained places you can start to drive competition by saying, "In these places, we want to look for competitive solutions, but it’s not always going to be the right thing to do."

Gardner: So, the jungle might work in markets at large, but internally that could be self-destructive. What do you think Brad Shimmin? Do you think that competition has a role or should we have a balance like China, where we have a market, but also central control, and a little bit of socialism thrown in too?

Shimmin: Well, as much as I love socialism I have to say that I’m on the capitalism side of this. I just want to throw out a caveat that if we’re talking about IT departments becoming providers of services, we need to make a distinction between internal provision and external provision, because not every company is a Google that’s going to throw out a Google Maps widget or WSDL, and suddenly everyone is going to be utilizing it, and everyone is going to be happy.

Companies have a better shot at that, however, than doing this internally, because internally you have a lot of federated environments, as we were just saying, with departments acting as their own companies. The whole notion of charging back for services and being accountable for how much you use those services is something that a lot of companies are certainly not willing or prepared to embrace right now. So, the internal adoption of this notion is a long ways off for a lot of companies.

Externally, there are providers and consumers, and the average enterprise is most likely not a provider of services like that. They’re consumers of services from companies, who, as Neil was just saying, hopefully are going to be around for a while. If I'm going to subscribe to one of those services, as a Web service, I want to know a couple of things.

One, I want to know that the company is going to be around a while, because I have an investment in that service, in setting up the interface for it -- for example -- and testing it out, because you don’t just grab a URL WSDL, throw into your designer, and poof, you have a service running. You have to test it and make sure the data models match, etc. So, I want to make sure that my investment is going to be protected.

The second thing is I want to know that the service level agreement (SLA) that I have with that provider is acceptable. If I'm getting the Yahoo! Search WSDL, and I’m limited to 10,000 hits a day, but don’t know that, and I’ve rolled out this enterprise application based on it, I’m going to get some angry phone calls from people around the company when we’ve gone over our limit.

Gardner: So, the notion of how to procure a service responsibly and appropriately becomes very important. That’s a little bit different than governance. IT departments have been historically good at procurement around systems, servers, hardware and software, and support maintenance, but the advancement, up an abstraction, of procuring a service or set of services, would fall more into the role of an SOA architect and not necessarily an IT architect.

Linthicum: Absolutely. The SOA architects ultimately need to figure how these services are consumed and how they have value with their internal systems. I would argue, however, that we are getting more experienced at picking outside-in service providers through the whole SaaS notion. However, they deliver it through a visual interface which is, in essence, a Web site, but it’s becoming much more than that.

Salesforce.com, Amazon, eBay, all those guys, are making tons of money now renting their Web services, and the ability to provide the behavior and the value through Web service is going to be a core component. Businesses are probably more advanced than we think they are in integrating those things into the core systems. It’s at a rudimentary level, but ultimately becomes outside-in services, integration with the SOA, and that’s where the architects come into play.

Gardner: There is a role here for free market influence. That is to say, as this matures and more services from different environments and providers become available, that SOA architect is going to be picking and choosing based on pure market types of forces. That creates the market dynamic, which has an overall governance impact itself. Furthermore, we've also talked on this show about the rationales for SOA. Why should you do it? Why should you do it sooner rather than later? What are the cost benefits? What are the analyses from a business and technology perspective?

For those that are listening, perhaps they get the sense that creating SOA internally, identifying and cultivating SOA architects, will allow them to, in the future, avail themselves of services across a wider panoply of sources and business models, and ultimately they can become the beneficiary of these market forces. This capitalistic approach towards lower costs, higher productivity, and less waste becomes another economic impetus to pursue SOA. What's your reaction to that, Tony?

Baer: I don’t think there's question that this really follows how companies are running their own businesses right now. It's just not possible to become agile and be competitive if you try and make every thing yourself. So, it will be inevitable that you will, at some point, consume external Web services.

I want to get back to the governance point from before. With all the regulatory compliance mandates that we have to deal with at this point, we have to worry about who is handling financial data. Who is handling customer data? Who is protecting identities?

There's no question that a free market is going to emerge for services, but, at the same time, it's going to have to be accompanied by some very heavy doses of transparent governance. I may need to know, for example, my partner’s mechanism for federated trust and authentication. The short of it is, I don’t think you can stop continental drift in this case. You are going to have a free market of services, but you're also going to need a very transparent governance to support that.

Gardner: That's governance at even yet a higher level. That's governance at a market level. In what sort of organization, what authority, under what auspices would that type of governance occur?

Baer: You have to conduct your own governance. It’s a federated model. It’s not going to be any sort of United Nations agency that's going to be top-down type. There probably will be industry best practices that will be drawn up by industry bodies, but ultimately the enforcement has to be at the enterprise level. You have to decide what protections you need and what you expect from your partners?

Gardner: Wouldn’t that federation at the level just above enterprise, or even above supply chains, have to happen under some sort of a brand. We’re already seeing Salesforce.com. We're already seeing Microsoft.

Baer: I thought you were going to go towards something more like the Underwriters' Laboratory model, where we have some sort of external certification agency. Actually, you made a good point where you take a company that's very established and say, "Well, we're going to trust Microsoft, because they're too big to try and risk having a bad reputation."

Gardner: You're more likely to see a business side approach to this than a regulatory side.

Baer: I would agree with you. I don’t see any new top-down agencies parachuting in. It’s going to be, "I'm going to deal with trusted partners. I expect Microsoft is going to be around for a while. I expect that, considering the type of business they’re in, that they’re going to conduct themselves with the highest standards of good governance."

Gardner: Did we go right back to where we had been, which is five or six major suppliers, and "I'm going to be a IBM shop -- or a Microsoft shop -- or an SAP shop," but it’s going to have to do with the ecology of services and not just the packaged applications or platforms? Is this just going to evolve into another instance of major vendors controlling this. Does anybody have a reaction?

Shimmin: Well, I think it will be the same ecosystem we see now, where you have different tiers in the marketplace. Depending upon how important that service is to your business and your application, and whether or not you can withstand a change in that service, of that service going away or not, having the service availability or not, having the governance or the ability to control the data as you would like, then you might be able willing to go with somebody that's smaller than Microsoft.

Gardner: You might be willing to go with perhaps an open-source community that might perk up around a set of services and applications, where the governance is based on community input, rather than just a single entity or large influence?

Shimmin: I don’t see why not.

Gardner: Anybody else have some thoughts about what impact an open-source approach to a services ecology might include?

Ward-Dutton: You’re getting towards a cooperative model in that kind of environment.

Gardner: I guess you would.

Ward-Dutton: You are, if you're talking about a community banding together to make its own of autonomous decisions about who's to be trusted and how things are going to happen, without any kind of patronage from a benevolent dictatorship like a Microsoft or whatever.

Gardner: Or perhaps if the benevolent dictatorships become less benevolent over time.

Ward-Dutton: There is space for all sorts of models. I know the B2B exchange thing in the late '90s was over hyped, but Cognizant is still doing some pretty healthy business in the automotive sector, for example. That's not an open-source model, but it is a kind of industry driven, bottom-up collaborative effort. There are all sort of models, a spectrum of commercial models which can work.

It wouldn’t be open source particularly, but a kind of a non-profit community-driven type of approach could be equally valid. I don’t think it’s likely to be one model winning over any other. There are a whole load of things that are likely to have a role to play.

Shimmin: I feel that the open-source vendors are well positioned to get into this space, because you're buying a service, whether it’s support, installation, custom development, etc. It's a service that you're buying from them. You're not buying software. Companies are becoming much more accustomed to that and comfortable with it. Let’s say you have an open-source Mule version that's hosted within the ESB, Mule ESB hosted, maybe New Source is the company that runs the data center somewhere, and they provide SaaS. There's no reason why they can’t, and it would fit well with their model.

Gardner: I suppose another important trend that is out there now is this whole notion of social networking.

If people have transparency on how services are used and consumed, they should also have transparency to share their experiences with those services. If openness and the ability to create community dialog around whether services are being well provisioned or governed, or if costs and benefits are out of alignment, that this might also provide some grease on the skids of competition and efficiency towards the best solution that the community then adopts or shifts.

Any thoughts about how openness in networking, from a sociological point of view across the network, might keep this in some sort of checks and balances?

Biske: It’s part of any kind of capitalistic approach to this. Customer feedback is just part of the market dynamic, and that certainly can make or break somebody. If you're not factoring that in and incorporating what your customers think in leveraging that information, it’s going to come back to haunt you.

I had another comment on the open-source piece as well. There is still going to be plenty of room for smaller companies, closed-source approaches, in these vertical domains. One of the bloggers that I follow has commented that you don’t see a lot of open-source efforts in niche vertical areas, and we're talking about governance and the impact of regulations.

You don’t see people who, in their spare time, are following the new SEC regulations that are coming out and are trying to offer open advice on how to best meet source regulations.

Those who develop software have always had formal jobs and careers based upon it. It’s also been a hobby for lots of people, and has helped the open-source movement. I don’t know too many people who like to follow regulations as a hobby. So, to have this kind of open-source model to handle governance, the broad horizontal domains will continue to see pushes in that.

In these niche areas, where it requires somebody being paid to understand all of those regulations from the various government agencies, that has to go to a proprietary model. One, it’s just not as interesting, and two, it’s not as big as a marketplace. So, you have to get a level of niche expertise and understand who your customers are going to be.

Gardner: That follows the pattern we’ve seen already with the boundaries between where open source works well and where commercial works even better. Okay, let’s wrap up this interesting discussion we've had about SOA and its role, services, ecologies, governance, competition, and the viral impact of social networks and openness.

Moving on to our next subject, it's a little less global or universal in scope. The IBM acquisition announcement about DataMirror. Brad Shimmin, tell us a little bit about how you view this announcement and how this will or will not strengthen IBM’s hand in information management?

Shimmin: If you look at this from purely a master-data-management, or just data-management, perspective, it makes sense. It’s something they should have done, and they already have a good deal of energy being extended toward that. So, yes, "yippee-skipee," but what gets me going about it is just thinking about how IBM, and every other vendor that has its toes wiggling in the SOA pool, is really looking to build complex event-stream processing and event-driven architectures on top of their solutions. This allows companies to stop thinking about their processes.

Just as you build the process, you throw it out there. If it maxes the server out, that's a bad thing and you go from there. You have your process running. It has interdependencies upon these other processes, these other servers, these other resources, etc. Over time, there are a number of events that take place relative to that process. Companies want to be able to look at that stream of events in context and over time and be able to make decisions. They also want to have automated actions taken upon those. Something like DataMirror gives IBM the opportunity to do that.

Gardner: Is there any question now that having a strong data story is essential to having a strong SOA story for any of these major vendors?

Shimmin: It’s a necessity. Everyone has come to understand that you need to understand the data in the transaction as well. Most of the vendors that we cover in the space either do that or have woken up and smelled the coffee, and are making a lot of efforts in that direction.

Gardner: What about HP and Sun Microsystems? They don’t seem to have quite the same data story. What do you expect to happen there? Let’s go to Joe McKendrick. You track data stuff. Are we getting to a point where HP and Sun are going to be at a disadvantage, or is there actually a benefit for their being a bit more agnostic or neutral vis-à-vis data and data services?

McKendrick: Well, that's a tough one to call, Dana, because they are not actively in the data-management space per se. They don’t have the robust data architecture products that IBM, Oracle, Microsoft have. Sybase as well.

I could see HP emphasizing its service aspect in this regard, really getting into this with its service’s space. Sun is a tough call. It’s sometimes tough to keep track of what Sun is trying and trying not to do.

Gardner: It seems that the software business is a pretty good place to be right now. Looking at the results just this week from SAP, Microsoft, and IBM, software is selling pretty well, and we're not just talking one or two elements, but across the board. Enterprise software is a strong growth business, somewhere between 10 and 12 percent, even after currency fluctuations are taken into account. What do you think, Tony Baer? Is software in SOA important, and is this large set of data capabilities essential? How do you see it shaking out?

Baer: Well, actually I was doing a study at, of all places, a Telco software company. What struck me was that as part of their SOA strategy they also embraced a very heavily federated data model. If you think about it, it’s the perfect compliment to a SOA strategy, because SOA is built on the idea of a loose coupling of all your processes. If you're really going to realize the promise of that, and you have a lot of different data sources, which is the case in most large organizations today, a federated data strategy, which essentially abstracts the data from its source, is the perfect compliment to realizing the promise of SOA.

I'm not surprised to see that the software business is doing very well, and especially companies that are very heavily involved with managing data in some ways are performing. It's not a big surprise.

Gardner: Doesn’t this DataMirror acquisition by IBM point to the fact they want to be more federated. It’s not all going to be a DB2 world. They want to expand on the ability for real-time data management?

Baer: No question. Think about the one key piece that DataMirror brings in there, which is the whole changed-data capture. I can imagine a lot of business scenarios, where that itself could be packaged as a very high in-demand service.

For example, if you're doing any type of market watch or real-time supply-chain tracking, you don't care about all the routine data, when there's no change in the stock or when things are moving on schedule in the supply chain. You want to know when exceptions happen or when things change. I can see this DataMirror capability being packaged into a service that’s not just changed-data capture, but event notification. It fits perfectly in that picture.

Gardner: Neil Ward-Dutton, how about the last word, the role of data within the SOA value? And, how about the competitive landscape question around whether certain vendors might be at a disadvantage for not being too aggressive on the data side?

Ward-Dutton: How long have I got? 30 seconds?

Gardner: 30 seconds.

Ward-Dutton: The role of data is very, very complicated and gives people a lot of headaches. Certainly, this seems to be like a wave-particle duality thing between service and data. If you pursue SOA too simplistically, you just say, "Listen, what's important in the way I design things and architect my environment are services and that's all that matters. That's the primary organizing concept." Then, what happens when you get to data is a whole heap of trouble.

Equally, if you take a very data centric perspective, it can cause problems, when you think about services. When you just think about a service that you're going to reuse in multiple contexts, you have to think quite carefully about implications for the way that data is stored. Clearly, consistently mapping data architecture and SOA architecture is a challenging thing.

What's clear from the adopters I've spoken to, is that you can’t ignore information architecture when you're pursuing an SOA initiative. It’s vital to understand the implications for data architecture and capabilities like federation, changed-data capture, and synchronization of data across boundaries. So, there is a very strong play here.

I wouldn’t go as far as to say that IBM, HP, and Sun are at a disadvantage. I think it was Brad just said, I am not sure, so many people on this call. It’s a cast of stellar participants. Someone was just saying that these guys have never been data-management players, and there are so many things to say about Sun's software strategy, I'm not going to get into it now. HP has managed to survive and is now thriving, but focusing in a completely different area of software delivery. They are much more about management process, quality, and so on, and they are very agnostic in terms of the middleware layer and data layer, and that works very well for them. So, I don’t see why there should be a bigger problem.

Gardner: Right, I suppose there is one thread that we can pull together on the two elements of our discussion today. Perhaps, as you shop around for services, as you look towards the natural-selection benefits of an open ecology in communication, even with services being cheaper, better, faster -- over the wire in some cases -- you still might want to keep your data services very close to your vest. So, perhaps there is a delineation in our discussion for another podcast about whether as we look towards a mixture of in-sourcing, out-sourcing, and multi-sourcing, the data issue is something to keep an intellectual property and ownership vise around.

I want to thank our participants. It's been another interesting discussion. Joining us today have been, Joe McKendrick, Tony Baer, Neil Ward-Dutton, Todd Biske, Dave Linthicum and Brad Shimmin. Thank you all for joining.

This is Dana Gardner, principal analyst at Interarbor Solutions. You've been listening to Volume 22 of BriefingsDirect, SOA Insights Edition. Thanks and come back again.

Listen to the podcast here. Produced as a courtesy of Interarbor Solutions: analysis, consulting and rich new-media content production. If any of our listeners are interested in learning more about BriefingsDirect B2B informational podcasts or to become a sponsor of this or other B2B podcasts, please fill free to contact Interarbor Solutions at 603-528-2435.

Transcript of Dana Gardner’s BriefingsDirect SOA Insights Edition, Vol. 22. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.

Apache Camel Addresses Need for Discrete Infrastructure for Services Mediation and Routing

Edited transcript of BriefingsDirect[tm/sm] podcast with Dana Gardner, recorded July 27, 2007.

Listen to the podcast here. Podcast sponsor: IONA Technologies.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today, a sponsored podcast discussion on a top level Apache Software Foundation project, known as Apache Camel.

It’s a way to help developers do better integration and mediation for routing in software-intense environments, and moving lots of objects and component services around, but also relying on rules and patterns to do so.

To help us understand more about Apache Camel, we’re joined today by James Strachan, technical director of engineering at IONA Technologies. Welcome, James.

James Strachan: Hi.

Gardner: Tell us a little bit about Apache Camel. It seems to be a subset of many other integration and infrastructure projects in the open-source community. Tell us a little bit about the history of how Apache Camel came to be.

Strachan: Earlier this year, Apache Camel grew organically from code and ideas from a bunch of other Apache projects, particularly Apache ActiveMQ and Apache ServiceMix. We found that people wanted to create and use patterns from the "Enterprise Integration Patterns" book in many different scenarios.

Some people wanted to use these patterns inside an Enterprise Service Bus (ESB), some people wanted to use these patterns inside a message broker, and other people wanted to use these patterns inside an application itself or to talk between messaging providers. Still other people wanted to use them inside a Web services framework or some other communication platform. So, rather than tie this routing code to a particular message broker or ESB, we tried to extract this code to be a standalone framework that can be used in any project.

Gardner: I assume that this is to simplify it and make it easier for developers? Is that right?

Strachan: Definitely. What we tried to do with Camel was give it the smallest footprint possible, so that it can be reused anywhere, whether in a servlet, in the Web services stack, inside a full ESB, or a messaging application.

Gardner: And, this is a plain old Java object (POJO), I believe, also built in Java. Is that correct?

Strachan: Absolutely.

Gardner: Tell us a bit about the team behind this and perhaps a little background for yourself?

Strachan: I'm an Apache committer. I’ve been in Apache for over five years now, and I am an Apache member for a few years too. I can’t remember how many. The Camel team is comprised mostly of people from the ActiveMQ team and also from ServiceMix and Apache CXF communities. Plus, a bunch of other people have joined since then.

Gardner: Tell us a little bit about your relationship with Apache. You’re involved with a number of activities there. Is that right?

Strachan: I've been around Apache for many years now, and I am involved in all kinds of different projects -- ActiveMQ, ServiceMix are the big ones -- but also on a bunch of others, like Jakarta Commons, Maven, and many others.

Gardner: There are a number of terms we’ve been bandying around. Let’s assume that not everyone listening is that deeply into technology. When we talk about mediation and patterns, let’s flesh some of these out. Mediation and routing, how is that different from what people might understand generally under enterprise integration?

Strachan: I definitely recommend people read Gregor Hohpe's book "Enterprise Integration Patterns." He offers a really good patterns catalog of how people should do mediation and integration. Rather like the original Gang of Four "Design Patterns" book, which describes low-level programming things, Gregor’s book describes very well how enterprise integration patterns (EIPs) can work and gives us a language for describing them.

Aggregate, re-sequencer, message filter, and message translator generally describe patterns people commonly want to do within an integration solution. One of the simplest patterns is "content-based router," where you want to route messages around your network, based on the content. It might be, for example, that for gold-level customers you want to use the big, shiny, fast machines in your data center, but for bronze-level customers you use the crappy, old, worn Linux box in the corner of the basement.

Gardner: You mentioned earlier ESB, enterprise service bus. Explain to me how this works in terms of other ESBs. This is ESB-agnostic. Is it something that people will use to complement ESBs or perhaps replace them in some instances? What’s the relationship between Camel and an ESB?

Strachan: That’s a good question. In some ways the term "ESB" is a little bit like the terms "object" and "component," in that ESB is used for many different things. It has kind of lost a lot of its meaning. We typically don’t tend to think of Camel as being an ESB. We think of it as a rules-based routing and mediation framework that you may deploy inside an ESB.

We work very closely with the ServiceMix community at Apache which has created a complete Java Business Integration (JBI)-compliant container of JBI components. Camel can be deployed within the ServiceMix ESB among JBI components, but some people don’t use JBI and they may just use Java Message Service (JMS) or they may just use Web services or they may just be JAX-WS clients or whatever. So, we try to make Camel agnostic to technologies. You can use it within patterns like Spring, or JBI or OSGi, or you can use it within any application.

Gardner: I’ve also heard you refer to this as a domain-specific language project. What do you mean by that?

Strachan: Programming languages are often very general purpose. They can be used to solve any kind of problem. Domain-specific languages have become popular lately, and they’re basically tied to typically a business domain, such as investment banking, telco, or whatever. In Camel’s case, we’ve made a domain-specific language that focuses purely on the EIPs.

The language itself describes how to do the various things you typically do in the patterns, such as wire tap, aggregator, content-based router. These are first-class concepts within the language itself. We’ve implemented this domain-specific language in both Java and XML, so you can stick to a Java IDE if you prefer, or, if you’d rather, you can use XML to describe this language and deploy it inside your Spring XML.

Gardner: So, we’ve had vertical industry domain-specific languages. This seems to be more of a functionally domain-specific language. Is that a correct statement?

Strachan: Yes. The EIP is the functional domain, or rather the business domain.

Gardner: And, this is all about getting more granularity, more specialization for integration that nonetheless can play across a more general or horizontal capability. Is that fair?

Strachan: Definitely. Across all of IT we’re seeing increased specialization in many different areas, where the specialization helps us solve a problem at a higher level. If you spend your life solving problems, with, say, JavaBeans everywhere, you have a lot of code to comprehend and understand, and things can get quite complicated. The higher level of abstraction helps us solve problems easily.

Gardner: And, because we’re doing it an open-source environment where there’s a community involved, the more likelihood that this will be applied across many other different types of platforms and technology?

Strachan: Exactly. The other benefit of being with Apache is that we have a very liberal license, so this can be embedded in any commercial product very easily. We've seen lots of partners using this technology.

Gardner: Right. A few moments ago, we talked about ESBs. There has been, as you say, some divergence in the term ESB. There are also a number of products and approaches in the field. How about for Camel as a rules-based routing and mediation engine? Are there many others of those? Who else is trying to solve the same problem, or can we safely say that the Camel is unique?

Strachan: It depends on how wide you cast that net. Camel is unique in a number of ways. What we’re doing with Camel is defining a high-level language to describe EIPs, which I don’t think anybody else has done before. The other thing that’s unique is that this language very closely maps to components that work inside Spring 2.

We use a lot Spring 2 features, such as declarative transactions, inversion of control configuration, and various utility classes for working, with such things as JMS and JDBC and Java Persistence API (JPA). What we’re doing is raising the abstraction level to make things very simple, reducing the amount of XML we have to write, but still exposing the wire-level access if you need to do the really hard stuff and roll your sleeves up and get down and dirty.

Gardner: Let’s move this a bit closer to the business side. I wonder what sort of a problem set it is that we’re facing here. Is this a new approach to older problems or a new approach to newer problems? What's the problem solution fit here in the market?

Strachan: The problem we’re trying to address is the routing and mediation problem, which lots of people have. They're taking data from various components and sources -- whether it’s files, databases, message queues, Web services, instant messaging, or other data systems, integrating them together, formatting them, and connecting them to the systems. From a higher level perspective, this could be for legacy integration of systems, for smart routing, performance, monitoring, or testing or monitoring transaction flows.

In general, integration is a very old problem. As soon as we relate to couple of different computer programs, we already have an integration problem and that just grows with time. The thing that’s new is the approach that’s used. We're increasing the abstraction level by making it very easy to work at the EIP layer.

People don’t have to worry about the low-level details of how to use JMS, how to use JBI, or how to wire together the Spring components correctly, and so forth. We're giving people a nice, simple, high-level abstraction, but yet we are exposing all the power of frameworks like Spring and still exposing the low-level details if you need them.

Gardner: Over the last couple of years, because of the popularity and use of Web services, the business side wants to expand the interactions across not only applications, but organizations, and perhaps in a supply chain setting. It sounds like this helps solve the issue of inclusiveness across domains and businesses, and then adds the opportunity to start moving toward event-driven computing. Does that sound fair?

Strachan: Definitely. And, as soon as people start moving toward distributed systems, service oriented architecture (SOA), event-driven architectures, and all these other terms, the one thing that keeps happening is that people need a pervasive technology they can apply anywhere. Camel is lightweight, and can be used anywhere, in a smart end point, in an ESB, or a message broker.

ActiveMQ5 is coming out in [August, 2007]. That's going to have EIPs integrated in the client and the broker by Camel. ServiceMix has integration with Camel and CXF, as a project at Apache, which implements the JAX-WS standard that’s got EIP integration via Camel. We’re seeing a large number of projects that have EIP integration via Camel. It’s already proven in the open-source world that it’s very easy to reuse this routing and mediation technology anywhere. Hopefully, we'll see this just grow and grow.

Gardner: It’s curious to me. It seems like yesterday, but I suppose we're going on 10 years now, that a lot of these functions -- singular, distributed, perhaps even proprietary -- were combined into what became conceptually the application server. Now, it seems like we’re decoupling things. That one large application server approach might not be the best fit, as we get more types of routing and messaging in these integration patterns.

Can you expand on that? What's been the larger trend, in terms of architecture?

Strachan: There’s been a definite move away from J2EE apps, the big application servers, and people are looking for small, lightweight solutions that solve only the problem they have and not problems they don’t have. A lot of software vendors suffered from the kitchen-sink syndrome, where they tried to have one of everything in a big box. That just led to lots of complexity and redundancy.

People really want small and simple-to-use components that solve the problems they have. I've seen that throughout all of our customers. People ask for very specific solutions. They don’t say, "Give me a SOA." They say, "I need a message router," "I need a message bus," "I need an ESB," or "I need a services framework." Often, people have very specific requirements and are very much cherry-picking the best-of-breed components from the open-source tool set.

Gardner: So, it’s the Swiss Army Knife approach.

Strachan: Exactly.

Gardner: That works well for developers. I wonder over time how Camel might find itself being used in terms of operators, specifiers, and architects on the operational side of things. Do you see this moving in that direction? What’s the short-, medium-, and long-term implementation roadmap that you foresee for Camel?

Strachan: Today it's very much developer-focused. The current Camel is aimed at developers who are savvy with Java, XML, or Spring. What we want then is to gradually raise the abstraction level, so the architects and operational people can monitor services, monitor business processes, perform operational behavior, plus be able to design and deploy EIPs in a simple way.

For example, building tools and making it easier to design EIPs, as well as to choose a monitoring tool, are important.

Gardner: How would the specifying roadmap shake out? Do you think this is something that would get bundled in or integrated? How would this be something easily rationalized in production environments for those people that are going to be looking at large data center implementations?

Is this something that would be part of another grouping of products, other Apache kind of projects? How would this combine to make a whole larger than the sum of the parts?

Strachan: What we’ve seen over the last few years is a huge diversity in how people actually deploy products. Maybe five years ago, people assumed the world was J2EE app servers and everything would deploy in a J2EE deployment unit. These days, we see people make their own deployment units. We're seeing a lot of people move towards OSGi bundles. So we see a huge difference in the ways in which people deploy software now.

One of the reasons we’ve taken extreme steps to split Camel up and make it a very simple and easy to use bundle is that we know that there is a huge range of deployment platforms. Some people want to use Camel within their own smart end points. Some people want to host it inside the message broker, in the ESB and so forth. So, we see it going into production in many guises and many different forms.

Gardner: In IONA’s case, how would this be used in the context of its offering? I'm assuming that, as with other IONA offerings, it will be the commercial open-source support approach.

Strachan: Yes. Camel’s pretty much the core foundation for smart routing and mediation within an entire product suite. We have a product called FUSE, an open-source SOA backplane that provides a services framework, a message broker, an ESB, and a mediation and routing engine. The mediation and routing engine is based very heavily on Apache Camel, and we're using Apache Camel throughout other projects as well.

Gardner: Are there forums or community sites? How are some of the dialog and collaboration around this taking place?

Strachan: If you go to http://open.iona.com, there is a whole raft of documentation online forums, a wiki, and so forth.

Gardner: I know this is fairly early, but maybe this will be a good time to lay out some of the milestones. Are we expecting some significant developments and/or visions within the overall Camel project? Can you give us a bit of a timeline?

Strachan: It’s still fairly early for the timeline, but the next version of Camel 1.2 should be out fairly soon, and it's going to offer improved monitoring and management tools and particularly improved visualization and tooling.

Toward the end of the year, we’re hoping to improve various other capabilities we’ve added recently for extract, transform and load (ETL). It’s a way of loading data into systems and load testing systems with data. Plus, another area we’ve been developing what we call business activity monitoring (BAM), which helps monitor transaction flows across different systems.

For example, you might want to track that a purchase order in a system maps through to an invoice that comes out of another system. You might be processing a 100,000 of those a day, but there is one that goes along every few hours, and you want to find out which one that is within a small time window. That’s another area where we are extending the reach of Camel to do very complex and powerful monitoring.

Gardner: That's interesting, because this provides the actual routing mechanisms that provide a window into what's good, and also what's not good about how those processes are being built and used.

Strachan: Definitely. We’ve found that as soon as you have this very flexible and reusable integration patterns, it’s very easy to build on that, to build higher level abstractions, whether this is powerful testing frameworks, stimulation frameworks, ETL or BAM. We’re seeing the levels of abstraction increase to remove layers and layers of complexity.

Gardner: Is this something that could be applied to the ongoing -- and I imagine increasingly arduous -- task of dealing with semantic issues in terms of data across applications? That is to say, helping to define a mediation and a rules-based approach, such as this help in how companies can deal with that diversity and then come up with more common approaches to data and content that can then be injected into a business process.

Strachan: Definitely. This is a very large and complex topic. There are just a few bullet points on the top, and we’ve already got a range of transformation and validation capabilities in Camel, but we’re looking to improve this in various ways.

IONA recently acquired a company, called C24, which has a whole range of technologies for doing semantic mapping and transformation of different data formats. For example, they've got models for pretty much all the standard financial formats, like SWIFT, FpML, FIX, and all these other things. And, they've got other models for telcos. We’re looking to open source some of that and integrate that into Camel for doing lots of data service type integration.

Another area that’s kind of similar, but slightly different, is in the business activity monitoring ground, where you sometimes want to just check the effect of systems. So, it’s a little bit like I am doing real time reconciliation systems, and you’re doing all this transformation between systems, but you want to make sure of the end results, like that final purchase order or that final settlement instruction does contain the values that you expect. For example, we might want to check that the purchase order amount matches the invoice amount, or whatever.

Gardner: As you say, it’s early in the evaluation of Camel, but are there instances where we can see how it's being used or how developers have identified this as a productivity benefit, or are there metrics of productivity? How is it helping in terms of getting a better job done?

Strachan: We’ve got a bunch of different customers using Camel in different ways. We seem to have a few different kinds of customers finding Camel for the first time. Some come from an ActiveMQ background. They're doing lots of messaging, and they want something a bit more powerful to do more complex routing within the message broker itself.

Some come from the ServiceMix background. They are very ESB focused. They love JBI. They like the idea of standards-based ESB. They’ve got a bunch of standards-based integration components, but they’re working together, and they just want a slightly more powerful routing engine to work with.

Some of our customers have been using Mule. They found Camel a little bit easier, and they’ve migrated across. Finally, there are people who’ve just found Camel, that's all they know, and they just dive straight into it.

One of the things we’re seeing from customers is that using APIs, like JMS and JBI in particular, and to a lesser extent JAX-WS, could often become complicated, but people find that wiring things together with Camel is almost trivial. People get things going really quickly without having to read pages and pages of specifications for things like JMS or JBI, or dealing with the lower-level details of Spring.

We see quite a rapid productivity gain, and it means that people can then use lots of different technologies, whether it’s JMS or JBI without having to spend a long time figuring out things in the Spring manuals.

Gardner: Among these early users are there any vertical industries popping out, where it seems to be making sense first, perhaps financial services?

Strachan: Yes, finance and telcos definitely are big markets, but we’re also fairly strong in the e-tailers. Those areas will be our big three markets right now.

Gardner: Can you explain the business problems that they're using it for?

Strachan: Pretty much all of our customers have similar kinds of problems. They have lots of silos in applications. There are lots of messages being exchanged between systems, whether those are messages via flat files, email, JMS, legacy message brokers, or web services. They generally want to do one or more of the patterns, whether counter-based routing, or message transformations. We see more people starting to investigate a business activity monitoring feature of Camel, and that’s something I'm quite excited about.

Lots of companies, once they realize they can tie together messages to business processes, start running rules off the back of that. That has been extremely useful for people. One thing about SOA that’s quite difficult is knowing what’s really happening in your system.

Today, people have often done such things as monitor message queues or use a JMX-based management console to look at numbers and statistics. Being able to tie together all the different things flowing through your SOA to actual business processes is extremely useful and valuable for customers.

Gardner: There seems to be some murkiness in how feedback will occur, when you have events-driven architectures, and how to create common approaches to runtime and design time with this BAM emphasis that we are seeing associated with Camel. Do you expect that Camel could fill some of that role in terms of a balancing act between what takes place in operations and what needs to be brought into a service level agreement of some kind?

Strachan: Definitely. I have worked with quite a few customers on various BAM-related problems, and it does seem that BAM solves those problems. It’s quite hard to have an off-the-shelf tool to do everything people need. Almost as soon as people start doing any kind of BAM, they find that they want to put their own code in there. You might be in financing, and maybe there is a trade in U.S. dollars. You want to take that trade to the settlement instruction, but you want to do a foreign exchange conversion in there.

Almost as soon as you start doing the “hello world,” BAM-type scenarios, you tend to want to roll your sleeves up and actually do some coding of some kind, to do custom reconciliation, custom calculations, custom visualizations. BAM is very much a framework-driven thing, rather then a black-box tool. Certainly, we see people take on the Camel framework itself and then expand and extend it to do more powerful things.

Gardner: I suppose that’s part and parcel of open source, community-driven projects. They tend to be pushed out in a variety of directions. Do you have any sense of what other functional parts or functional capabilities we might see brought into Camel?

Strachan: A big area is visualization. It’s incredibly hard for people to follow what’s happening in general. Once you can create a wire tap into any system, it’s very easy to glean information from it and find out what’s happening, what its throughput is, what it is actually doing. You can then do things like trace messages around the system, correlate messages to business processes, reverse engineer whole business flows from legacy systems.

That whole area is a visualization and tooling problem. I see that as a very large area of research with the Camel project. We hope to be to be rolling out in Q4 the first batch of visualization tools. It's very much a work in progress in the next few years to get even better, more powerful operational controls and visualization engines.

Gardner: By getting more visual, it seems like we can bring more people into the process, and that maybe elevates us beyond BAM into business intelligence. Is that fair?

Strachan: Definitely. Absolutely.

Gardner: I think a lot of business people would like to get more of a real-time sense of what’s going on among these processes to provide more confidence, as they move processes from manual and older message into this new SOA approach.

Strachan: Definitely. Plus people want early warning notice of any kind of failure. One of the nice things is that you can be alerted that maybe something's going on that hasn't been found yet. Maybe it's running a bit slow today, or maybe it’s just a heavy trading day. But, it might not be, and maybe something is really wrong. So, rather than waiting for some extreme failure that results in a fine or extra interest being paid, you can respond more rapidly than that.

Gardner: Let’s just circle back quickly. Developers seem to be the core short- to medium-term audience for this. What sort of developers specifically should be more aware of what Camel’s capable of?

Strachan: I’d say anyone who’s doing any kind of SOA or distributed programming. They should at least take a good look. Definitely, anybody who ever read the EIP book. If you read that book, you’d probably just see Camel and you’ll immediately just figure out exactly what it’s doing. It’s very simple.

If you're doing Web services, you might need mediation. If you're doing any kind of messaging using multiple databases, if you have any kind of integration problems, and certainly if you’re using any kind of ESB, then definitely take Camel for a ride.

Gardner: Now, when we look toward getting more information on this, we have the resources at Apache Foundation, and also, as you mentioned, http://open.iona.com?

Strachan: Yes.

Gardner: Any other resources? You mentioned the book. Are there any other resources on this that people should be aware of?

Strachan: Those are the big three. There’s also my blog: http://macstrac.blogspot.com/.

Gardner: Very good. Well, we’ve been discussing a budding Apache Foundation project known as Apache Camel. It’s a rules-based routing and mediation engine for POJO-based implementation of EIPs.

Discussing this with us today, we had James Strachan, technical director of engineering at IONA Technologies.

I'm your host and moderator Dana Gardner, principal analyst at Interarbor Solutions. Thanks for listening and learning more about Apache Camel.

Listen to the podcast here. Podcast sponsor: IONA Technologies.

Transcript of Dana Gardner’s BriefingsDirect podcast on Apache Camel. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.