BriefingsDirect SOA Insights Analysts on Citrix, XenSource, HP and Opsware

Edited transcript of weekly BriefingsDirect[TM] SOA Insights Edition podcast, recorded August 13, 2007.

Listen to the podcast here. If you'd like to learn more about BriefingsDirect B2B informational podcasts, or to become a sponsor of this or other B2B podcasts, contact Interarbor Solutions at 603-528-2435.

Dana Gardner: Hello, and welcome to the latest BriefingsDirect, SOA Insights Edition, Volume 24, a weekly discussion and dissection of Services Oriented Architecture (SOA) related news and events with a panel of industry analysts and guests.

I’m your host and moderator, Dana Gardner, principal analyst at Interarbor Solutions, and ZDNet blogger. We are joined by a large stable of experts and analysts this week, and this is the week of August 13, 2007. Let me go through our list of contributors. First I would like to welcome Jim Kobielus, principal analyst at Current Analysis.

Jim Kobielus: Good morning, everyone.

Gardner: We are also joined by Neil Macehiter, research director at Macehiter Ward-Dutton in the U.K. Hello, Neil.

Neil Macehiter: Hello, everyone.

Gardner: Making his debut on our show, Dan Kusnetzky, principal analyst and president of the Kusnetzky Group. Hello, Dan.

Dan Kusnetzky: Good day, everyone.

Gardner: A return guest, Brad Shimmin, principal analyst at Current Analysis.

Brad Shimmin: Greetings, everybody.

Gardner: Also, a regular these days, Todd Biske, enterprise architect at Monsanto, formerly with MomentumSI.

Todd Biske: Hi, everybody!

Gardner: And one additional newcomer, JP Morgenthal, CEO of Avorcor. Welcome to the show.

JP Morgenthal: Hello, everyone.

Gardner: And last, but not least, Tony Baer, principal of onStrategies. Thanks for joining, Tony.

Tony Baer: Good morning, Dana and everybody.

Gardner: We want to jump into two major topics this week, and they are both a result of mergers and acquisitions. First, we're going to discuss something that has further highlighted a hot area, virtualization. That is the acquisition by Citrix of XenSource. We're going to look at virtualization as a result of this $500 million deal at a very high multiple. We're going to look at the implications for Microsoft, for open source, and for virtualization in general.

We're also going to step back a few weeks into July and discuss the $1.6-billion acquisition of Opsware by HP. This is a case where we are looking at management, and an acquisition that might help redefine or expand the role of management, automation, and operations, and perhaps have an impact on SOA as well.

First, let's get the low down on the XenSource and Citrix deal. Why don’t you give us a quick recap on that Tony Baer?

Baer: Well, Citrix has made roughly a $500-million offer to buy XenSource, which is itself a company that was formed around commercializing the Xen hypervisor open-source technology. In fact, it only morphed into a real commercial company within the past 12 months or so, maybe even less.

The interesting thing about XenSource is that it’s been considered to be the leading, emerging alternative to VMware. It essentially virtualizes the machine to a slightly more native approach than VMware. It's a very interesting acquisition because Xen has had a relationship with Microsoft, where it gets access to Microsoft's virtualization technology, and it also fills a key gap for Citrix.

Dana, you and several of the other folks here who were on the call raised a lot of questions in terms of how this impacts the relationship between both companies and Microsoft.

My quick take on it is that in the grand scheme of things, it's not going to make huge changes. Of course, there's a lot of wiggle room here, but my sense is that, when all the debris settles, Microsoft will still need some way of interoperating its hypervisor with the Linux environment. So, even though the relationships may change somewhat in the long run, there will still be some sort of technology sharing here.

Gardner: Let's go to Neil Macehiter. The question for you is how big a deal is this really, and is it a game-changing event, given the high multiple?

Macehiter: I'm not sure one can necessarily equate the high multiple with its being a game-changing event. Clearly, XenSource has about $8 million revenues. We think its paid revenue is $20 million next year. So, it's a monstrous multiple, which indicates that virtualization is very hard, as we have seen with the VMware's IPO. They are now the top publicly traded software company in the world, based on market cap. What this acquisition emphasizes, more than anything else, is where the temperature is rising, and it's not in the core hypervisor.

The hypervisor that Citrix has acquired in Xen is actually open source, and the commitment is to retain that as an open-source initiative. The real value in this acquisition is XenEnterprise v4, which addresses some of the lifecycle management issues around virtual machine infrastructure provisioning, image management, etc. That’s where the temperature is already rising, and it was an acknowledged gap in the Citrix portfolio around desktop virtualization. So, I think it is indicative that virtualization management is hard. I must admit I was quite surprised that Citrix was the acquirer. I was thinking that XenSource was an acquisition target, but I was thinking more of the OS providers, the Red Hats or Novells, acquiring the capability, particularly around the management.

The Microsoft relationship isn't game changing, primarily because Microsoft’s hypervisor technology and the management around it, in Viridian and Virtual Machine Manager, is still a year or so away. It's going to be very interesting to see how the relationship morphs, as Microsoft becomes less dependent by virtue of having it's own hypervisor within Windows Server. Microsoft Research in Cambridge, just down the road for me, was involved in some of the early development at Xen, because that came out of Cambridge university.

Gardner: Okay, Dan Kusnetzky, you write the Virtual Man blog on ZDNet, and you have been covering virtualization for a long time. As a former IDC Group head, tell us a little bit about your perspective on this. From what we saw, from quotes out of Citrix, this deal had been under discussion for a long time. The fact that it came within days of the VMware IPO, which had very great reception, doesn’t seem to be related to this. The timing doesn’t seem to be related, and is somewhat of a coincidence. What's your perspective on this virtualization market?

Kusnetzky: First, I want to expand the notion of virtualization, and to point out it's not anything that’s really new. The model of virtualization that I have been using to examine the market includes seven different layers of technology, each moving some part of an application environment from living in a physical world, to move in a logical or virtual world. If we look at Citrix's strengths, they've always played very well in access virtualization, that is, making it possible for an application to display on a remote device without the application having to know a great deal about what the remote device looks like, the network that it's on, or anything.

They've also got some form of application virtualization, allowing applications to be accessed regardless of where they are, what operating systems they're running on, and the like. They’ve never had a very strong processing virtualization, storage virtualization, network virtualization, security story for the virtualized environment, or management of physical and virtual resources.

If we look at just the idea of what XenSource was doing with their processing virtualization management security, particularly their recent announcement of partnership with Symantec for the Veritas Storage Foundation software to be included in XenEnterprise, you could see that Citrix starts to have a more top-to-bottom virtualization story than they every had before. So, from a product portfolio view, this acquisition appears to make some sense.

From another point of view, though, I have some pretty serious doubts. If we look at the history of technology company acquisitions in the past, corporate culture and management style makes a very big difference in retention of key developers, key architects, and key strategists.

We don’t have to look too far to see some pretty major disasters where a big company acquired a smaller company that had a different approach to business and the key designers left. You could look at IBM acquiring Rohm to get into the telecommunications market, and very short order, all of the people who had made Rohm what it was, went to Northern Telecom. Another example would be Computer Associates acquiring Ingress, and the significant designers and architects moving to Sybase and Oracle, leaving CA with a shell of a company.

Gardner: Alright. So, given that there is some inherent risk in mergers and acquisitions generally, there doesn’t seem to be any obvious cultural symmetry or alignment between these two companies.

Kusnetzky: There are even strong differences. Citrix has been married to Microsoft for years, even though it had technology that allowed it to work with Unix, Solaris in particular, and Linux. They really didn’t focus on that because it would anger the folks in Redmond.

Gardner: Right, and they have managed that relationship very well. It’s been something that I was thinking would not last very long, but it has. Nonetheless, so what gives with this deal? If there are these risks, this lack of symmetry culturally, this high multiple, is this a desperation move, and why the big land grab in the case of this virtualization technology?

Kusnetzky: If we look at Citrix's portfolio, every single piece, service, or product offering is matched by something Microsoft is pushing now. That, in essence, means that Microsoft is trying to acquire the business that Citrix has and slowly remove Citrix from the limelight and off to the sidelines.

They have their own presentation services in the form of terminal services. They have their own clustering. They have their own Go-to-Meeting-like software with the acquisition of another company’s business. Go-to-Meeting was Citrix and Live Meeting is Microsoft. If you go down the list, Citrix is being increasingly pushed off into the corner, and they needed to do something that allowed them to come back and be focused. They needed a broader strategy, one that wasn’t focused solely on access mechanisms. The acquisition of XenSource gives them a broader story.

Gardner: Is Citrix drawing a line in the sand and saying finally, "Okay Microsoft, we played this competition thing for a long time. We've been the little sucker fish on the side of the big shark for a long time, but now we're going open source. We are going to use that strategically, and we are going to go off on our own and try to make real hay on this virtualization thing."

Kusnetzky: That is, at least, a possibility. The majority of Citrix's business is centered on Windows, so they can’t go too far along the path of angering Microsoft, because it would become very difficult for them to get access to key technologies that allow their product to work.

Gardner: Let’s look for a little level set with some practitioners in the field. First, Todd Biske, how interesting is virtualization to you as an architect? Is this something you see off in the future? Is this something you look only for operational efficiency or do you see this as something strategic and perhaps related to SOA?

Biske: Virtualization is definitely something that organizations are looking at right now. For the clients I've worked with, it's been a mix. Some are really trying to embrace it on the server side and make use of it right now. Others are looking at possibly using it on the desktop for developers, when they need to get a specific development environment, but it’s definitely in people’s minds today. So, I would definitely classify it as in the list of strategic initiatives that companies are looking at and determining how to use appropriately.

This is a really interesting acquisition that will help XenSource at least get more mind share in the enterprise. Companies obviously have lots of Microsoft investments on all their desktops. There's a good chance that major enterprises have significant investments in Citrix as well, if they've got any need for remote access for their systems, terminal services, etc. It will open it up to a few more environments to add in this virtualization capability for organizations that were still unsure about what to do with open source. It’s a good thing from that perspective.

Gardner: Citrix is in a lot of major accounts, they're well entrenched, they have a sales force, and they have a presence.

Biske: Absolutely. I compare it to IONA’s acquisition of LogicBlaze. For companies that were looking at open source, having IONA behind that was a good thing, because they've now got a strong name, somebody they can go to for the support around that. Now, you've got the same situation with XenSource. When you have these smaller companies trying to commercialize open source, they still haven’t built the reputation of these larger vendors. So, overall for XenSource it’s a good thing. We’ll have to see whether it really enables them to compete even more with VMware and Microsoft’s coming solution in the space.

Gardner: Let’s go to another practitioner. , tell us a little bit about your company and then quickly how you see this virtualization trend appearing in your client base?

Morgenthal: Thanks. Avorcor is delivering what we call the supply chain as a service. We're really involved heavily in a vertical market SOA around supply-chain management. We've been doing a lot with taking SOA and driving it out in a vertical nature, not just horizontal, where I see a lot of the early work being done. Because of what we're doing, we are working with clients who are facing these issues.

I’ll give you an example where the state of art is, where the state of the decision making is. We have a client that is looking to make a very large enterprise application decision. The company from which they're buying their enterprise application software refuses to commit to virtualization or running on a virtualization architecture. Even though there has been significant evidence of enhanced performance that are running natively on a flat Windows platform in a virtual machine, this company would not commit to running in production in a virtualized architecture. On the other hand, we find that we’d perform better in a virtualized architecture.

We've been dealing with issues of the Microsoft platform for a long time around resource management, where we're fighting with SQL server and other applications or resources, and each one has different memory requirements. This virtualized environment allows us to focus on giving our application 100 percent of the resources, and thereby never running out of things like TCP/IP sockets or having memory thrashing errors slowing down wireless communications, which is critical to Web services doing their job. So, it’s having a profound effect on the production environment.

Gardner: Is virtualization making the operations and architects an offer that they can’t refuse, and is Microsoft going to have to adjust to this reality?

Morgenthal: From a development perspective, that's a given. I, myself, have done what no one expected. I'm actually running on a Macintosh, and that's because I run all my development from my Windows platform stuff in a parallel virtual machine. The reason I do that is because the the MacBook Pro is one of the best Intel machines on the market today. It’s solid. It’s stable. The applications and everything else I do using the Web in my job, run perfectly in that environment, and when I need to do development. So, from a development perspective, it’s a given that people are moving in this direction in busloads.

Gardner: How about the datacenter side?

Morgenthal: On the datacenter side, the promise there is better utilization of resources. As I said, if you really want to get into it, you could find a way to tune Microsoft, but I have a sys admin working in one of my clients who is fighting with this 3GB initialization parameter. When he puts it in one way, one app gets hit when he puts in there. When he doesn’t put it in, other apps get hit, but mine works fine.

This is a clear case of where you go out and get an additional operating system license and put this into each application and in its own VM running on a four-way or eight-way Intel Duo Core 2 machine -- they are running an SAN -- and you have one of the cleanest, most high performing environments I've ever seen.

Gardner: Let's go to Brad Shimmin. Brad, were talking now about the big issue: the high cost of ongoing operations and the interdependency between applications and services within an infrastructure. What’s your take on this virtualization opportunity? Is this really going to be a matter of economics pure and simple?

Shimmin: Absolutely, it's a matter of economics, but not just for the reasons we were talking about. We were mentioning that this is something that helps you better utilize your hardware. This lets you better utilize your software. It basically frees you from having to worry about the stack and the interoperability issues that arise over time in managing a stack.

Gardner: If I can just pause you there and stay on what you just said. SOA is, in effect, heightening this issue, because of the need for discrete services running with their own horses and their own power. What do you think is the relationship between the emergence of virtualization and the accelerating effect that SOA is playing?

Shimmin: You're right there, Dana. There are two levels where this really takes effect. One is, as you were just describing, on a broader level, you need to have interoperability. That interoperability is something that is multilayered and something that you need to be able to standardize over time in a SOA infrastructure and virtualization, once you abstract away all those things that mitigate your ability to do so.

On a more finite level, the part that interests me is the fact that, when you set up a SOA installation, you have a lot of issues you have to resolve initially in terms of, "I'm going to have this version of the operating system, and this version of application server, and this version of USB up the stack."

As we've seen a lot of the vendors, Red Hat and now Novell with their agreement with IBM this week, are trying to say, "Okay, look, we are going to, standardize the stack for you. We're going to tell you that it's definitely something you can count on over time. It's not going to break every time we version a piece of it." What virtualization does is let you set up that verified stack on your server and not have to worry about breaking it down the road, because it's sitting in it's own virtual environment.

Gardner: And doesn’t virtualization also allow you, if you're have a Microsoft shop or you have a lot of Microsoft applications running, to continue to exploit and leverage those investments, but extend them, and without an additional high cost on the infrastructure side of it?

Shimmin: Absolutely. Your lifespan for software is dramatically increased by virtualizing it.

Gardner: Alright. Jim Kobielus, what are we missing here? Is there something that we are not addressing in terms of the impact of 1) this deal, 2) the emphasis that’s placed or the spotlight it's placed on virtualization, and then, 3), how does virtualization affect Microsoft going forward?

Kobielus: I don’t know if we're missing it, but may just overlooking the fact that virtualization is hot this week. Clearly, there have been very important announcements, both XenSource with v4 of its product, the IPO by VMware, and, of course, the acquisition of XenSource by Citrix. That’s hot this week, but really SOA is a virtualization approach.

Virtualization is a long-stroke perspective, as is any approach that further decouples the application logic from the underlying metal. That’s what SOA is all about. Another way of putting that is, abstracting the external calling interface from the underlying implementation of a service or set of services. Virtualization has been around since the dawn of computing. When we stopped programming in machine code and started using FORTRAN, COBOL, as well as compilers and so forth, to put greater distance between the logic and the metal, that’s when virtualization, as an approach, began. It was the big bang of the 1950s and 1960s in computing.

So, in a sense, virtualization is like a cosmic background radiation that still radiates and still manifests itself in complex ways, including, of course, this new focus, this recent focus on hypervisors and so forth.

Cycling back to the importance of this week’s events from Microsoft, what happened this week is good for all three major players. It's good for XenSource, Citrix, and Microsoft. Clearly, it's good for XenSource, because, they've got a tough challenge. They're in a market where their primary competitor, VMware, currently has 85 percent of the market for server virtualization. So, XenSource needed its own mini big bang to give it further momentum to overtake VMware/EMC fairly soon.

Citrix is much larger, more deeply capitalized, richer in R&D and has extensive global sales and marketing, and is a logical suitor for XenSource. It's good for Citrix, because Citrix has several people who have been in the access and presentation of virtualization space for quite some while, and it's well entrenched as a Microsoft partner. So, this allows Citrix, by acquiring XenSource, to put together a complete end-to-end virtualization platform, from storage to processor, to server and access virtualization -- the whole soup to nuts.

Gardner: What about this notion of desktop as a service, where you are not just delivering, as Citrix has done, application interfaces? We're not just talking about the back end, where we are virtualizing instances of a runtime environment, operating system, or stack, but are talking about delivering to end users, the essential ingredients of a operating system desktop with all the associated services and applications that come down in that environment. That's something that might be of great interest to your cable company or telco or someone who is working towards software as a service, but wants to expand on that in to a fuller environment subscription based business model. Does this set Citrix up to move in that direction?

Kobielus: Oh, sure. They've been moving in that direction, of course -- thin-client application virtualization and a multi-client environment focused on mobility. It definitely positions them better. Citrix acquired NetScaler, a year or two ago, and they've got their own MetaFrame Technology. I'm hoping that they will then unify or de-siloize their various virtualization product Citrix upon Xen as their dominant hypervisor, without losing any connectivity or interoperability with Microsoft, on both the access side and on Viridian, as Viridian gets built up.

One of the most powerful things is that Citrix is now the bridge, virtualization-wise, with the open-source community. Clearly, Xen has great traction with Red Hat, Linux SuSE, and, of course, Microsoft, because they have been Microsoft’s primary access virtualization partner for quite sometime.

I want to close the loop on your question, Dana. The XenSource acquisition by Citrix is good for Microsoft, because it allows Microsoft to buy some time until Viridian is ready. A year from now, Microsoft can say, “Oh yeah, we don’t have Viridian ready just yet, but look at this. Two of our primary virtualization partners have gotten together to field an ever more comprehensive virtualization product portfolio, which is integrated or will be integrated fairly tightly with Viridian when that comes out."

So, we'll hear Microsoft saying, “We don’t have it all together today, but we have partners who can give you a fuller virtualization portfolio to compete with EMC/VMware."

Gardner: Hold on. What you're saying is that Microsoft is saying, “We will let our partners go out and to get the beachhead on this business, compete against VMware in the short-term, but then we'll come up and take the whole thing later."

Kobielus: Right. I was surprised that we haven’t seen Microsoft acquire Citrix outright.

Gardner: There's another thing I've seen on the blogosphere is people saying, “This sets Citrix up to be an acquisition target.” It might explain why they went for such a high multiple. They think they can recover some of that from perhaps an acquisition from Microsoft, HP, or maybe even IBM.

Macehiter: Can I just chip in one quick thing here?

Gardner: Certainly. Go ahead.

Macehiter: One interesting element of the acquisition is that historically XenSource has been primarily focused on virtualization in the server infrastructural environment. Citrix has historically focused on virtual desktop, and VMware had it's VDI initiative around that. One question about this is the extent to which the Xen capabilities are harnessed by Citrix to address some limitations of the virtual desktop infrastructure environment, in terms of an integration with things like the Citrix Desktop Server. That becomes the primary focus in terms of offering a virtual desktop infrastructure, the expense of the core backend server infrastructure virtualization.

There's an interesting dynamic there. Although, in principle, they have the soup-to-nuts solution, the question is whether they bundle this as a core capability, which means that they are primarily focusing on virtual desktop, versus virtualization across the page. A key issue is the scalability limitations around things like presentation server, where you just see organizations throwing in more and more servers just to deliver the scalability of presentation server.

Gardner: Let's go back to Tony Baer. Obviously, it's a very complex acquisition in terms of its implications. I think we're still wondering. There's a lot of mud in the water. Microsoft can play this in a number of different ways. Tony, do you have a contrarian view about whether this is good or bad for Microsoft?

Baer: I've also been monitoring a bunch of the blogs, one of the more interesting cases that I have seen is that -- and which actually makes some sense -- is that Citrix is not, at its base, an open-source company. As Dan was talking about before, there are some cultural challenges there. XenSource was an academic project at Cambridge, and was then spun off into a non-rofit foundation like the Eclipse.

That would clear the way for Microsoft to pursue an acquisition here, and make for a nice clean break. I don’t know if that answers your questions, Dana. but my sense here is that this might actually smooth some rough edges around the challenges Microsoft has had in developing Viridian.

Gardner: So, you think that they would maintain a close relationship with Citrix, start begging off some of the technology from the open-source community around XenSource, and use that to bolster its entry into the market, particularly to compete against VMware.

Baer: If you have that clean break, where the open-source stuff is taken care by an outside foundation, that could set some sort of precedent from Microsoft, because Microsoft had been making some initiatives towards the open-source community in the area of interoperability.

Gardner: Microsoft is the only major software company in the world that doesn’t have an open-source benefit that helps it with its R&D, helps it in terms of its entry to market, or it gets community to help it develop around the edges of commercial products. That’s not sustainable, is it?

Baer: Not in the long run, but my sense is that, what Microsoft is trying to do in sort of a life-extension mode is organize these interoperability agreements with folks like JBoss. That type of precedent could happen here with the Xen Technology, and if XenSource under Citrix divests the technology and it's absorbed into an open-source foundation, that would clear the way for Microsoft to open up a much closer relationship with Citrix. Ultimately, Microsoft could acquire them, if Viridian is further down the pike that they think.

Microsoft has had challenges developing this technology over the past few years, and Microsoft hasn't been reluctant in the past to acquire technology. The only thing that might be a game changer is that Microsoft tends to acquire small start-up companies, and Citrix wouldn't fit that mold. But, this does open up some possibilities.

Gardner: Todd Biske, as a practitioner in the field, you're dealing with a lot of environments where there's Windows. There are application costs that are increasing and there’s complexity around SOA creating the need for automation. Do you think that this deal for a buyer in an enterprise makes any sense, and how would it help them?

Biske: If you start getting into the automation space, the HP-Opsware deal is obviously the more interesting one. There's a natural connection between the virtualization space and some of the movements in the management space. Your panel here discussed SOA and virtualization almost a year ago, and I have some comments on my blog about it.

When you really embrace SOA, you're going to wind up with more moving parts for a given solution. And in doing so, you could create this management challenge of how to allocate resources for each of these individual services that have their own life cycle. There's a natural potential to move towards server virtualization to do that, so you can get your arms around that whole management concept. Where I've been disappointed in the management space, however, was that we really haven’t seen anything from the large systems management vendors to start tackling this problem.

Gardner: That’s a great segue, and let's go right into the next subject. We found that SOA will have an accelerating effect on the need for, or the pursuit of, virtualization benefits. The same can probably be said for management, but management is, like virtualization, a large and nebulous topic that encompasses many different things. We’ve talked a lot on this show about the governance and design-time management issues, but how do we start looking towards some automation on the operations side to create some kind of a feedback-loop opportunity for services, how they are used, how they are provisioned, and how they are governed?

So, to this Opsware deal. Opsware was founded and created by Marc Andreessen of Netscape fame. It had an interesting see-saw ride in terms of its market capitalization and then the dot-com boom and bust, hung on through the tough times, and now has been acquired by Hewlett-Packard. Tell us again with a little bit more depth, if you would, Todd, how you see management on the operations side and SOA relating?

Biske: What's most interesting for me is applying SOA to the management space. So, if we are creating lots and lots of services -- you may now have 500 or 1,000 services -- you have to look at that and say, "I have a bigger management problem." There's no reason we can’t take the concepts of SOA and apply them to the management environment.

So, whether it's automated provisioning of solutions, automated policy management, a need to change SOA’s or enable more resources for a particular consumer, there is no reason that I shouldn’t be able to have my management systems call a service to do that. I may want to set up custom orchestrations for how to manage my infrastructure. I may want it all automated out of the box and just push a switch and have it happen.

In order to get there, we've got to have management services on all of our infrastructure, and that’s where there's a huge gap. Everything is still intended to be used by a person. Maybe with some creative scripting, people are able to do it, but you can compare it to the days of Web-enabling mainframes, where the technique was to screen scrape off the green screens. You almost have to do the same thing from the management side now. Look at these user-facing consoles, figure out what glue you can put in front of it to script it, and automate it.

Gardner: Let's go to Brad Shimmin. Brad, are we in an area here where we've got management integration pains, and is HP in a position through acquisitions in it's OpenView legacy to help solve some of that?

Shimmin: They're absolutely in a position to help us with this, both from what Todd was just saying there, and the inverse of what Todd was saying, which I want to talk about, but they have the products and the technologies. The Systinet acquisition that I think is the linchpin here for making this all work together. As you were saying, we’ve seen precious little come out of them and others in the space. The real go-getter here for me is the opposite of what Todd was saying. I would want to see my SOA installations able to speak to and hear from my datacenter systems management solutions. And right now, for that closed loop you were talking about, everything we see is the basic SNMP traps that may get read by Tivoli’s program.

That lets you say, "Okay, there is an alert that one of my servers is overrun on memory. I’ve got these applications running on it, I am going to need to do something, and I see an alert that I can drill down on, and do some basic root-cause analysis." That’s not enough for a true business technology optimization and being able to utilize the resources you are trying to marshal for a SOA environment. I want my Tivoli Application Manager to fully automate that process, look at the variables and the event stream coming from my systems, correlate that, and put it into some sort of context with my applications that are running on it.

Gardner: Morgenthal, in some of the installations that you’ve been working with on SOA, has management already become an issue? That's to say, it’s hard to factor what’s running where and how, and getting a holistic view is next to impossible.

Morgenthal: I had a conversation with Paul Preiss, who heads up IASA, the International Association of Software Architects, about this very thing. He has raised a point and is trying to drive attention towards the exponential growth of SOA, as people start to add services and services become dependent upon services and organizations. I don’t see that problem. One of the differences is that when you come at it from an enterprise IT perspective, and when you come at it from a product perspective, you end up with two different SOAs.

From a product perspective, when you deliver a product as a SOA, you're delivering the architecture, and then you are delivering the implementation of that architecture. Therefore, you have a very controlled instance in which you can manage very easily without needing large governance controls, because you're providing all the infrastructure for management of that SOA as part of your application.

Yes, when you have a lot of legacy infrastructure and legacy investment, and people without the sophisticated SOA architecture experience on staff start developing services, you open yourself up for potential disaster. In those instances, the organization has to ask itself, "Are we ready to invest here?" Every organization obviously wants to take advantage of the latest technologies. This is one of them that can really end up biting them if they are not careful.

So, they need to step back and think about what it takes to invest in SOA and start to wrap their legacy systems and make them available. For one client I worked with -- I love to tell the story, it’s hysterical to me -- we went in, did a big installation, and they knew nothing about Web services. We introduced a lot of the organization to what the Web services can do. Before we knew it, there was a line out the door: "Can I have a Web service. Can I have a Web service?" We had opened up doors to data that had been previously locked tight and made it very difficult for them to do their jobs.

I told the CIO, "You have to think about a security architecture around this," and he agreed. I came back three months later to do a follow-up meeting with him, and I said, "What did you end up choosing?" And they hadn’t. And this is an environment where they don’t even allow you out to your Yahoo Mail. Every Website is locked down tight, but anybody with a notepad, and knows how to write an XML document, can now get whatever data they want out of the mainframe.

So, you see what’s happening. Governance is at the point already where you have a consolidated approach. Prior to that, you need a plan, you need a well thought out approach to what it means to invest in this type of technology and architecture.

Gardner: Neil Macehiter, it looks as if HP is drawing some lines between governance and operational management. Clearly, they’ve got a lot that they can pull together to pull off something like that. In your thinking, is governance more important or is management something we need to address in the context of governance? How does management governance SOA fit in your thinking?

Macehiter: I take a slightly different perspective on the distinction between governance and systems management. I actually think it’s a continuum. Governance is about the way you manage the entire service lifecycle. Historically, there has been this distinction between the governance of design-time processes and the design and development of application solutions, and the governance of the operational environment. Primarily, that’s been because applications have been developed and thrown over the wall.

As we move toward more of a service-oriented approach, the service actually becomes a common element that spans development and operations. That’s one of the key benefits of a service-oriented approach. The implication of that is that you can’t make these broad distinctions. There needs to be continuum. And that’s where things like the registry and the repository become key, because they contain and manage the artifacts that can provide that linkage in terms of things like contracts, and then policies.

There has to be more of a joined perspective. HP, in principle, through its acquisitions, has the assets. For example, when they acquired Talking Blocks that was one the early Web services management players that had assets around management of Web services. Now, they're using it to manage their infrastructure, using Web services. But the Systinet acquisition was part Mercury, which is primarily focused on the design-time. HP really needs to join this up. The other issue is around granularity. When a lot of the management vendors talk about managing a SOA, they're really talking about managing the service-oriented infrastructure, rather than managing the services themselves. So, there is a granularity issue.

Opsware is very good at automating provisioning in the lifecycle, but it’s around the infrastructure that’s running the services, not the services themselves. That’s where the linkage needs to come. I tend to share Todd's views. The vendors in the space -- the BMCs, IBMs, HPs, all of them -- have really missed this, and they’ve been lacking in explaining how they're really going to manage the services, because they're so fine-grained. Historically, managing an instance of an SAP application server is very coarse-grained, and that’s comparatively straightforward. But, when you're talking about disaggregating that and having application components everywhere, you have to disaggregate the way you manage as well.

Gardner: It's a much more complex situation. Tony Baer, in the past, we’ve seen two different directions, when we were looking for a continuum-type benefit. If we're looking for a management and governance benefit around SOA implementations, it seems we can go with one big honking vendor who can do all of this. There are probably only a handful of those. Or, we can look toward standards, and such that many different parts can play together and create a solution approach.

It seems like we’re missing not only, as Neil pointed out, a vendor to step up to the plate and solve this on their own, but we are also certainly missing a management-standards approach that would allow a solution-based, ecology-based, or even an open-source based approach. What do you make of that?

Baer: There’s no question about that. Part of the problem is that at that level, when you start dealing with questions of what defines a service level and a service-level compliance and that starts to get into some higher level areas. When you look at the history of standards, the OSI stack is probably a great example of this with the seven layers. I may get my numbers wrong here, but it was relatively easy to standardize Layers 1 through 4, but when you got to layers 5, 6, or 7, where you get close to the application layer, it gets a lot closer to a lot of the assets that vendors consider to be their value adds.

So, I wouldn’t hold my breath waiting for standards. I have always been concerned about what I wrote about a couple of years back as sort of a blood-brain barrier, which is that you had this conception of runtime governance of SOA, then you had this idea of runtime management, which is essentially more of an infrastructure area. They're handled by two different constituencies, one area being handled by datacenter operators and sys admins, and the other area being handled more by the application folks.

So, when I speak with the AmberPoints of the world they say, "Well, what we deal at service level, we're just dealing with it more in terms of tracking -- is the service level maintained -- but we are not going back to the root cause, which is that a particular data server goes down or something like that." There’s just a huge gap there. I was very disappointed with HP, after they acquired Talking Blocks. I never saw much action there in terms of trying to bring that more into what was then OpenView.

HP has a golden opportunity right now, especially with the way they’ve been running HP software. They reversed the acquisition, where they buy Mercury and then put most of the Mercury execs in charge of strategy, which is a brilliant move. They never had much strategy there before that, but they also were taking management to a much higher level. They need to do provide that sort of unity there. That way that you can get a pass-through, so that, for example, when Mercury Quality Center is recording a problem with the design time, that can trigger a test.

Another area is what I would call IT process automation, which traditionally has been called "run book automation." You might recall Opsware had acquired this company called iConclude, which had that type of product that allows you to automate the workflow of running the data center. This is another way of looking at it. You could call that workflow as a service, and that’s something that is an incredibly open opportunity.

Gardner: And, interestingly, will probably play a huge rule in virtualization, if that scales out.

Baer: Exactly, The other thing that Brad and Neil referred to is the whole idea of governance and governance as a lifecycle. It’s not synonymous with systems management, because that’s just an instance of runtime. There’s the whole design time, change time, and retirement time.

Again, there's a huge possibility for the HPs of the world to knit together a nice federated solution here. There are a lot of opportunities that vendors have not exploited so far, where we could finally start to make runtime governance a real actionable component, as opposed to something in which I just look at a dashboard and then get on the phone to my systems operator.

Gardner: Okay. I hope we haven’t presented more questions than we’ve answered today, but clearly two areas that need a lot of attention over the coming months and years are virtualization and how SOA is a catalyst towards a deeper use of virtualization and more economic benefits from virtualization, and, on the other hand, management, how it relates to governance and how SOA will accelerate the need for more of this continuum benefit. Management on the operations side, governance, services, orchestration, infrastructure of services all need to somehow be manageable.

I’d like to thank our panel for joining. These are topics I’m sure we will be revisiting. We’ve had great insight today from Jim Kobielus, principal analyst of Current Analysis; Neil Macehiter, research director at Macehiter Ward-Dutton; Dan Kusnetzky, principal analyst and president of Kusnetzky Group; Brad Shimmin, principal analyst at Current Analysis; Todd Biske, enterprise architect at MomentumSI; Morgenthal, CEO of Avorcor and Tony Baer, principal at OnStrategies.

I'm Dana Gardner, principal analyst at Interarbor Solutions. Thanks for joining. Come back next time.

Listen to the podcast here.

Produced as a courtesy of Interarbor Solutions: analysis, consulting and rich new-media content production.

If any of our listeners are interested in learning more about BriefingsDirect B2B informational podcasts or to become a sponsor of this or other B2B podcasts, please fill free to contact Interarbor Solutions at 603-528-2435.

Transcript of BriefingsDirect SOA Insights Edition podcast, Vol. 24, on industry mergers and acquisitions. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.

Analysts Debate IT Management and Monitoring Needs for the SOA Era

Edited transcript of SOA management trends and analysis discussion with Interarbor Solution's Dana Gardner and ZapThink's Jason Bloomberg, recorded before a live audience at the Harvard Club of Boston on Sept.14, 2007.

Listen to the podcast here. Sponsor: Tidal Software.

Welcome to a special BriefingsDirect presentation, an IT industry analyst panel podcast created before a live audience at the Harvard Club of Boston. Our sponsored discussion centers on the role of management on Service Oriented Architecture (SOA) use and operations.

Our panelists consist of Jason Bloomberg, managing partner and senior analyst at ZapThink, and Dana Gardner, president and principal analyst at Interarbor Solutions. Moderating the discussion is Martin Milani, chief technology officer at Tidal Software. The in-depth presentations from the analysts are followed by questions from the live audience.

Listen as these SOA experts explore how IT management will evolve in the world of service-based applications. They delve into issues of new standards, how SOA demands that performance management and change management augment and elevate the role of systems management, and how the integrity of services delivery requires a deep and wide approach to management in total across a services lifecycle.

Now, let's hear from our moderator, Tidal Software's CTO Martin Milani.

Martin Milani: Thank you. I guess it's no secret to anyone that SOA has finally arrived, and that SOA deployments are increasing rapidly -- and far more mission critically -- in the past couple of years. It's one of the fastest-growing segments of the software industry as a whole.

So, with that I want to see if the analysts could share some of your thoughts on the industry, and then some of the challenges with SOA in general. Jason?

Jason Bloomberg: Well, thanks a lot, Martin. It’s great to be here. I’d like to start with the definition of SOA for a level-set. SOA is essentially an approach to organizing IT resources to better meet the changing needs of the business. Fundamentally, it’s an architectural approach, a set of best practices, for leveraging IT in a flexible way.

The core business motivation for SOA in most organizations is business agility, to be able to respond to changing business requirements and to leverage change for competitive advantage as well. As a result, one of the key challenges, if you are looking to architect your IT organization in terms of flexible services to meet this business agility benefit, is being able to create, manage and evolve these services over time.

One of the core challenges we’ll be talking about today is loose coupling, where you want to build services that you can control and manage independently of the consumers of those services. That’s a key part of the business agility benefit of SOA. That’s how you would actually achieve business agility in practice.

In turn, the way you achieve loose coupling is through management, and that’s something we’ll be talking about as well. Management then becomes the critical enabler for loose coupling, which is the critical enabler for business agility. That’s how it all fits together.

Milani: Dana?

Dana Gardner: Thanks Martin, and thanks to Tidal Software. I agree with Jason. I think also that SOA has some catalytic implications for companies that begin a journey toward this architecture. It can foster change, and perhaps also benefit agility, but if they have not done their homework, if they are an organization that’s very much in silos, both in technology as well as practices, there could be a lot of risk involved.

There's going to be a period of time, where people look at SOA and find that the opportunities are going to depend on how they’ve done their preparation. We've had a lot of work toward service enablement of data, cleansing data, and putting it into a form in which it can be delivered across multiple applications and processes.

We’ve seen companies begin their journey toward better integration that’s open, that fosters interoperability, and we’ve seen management, but mostly on the level of "speeds and feeds," of how to make sure that the trains run on time, when it comes to delivery of data, whether transactions are going to be fulfilled in a timely manner and if application performance is maintained.

But management for SOA needs to go a step further and take into consideration many systems and interdependencies -- perhaps involving services coming from outside the organizational boundaries, be it partners across the supply chain, even hosting organizations and commercial-services providers.

So management is going to be, as a topic, very important to SOA in a traditional sense, but also management in a new sense. If SOA benefits technology, as well as the business goals and objectives and agility, then business management, IT management, and SOA management need to have some commonality.

Relying on individual people -- "wetware," if you will -- to bind these together and to hand off one to another isn’t going to scale. It’s going to be expensive. So I'm hoping that management is propelled through SOA into a more advanced concept that bridges a gap between organizational management, IT systems management, and the management of services themselves.

Milani: It sounds like SOA is affecting the application assistance management landscape as we know it. Would you agree with that?

Bloomberg: It definitely is. The key thing to keep in mind about SOA in this context is that services are an abstraction. That is, they help to provide flexibility to the business, but they don’t actually simplify the underlying technology. Many architects are a bit surprised by this, that SOA doesn't make their jobs easier or make the job of IT any easier. If anything, it’s more complex. There's more of a challenge for IT to meet the business requirements for flexible, agile, composable, and loosely coupled services. As a result, you have this need for the IT organization to rise to the challenge of services.

This is especially true in the management area because the services essentially have to behave as advertised. These are contracted interfaces to various application functionality of data across the organization. They have to do what they're supposed to do.

That's the core of the loose coupling. So, any consumer can come along and leverage a service, and it does what it is supposed to do. If there is a problem with the service, then it’s not loosely coupled. The core challenge is essentially saying, "Well, we want to make sure these services do what they're supposed to do." That’s fundamentally a management challenge.

Gardner: The performance of applications has been problematic, particularly the transition from mainframe to client-server and then to the Web. Waiting for an application performance issue to crop up and then chasing down the core problem has unfortunately been the norm. I'm not sure that that’s going to continue to be possible.

It’s never been the preferred way -- firefighting as the means to application performance. When you take this to the step of decoupled applications or services, you recognize that so many different systems are now supporting processes. It's not just a single model of a stack of support and platform beneath a specific application. You need to start working toward a predictive, analytical approach to the management of performance.

The implications for those dealing with applications is that you are going to service-enable those applications, decouple, and decompose them into essential core services, and then repurpose them by cross-compositing processes. What is that going to do to you, if you think you are going to go to firefighting mode when you have performance issues? It’s simply not going to work.

You need to rethink management and support, and you need to try to get proactive in how systems will be supported to head off performance issues and create insurance policies against blackouts, brownouts or other snafus. SOA is really a catalyst toward a different approach to the management and support of the services.

Bloomberg: That’s an important point. In the context of SOA, we're thinking in terms of business processes that are implemented by composing services. What’s happening here is that the definition of an application is beginning to shift, especially from the perspective of the business.

The business doesn’t want to think of an application as some big huge piece of software that costs them a lot of money and introduces a lot of risk, that doesn’t directly meet business needs, that’s not what they are asking for, but is what they’ve had to buy because it's what was available.

In the context of SOA, though, an application is what you do with the services. You compose them into composite applications or service-oriented business applications that implement processes in a flexible way to meet the needs of the business as they continue to evolve.

From the business perspective, this notion of application is whatever you are applying the services to. The root word of "application" is to apply something, and, as a result, the management challenge, especially the application management challenge, in particular, is an entirely different set of challenges.

It’s not saying, "Well I have this big enterprise application, which is some monolithic suite. I need to manage that." If you have those, you still need to manage those. So, that’s not going away, but SOA introduces this whole new concept of flexible, composable business process-based applications, which now have to be managed as compositions of services.

Services are obstructing functionality across various systems and various applications, and the management challenge becomes immensely more complex. If anything stops working, then the whole thing falls apart. Management now becomes even more important. You can’t just have a firefighting approach, and when there's a problem, you send an alert to the poor systems administrator, who has to wake up in the middle of the night to fix it. That just not going to cut it anymore.

Gardner: One more factor in this is that we are not just talking about SOA in a vacuum. We're not just moving toward SOA or services-enablement. We are also dealing with application modernization, pulling them off older more expensive platforms and putting them on standards-based or commodity platforms. We're talking about virtualization where we are going to create containers and try to get higher utilization and efficiency rates off of our underlying investment for the support of these services.

We're talking also about business continuity issues where we are going to try to have replication of services in the event of disruptions, be it natural disasters or otherwise.

So when you think about SOA, it’s happening at a time when there are many other IT mega trends under way, and the management needs to be considered within that context as well.

Milani: Obviously, SOA is a disruptive event. There is a radical change in the architecture of the applications, as we know them. There are far more points of failure, far more pieces of the application that could reside across separate systems, separate technologies and perhaps across enterprise silos.

So the traditional management approach has been used for the past 15-20 years, and is very client-server centric. Can these traditional monitoring and management approaches handle the SOA deployments of today and tomorrow? Dana?

Gardner: You're not going to throw out the baby with the bath water. You're going to continue to leverage your previous investments. Just as we’ve seen with storage and application deployment strategies, it’s about elevating to a higher abstraction -- perhaps through metadata, perhaps through standards -- so that you can increase outputs qualitatively and quantitatively from your systems.

Therefore, you can offer a management overview, whether it’s a dashboard, or even the equivalent of screen scraping one management view with another. At least, you can assimilate them in a such way that you can start to get a total view, a comprehensive view of systems.

I do think that you are going to continue to leverage existing management approaches. I'm sure that the older-line management vendors are going to continue to augment and support more advanced processes or more complex interactions between elements of infrastructure and applications.

It needs to be looked at not just as management of discrete parts, not just trees within the forest that each stand on their own -- but the forest itself. I'd like to see that get to the point where it becomes something that can be assimilated further than just the systems -- with the business objectives as well.

Furthermore, one of the things that’s been unfortunate about systems management up until now is that it tends to be binary -- on or off, green- or red-light, it’s working or not working. SOA is going to require more of a blended approach to management.

That is to say, you are going to want to tune how your applications and services are delivered, perhaps to live up to service level agreements, or perhaps so that you can give priority to certain data, application, or services streams over others. You're going want to be able to fine tune, rather than just throw a switch on or off. That’s going to require a different level of management. It’s really about leveraging the old, finding ways to assimilate and then put a more operator- or policy-driven -- perhaps even automated -- approach on top of it.

Bloomberg: It's interesting that you say SOA is disruptive, because SOA is disruptive -- but not in the way you might think. It’s not really that disruptive on the technology side. In fact, from the technology perspective, SOA helps leverage existing technologies. It has a heterogeneity story that says, "Well, you don’t have to replace. You can leave and abstract as needed."

So, if SOA is architected properly, it doesn’t have to be disruptive on the technology side, but it is most definitely disruptive on the organizational and political-cultural side, on the human side, because what SOA tells the IT shop in particular, is that we can’t do business as usual.

We can’t simply have a siloed approach -- here are the operations people that run this, here are the Windows people that do this, here are the application people that do this. That’s not going to work anymore. In order to think about services properly and to get the benefit of SOA at all, you have to think in a more cross-functional, comprehensive, architected, best-practices way about how IT does what it's supposed to do. How IT is going to meet with the needs of the business is shifting in a broad way, and that’s how SOA is disruptive.

When it comes time to think about management at all levels -- whether it’s networks, systems, application management or the business service management -- these are being reworked, because you can’t think of any of them in the narrow silo.

It’s like saying, "Well, I have this application; I have to manage this application. I have my TCP/IP network; I have to manage the network." You still have those technologies and you still have to manage them, but now we also have this whole notion of services cutting across different parts of the IT shop, meeting different needs of different parts of the business.

A single service might abstract multiple databases, multiple underlying applications, and multiple platforms. That same service might serve multiple lines of business and might serve the needs of internal as well as external users. So, in that context, what it means to do management is being entirely disrupted by the service-oriented approach.

Milani: Traditionally, the way people have looked at the Web services in SOA management today, has been to manage or monitor the interfaces, which are actually an interface to a runtime environment that holds some sort of service.

What do you think is required to deeply understand how to monitor these services beyond what the interfaces are doing, and finding out what’s under the tip of the iceberg? Dana?

Gardner: Well, with SOA you need to gather information about your systems both deeply and broadly: deep and wide. You can already get a fire-hose of data from your systems, log files, and agent- and agentless-based approaches already on the market. You get a ton of data.

It’s working with that data in the context of a horizontal business process that’s the hard part. The type of applications that we’re going to be seeing in the future, as Jason mentioned, cutting across different aspects of the organization, are going to require redundancy. If one aspect of a process goes down, that’s the weak link in that chain, the whole chain could be at disadvantage.

In the past, you might have one application down, but people could go off and do another task, because that mainframe would be backup at two o’clock. You can live with that. If your entire supply chain is disabled for a period of time, that’s a higher price to be paid. So, we're looking at a different level, and I don’t think we’ve seen the solution yet.

Bloomberg: It’s important to note that in the context of SOA, monitoring is just not good enough anymore. It’s not good enough to simply say, "We need to find out when we have a problem." It’s like, "Great, okay, now we know that Titanic has sunk." That’s not going to do it for you. You need to have preventative and corrective management. For a service to be loosely coupled, it has to behave the way it’s supposed to behave, regardless of what’s going on beneath the cover.

So it’s not good enough simply to say, "My management tool is telling me my service is down." You need to have a way of understanding that problems are brewing, that they are in the works. You have to be able to identify a potential problem before it actually impacts the behavior of the service and you need to take corrective action, ideally before the consumers of the service are impacted.

This is a high bar to set, an enormous challenge, and it’s not likely anybody is going to be able to do this perfectly. There is nothing perfect in this world and there will always be instances where there are some problems that a preventative, active management tool won’t be able to catch, but it’s critical for the architect to plan for this level of management, preventative corrective management.

That’s something the architect has to think of ahead of time, when they plan how they are going to build and implement those services. You can’t let it go for later. It’s not like you say, "We're going to launch our software and then let operations deal with management." There has to be something that’s part of your initial thinking when you are putting together the plan for your SOA.

Gardner: Fortunately, the way we’ve seen SOA evolve in the marketplace has been more on the crawl, walk, run basis. People aren't going out there saying, "We're going to switch over to SOA on Monday and all systems will be services." That’s just not the way.

There is an opportunity to learn as you go to encounter problems and then to put into place the management feedback and create the feedback loop into the remediation. Another important aspect of this is to start finding commonality between pre-production and post-production when it comes to applications and services development.

We're beginning to see some products in the market that try to take data that’s collected in the development phase and make it available to post-production to start feeding a loop of communication. So, when something is amiss in production, you can not only look to what is at issue in the systems and operation, but also look to how could we make this service or application better. So, the requirements for service and application are being impacted by operations.

For a long time, there was a significant wall between these facets of IT. It’s still very problematic, but if we can create a management feedback loop, so that -- as we get into faster iterations of development, when the test-debug cycle and build cycle is faster due to agile and lean practices -- we can start saying, "Let’s find out what’s going on in the field when something is wrong."

Do we just throw more hardware at it? Do we just add more servers? Or, do we say, "Can we actually design and engineer the service to be better," and then make that happen with a matter of weeks or months?

Milani: It sounds like the playing field again is far more complicated and complex with SOA, and is going to get more complicated from here on. You have business processes that go across multiple technology silos, multiple enterprise silos, and probably to other enterprises. So, you have different constituents, different organizations, different groups, and many different types of technologies.

This area was pretty hard to monitor and manage in even a simple three-tier architecture. Now, you have an N-tier architecture which you could almost call an "N-to-the-N-tier" architecture. Now, you have different types of services that you could be calling on, depending on the quality of service and security, trust and policy, and so on. So, it’s a different game. If you were to fast-forward five years or seven years, what do you think monitoring a management system, which could address all the issues we just talked about, would look like?

Bloomberg: It’s important to understand that management is not a practice in isolation. It’s really part of a family of capabilities that includes governance and quality, as well. You mentioned policy, security, and a variety of issues. All these are part of the SOA infrastructure story. So, to answer the question where things are going over the next several years, it’s really maturing the family of SOA infrastructure capabilities, broadly speaking.

We like to call that the governance quality management, or GQM, suite, which handles design time as well as runtime issues, creation and communication of policies in the context of governance, management in the context of runtime. It's not just thinking of runtime, in and of itself, because runtime is only part of the story. We have the full lifecycle now with design time creation of services, as well as publication and discovery of services.

The runtime part, which is the current focus of management, as well as the change time part, where you reconfigure and recompose services essentially in runtime context without any underlying co-changes, that’s where we see a lot of the activity going on in a mature SOA environment, is at that change-time configuration, composition level. So more and more of what has to happen there, from the infrastructure perspective, is this pulling together of the quality management in governance roles into full lifecycle quality management governance suite and that’s what we see happening already today, with a lot of the products in the space.

Gardner: In five to seven years, we're going to continue to see an increase in the pressure on businesses to adapt in markets. We’ve got globalization well under way and it’s going to continue. We have lower barriers of entry to companies, where digital assets are made available. We are going much more from bricks to clicks. Therefore, companies need to adapt readily and they can’t go to their IT departments and say, “We need to change; how long will it take?” They need to say, “Here’s the change; implement it.”

Let’s take an example of something from recent business history. Let’s say you’re a manufacturer of toys and you're told that a portion of these toys are now in the field with toxic paint that is an violation of the regulations in your country. You need to take quick action. You need to find out through your supply chain where the problem is. You need to find new suppliers. You need to go back out to the field and conduct a recall, and coordinate this with your marketing and your PR department and your investor relations department, so that you don’t lose the entire value of your company overnight.

How are you going to do this? You might do this through your IT systems. So you’re going to need to examine what’s going on in your supply chain and find out where the problem is and stop it. You’re going to say, “We don’t want to just keep the trains running on time. We want to pick up the tracks and move them somewhere else.”

IT is going to have a great opportunity to become far more valuable to their parent organizations, to be the real partnership that they should be, through the exploitation of SOA and through the proper management of IT and processes.

To answer your question: The value of IT here can be much greater. It can be an enabler, not a cost center. It can be the way in which not only is information relayed about what’s going on, but can determine what we want to happen. We want to change that supply chain. We want to change that distribution, recall these products, get a list of every single product and every serial number, and we want to relay that to our sales force.

That sounds straightforward, but if you try to do that with a lot of IT systems today, you’re going to find yourself up there with the equivalent of mimeograph and crayons, doing it by hand -- and that’s just not acceptable. So in the future, a company’s very existence could be at stake if they don’t have agility in these processes.

Bloomberg: This is a really important point. SOA is really not optional. Companies that don’t get this right will suffer the consequences. They will suffer lawsuits and suffer a competitive disadvantage. They are going to go out of business. This is an important thing to keep in mind. IT is not playing around here. You can't say, "Maybe we will do SOA, if we can figure it out, or maybe we won’t. We’ll just do things the old way, where we are siloed and we keep on going."

If you keep doing things the old way, you are going to lose out one way or the other, whether it’s some sort of regulatory or competitive pressures, some disaster like lead paint, or credit card numbers getting out on the Internet. Whatever the problem is, it’s going to happen, and you have to be prepared. Organizations that don’t get this right are not going to survive.

Milani: SOA, again, is very business-driven. One could say it’s totally business-driven. It gives business agility and adaptability. Often people call it agile enterprise, real time enterprise. Inherently, SOA means real time integration across separate applications and separate technologies.

Something we touched on a little earlier was preventative automation and correction. Can you guys elaborate on that a little bit, why it’s very important. If you’re talking about an agile, active enterprise or real-time enterprise, then that automatically means that the management cannot be reactive and the management cannot be an afterthought. Dana?

Gardner: I recently read a book on SOA that I found very useful. Dr. Paul Brown is the author, and it’s called, Succeeding with SOA: Realizing Business Value Through Total Architecture. A lot of the book has to do with this concept of "Total Architecture."

It’s the architecture of the business and the architecture of IT having some relationship to one another. We can borrow from this concept and say that this should be a "total management" approach as well, so that we have this ability to manage processes and infiltrate the systems in such a way that it becomes a two-way street.

Preemption is really about latency. You can be reactive, if you can do it in a nanosecond. But, preemptive means that you’re going to come to a time where the lights are blinking and there is something wrong, and, before you’re totally out of business or some process is brought to its knees, you want to take remediation. It’s really about balancing the latency of reaction toward a point of preemption. That's very hard to do, however.

When we have system redundancy, we have all this log data. There are probabilistic approaches. There is looking at performance states that are normal, when you compare that over time, but when we get into this area, there are many unknowns, many more variables across the supply chain. When we are dealing with services that are coming from organizations that you don’t have authority or control over, it’s going to be a difficult approach.

I don’t have a quick answer to it. I do think that the latency issue needs to be addressed -- the amount of information that's shared. As we get toward this concept of total management, we need to bring in what incentives are being applied to people and systems.

Are we going to pay for systems based on a steady stream, are we going to start putting in incentives that penalize people for down performance, while paying them higher prices for greater performance? As we see services that are acquired on a subscription basis or a pay-as-you-drink basis, we are also going to start seeing a lot more monetized incentives around performance.

So whether the systems react or not, the price could be so high that you’re going to have to make the investments. I think economics and the concept of total management need to be brought to this.

Bloomberg: There is no magic wand here. Systems aren’t perfect. Software is never a 100 percent bug free, computers are never 100 percent operational, and networks are never up 100 percent of the time. There are always problems. There are problems on the business side as well. Secrets sometimes do get out, and products sometimes are defective and whatever it is, there are always problems. This is just the way of the world, the reality of life on earth.

How do businesses deals with this traditionally? Well, you hope there aren’t any problems, and you have certain plans in place, but fundamentally, when there are problems, you step away from your technology and you deal with it however the people can deal with it. So, you run around and try to fix the problem.

We are looking at SOA, saying, "We can do a bit better with our technology in terms of dealing with the reality that there will be problems." As far as agility, we want to be able to respond to change, including changes we don’t want. That include problems with systems, with the business, with regulation, competition, or global disasters -- whatever it is.

Instead of just saying, "We can’t rely upon our technology, when problems come along," we want to say, "Yes, we have a more flexible way of dealing with problems, even though we can’t predict what they are.” That’s part of the benefit of SOA, part of the agility benefit. We are dealing with unexpected change, including various problems.

Instead of just running around and not being able to use technology, we want to have a governance plan in place, saying “This is how we’re dealing with problems. Here is how the technology will rise to these challenges." And, we make it a matter of policy. So now, instead of just having to wing it when you have some sort of issue, there is an infrastructure in place for helping you deal with issues as they come about.

A key to this is the management challenge. As management technology improves, it is less and less about just monitoring stuff, and more and more about being able to deal with issues as a matter of policy, where your policy is in place for dealing with problems that you can’t predict -- and those are the most challenging ones. That’s what we see happening over time.

Technology is getting better at dealing with these unpredictable situations. A core part of what we need from agile IT is being able to deal with the unpredictable.

Milani: Great point. It dovetails to my next point, which is businesses are moving to standard policies and standard practices in the form of a business process. More and more, they are moving the way they do business into a well-defined practice of business processes. So, using technology such as BPEL and business process management systems, why wouldn’t the management of some of these technology and software resources look like how processes are handled on the business side of the equation?

Bloomberg: Actually, they might be. You want to think about business process in the context of whatever the business does, and that applies to IT as well as it does to the business. IT should be run as a business, as part of the business. So, it’s not like IT is some different world. It’s just part of the business, one of many divisions, and it has a certain role within the context of the business.

Management, to the business side, means a whole range of things, of which technology is only a portion. It means making sure that you have a management hierarchy within the organization. It means that your business is running properly and that the business process is running properly. All of this is part of what the business means by management.

Part of that is IT enabled and part of that is specifically IT centric, but from the business perspective, it’s not like they are drawing a line and saying, "This is the stuff we mean by management in the business context, and this is the stuff we mean by management in IT context," as though they were different things. There is a spectrum, and as we move to SOA, we’re seeing that IT is becoming more of an enabler of the business in many more flexible ways. So, the term "management" now becomes much more of a business-centric context, of which IT is now an enabler, as opposed to two different kinds of management, one for the business and one for IT.

Gardner: If you could meaningfully and successfully divorce IT from the organization, you would have seen a lot more outsourcing. Companies tried to outsource lots of aspects of their IT. Some are successful, but in many cases it was brought back in.

IT and business are intrinsically bound. It’s a competitive differentiator: The way in which you do IT better than your competitors. I don’t think we’re going to see a divorce here. We’re going to see more assimilation. But the management of IT is relatively new, compared to the practice of business management.

The modern corporation is over 100 years old. Mercantile economic activities are 600 years old. You can find other aspects even older than that as to how people are governed and operate as teams or individuals.

If we have things like Balance Scorecard, Six Sigma, Design for Manufacturability, Simultaneous Engineering -- these concept have evolved on the business side, and on the manufacturing side. We should expect to see the same aspects for IT, and perhaps have the same overview of a Balanced Scorecard on how the business is operating, as well as how the IT is operating -- someday, maybe even together.

What we're really talking about here is the maturation and the natural process for IT to become more like business, and not be off in its own corner. It takes time. These were complex things that happened very, very quickly. We used to call it Internet Time. It’s probably, in some respects, even faster now, when you think about Enterprise 2.0 Time.

These IT systems have evolved so quickly, and companies have implemented them in such a haphazard way with lots of different heterogeneity involved, that it’s natural that it’s going to take time. Ultimately, we're going to get to a maturation where IT catches up to the business. Then they can be governed very similarly.

Milani: Are there any standards required sooner than later? Is there any specific standard that you think can enable the deployment of these SOA environments?

Bloomberg: There are standards in the works; there is Web Services Distributed Management (WSDM) and WS-Management and a variety of others. There are some that are datacenter centric, others that are more network centric. But, by and large, I agree with Dana that this is still the early days in terms of management standards.

What’s happening in the management standards world is a pissing match between the big vendors. You have the Java guys wanting to this and then Microsoft guys wanting to do that, and nobody listens to them, because they can't agree with each other. So, they'll realize, "Hey, all of our customers are ignoring us. We'd better get our act together." It’s become this big political thing that’s just slowing whole thing down.

From the enterprise perspective, you don’t have to wait around for the vendors to grow up. You can get stuff done today. This isn’t going to stop you from being successful with SOA initiatives today. It might mean that two products you buy off the shelf might not interoperate as well you like. That has to be part of your plan. It might mean you have to come up with your own internal standards for the time being.

Lots of organizations are doing that as well, because the open standards are just not mature enough to do everything you wanted to do. But that doesn’t mean you can’t be successful. Interoperability is part of the story, but a lot of what you need to get SOA to work are other challenges that you can work on. You can get SOA business value.

Gardner: We need to see standards at a higher level. It's not just about systems level, and it’s not just about framework level, like Java or .NET. It’s really at the level of how the people who are responsible for a business process get a view into that entire process.

Ultimately in these organizations, we're not going to have people responsible for a server farm, a database, or an SAP implementation. They're going to be responsible for a order-to-bill capability. For those people to get a view into all of the aspects for which they are responsible, they're going to need a management view at a higher elevation. That’s where standards need to be applied, so that they can get that view, so that there is interoperability, sharing of data, and a canonical view of management.

Milani: We're almost out of time. I want to thank you for your time and quickly ask you if you want to close and sum up what you think people should consider, as they deploy more and more SOA implementations, and what they should consider with respect to management.

Bloomberg: SOA is not just about the technology that you can't buy from a vendor. It’s not something you buy. It’s something you do. It’s a set of best practices that are still relatively loosely defined.

You can be successful with SOA by taking it a step at a time, achieving real business value. You don’t have to set the bar so high that it becomes impossible. But, that being said, even if your SOA initiative is relatively modest and you have a few services in the context of that architecture, management is something that you have to think about very early on. In fact, you need to think about management with services even before you get to SOA.

If you just have a few services and they are not managed, then they could be redundant or they could be incompatible. Worst of all, they could be rogue services, services that are not on the radar of anybody who is running these services. They could expose confidential information or bring systems down. Who knows what the problem is?

A rogue service is one that you just have no idea what sort of damage it can cause. This is a real risk, because, if you’re building a Web service, in particular, it might just run over Port 80. It might just expose information over the firewall, and you have to think about this very early on in your services initiative, even before you get SOA.

As you get SOA, you have to think about all the services you already have and plan for loosely coupled services in the context of your architecture. All of that has to be done in the context of management, as well as governance and quality as well.

Gardner: As companies move closer to SOA, it forces them to grow up. It forces them to think across boundaries. In the past, complexity has forced companies to divvy up issues into small compartments, put a box around them, and assign people to that item of complexity.

But that has stifled the ability of interoperability and of addressing things holistically, of being fleet and agile. It’s made them brittle, has made them slower, and has made things expensive. SOA forces companies to start binding what happens in pre-production to post-production, what happens in an application with what happens in an infrastructure, and what happens on a service level from an outside provider to what happens as a shared service internally.

There are great risks if you try to do SOA without growing up, but there are super opportunities if it’s done properly. It can elevate IT as a function within the organization from being an inhibitor to the absolute enablement for new business and growth and opportunity.

If you’re inside an IT department, if you are a vendor, if you’re a consultant -- consider that if you do the diligence, if you come up some standardization, it’s the methodological approaches that work. SOA will make the company better and will make IT indispensable in the best possible way.

Rod Butters: I want to thank all of you for the discussion. It’s actually ranged all the way from the value of SOA to the business and the kinds of business opportunities that enables how management keys into that and feeds into that from a technology standpoint, and the interesting ideas you shared today about how management ties with the actual SOA architecture to provide agility to the business.

So, with that, thanks very much for this great discussion today and thanks to everyone here who has attended our live debate and discussion on SOA management. At this point, we’re going to open this up for questions.

Question: I'm from Bank of America, and I was really glad the way you guys started to talk about the management top-down approach to addressing the changes that the organization will face. You also talked about agility and business demands and bringing IT management bit closer to the business management side. Business, for all intents and purposes, doesn’t really care what’s underneath the covers, right?

They have a business value proposition, they have business ideas and perhaps a very cool looking interface, if one may just look at the Internet only, a Web 2.0 experience. For them, agility and latency has a lot more meaning than what we are talking about with respect to SOA. With so much organizational change, there’s going to be some time before these things are going to get settled. How do we really achieve agility in terms of business demands and SOA, which is lot more organizationally heavy it seems? That’s one question.

Second, there is one notion of introducing and implementing SOA, which is going to take some time, and we are talking about management needs to get together into a common consensus of implementing SOA in a most efficient fashion. Now again, talking about agility in terms of maintenance and regular changes to the environment, from the business perspective, they want change to be implemented faster.

The owners define pretty much for the entire organization enterprise wide. How does business still achieve agility into their maintenance mode, when they want to make small change to an application which is now going to require a lot more changes on the back-end side?

Bloomberg: You’re quite right; the business doesn’t have visibility into the inner workings of things, but they also don’t want to know. They want SOA stuff to work the way it’s supposed to work. They will have a change. They want it to happen. They don’t want to hear about any problems. They don't want to hear a bunch of tech-speak in response to some request that they have. That’s one of the core challenges of SOA, thinking about services in the context of the business. It’s a role of IT to build and support business services that abstract the underlying technical complexity to provide the business with that flexibility that they need.

If you can achieve this with individual services, then you can achieve this with the compositions of services. You have to build services to be composable. You’re trying to enable the business to build and evolve business processes by composing, recomposing, and reconfiguring services. If you do this right, if you get the architecture and the implementation correct -- it's a big “if” because it’s a challenge -- then you can build big services out of little ones, because you can compose services into composite applications and expose that as a service as well. So, this big exposed composition of services as a service gives you two core things.

One, it’s recursive, if I can build big ones out of little ones. Second, if you were to see some sort of business service, you don’t know if it’s actually abstracting a process or not. So, in telco, you could have a user-provisioning service, where it’s actually in multiple steps. Or, in banking, you could have an open-account service. That might involve six or eight steps, whatever it is. From the business perspective, from the customer prospective, they don’t want to know.

They want to open an account. If they can push a button and the account is open, that’s great. If you can build loose coupling, not only into individual services, but into how you compose services as well, then you’re able to build flexibility into the processes themselves. So, it’s a challenge, but that's the goal -- to build this level of flexibility into the processes, because that gives the business now the ability to have the agility at that business level.

Gardner: If the business side of house wants to have buttons and levers that they can push and pull, then it's up to IT organization to take those inferences and those instructions and then make that into something that changes. It’s a change management function. And, as Jason said, if the processes are composed of individual services, you can rearrange those services, and you could not only rearrange your service within a process, but you can create new services rapidly.

If you have a service that needs to be changed, you don’t necessarily have to rip and replace. You don’t have to shut things down. It’s not a three-year replacement cycle. You can actually recreate that service, make the changes you wanted and then slowly bring that service into production across more processes.

While one of the benefits of SOA/services enablement is reusability, you can also get redundancy. You can create services that are rather similar. You might want to not let anyone be that prolific and write too much, because then you’ve got multiple slices and dices of services. But, it allows for the ability for moving functional sets without having to recreate the entire application.

You’re going to have more iterations, smaller changes within services. You can have services that are similar, and you might well combine them to be a fuller set of requirements within a single service. Then, you can bring that into production across more and more of the organization. It’s just a more flexible approach. You cannot do this when you have brittle applications on a single silo.

Bloomberg: From the business perspective, what you’re saying is exactly right, but, from the business perspective, you can abstract a set of redundant services or revolving services. Think of those that are at a lower level. From the business perspective, they’re all just one service. You have your account-opening service, it’s just the one service, and it works the way it supposed to work.

Behind the scenes, you have redundancy. You have versioning policies. You have management infrastructure. You have all the stuff going on behind the scenes, from the perspective of the business, to get that service to be flexible, so as business requirements change, it does what it’s supposed to do.

From the business perspective, it’s an account-opening service. It opens accounts, and it works for all my lines of business, for all my different kinds of accounts, and it continues to work even if the requirements for that service change. It’s not easy, right? But, you can make it simple for the business by taking these infrastructure and architecture steps within the context of IT.

Question: You talked a lot about the active aspects of SOA management. How is that different from an enterprise service bus (ESB)?

Bloomberg: If you noticed, we didn’t talk about ESBs. Normally, we don't talk about ESBs. We didn’t talk about integration infrastructure at all. We didn’t talk about middleware at all, and that was intentional. When I talk about SOA infrastructure, ESBs are not high in the list. Middleware, in general, is not high in the list. I talk about governance quality and management as the core infrastructure capabilities that SOA requires. From the context of middleware, most organizations already have a lot of middleware. They don’t necessarily need a lot more.

Now, if you’re getting into SOA implementation and you have your architecture, you have your service design, you have your governance, and you’ve thought about governance quality and management, you may find that you need middleware at some point, because whatever middleware you have is not to the task. Then, there may be a requirement for an ESB, or other middleware solution.

But, you don’t want to start there. If you start with the ESB, because some vendor came and said, "Well, to do so you need to buy an ESB," you’re not going to end up with SOA. You’re going to end up with an ESB, actually what the blogosphere is trying to call an ESB-oriented architecture. The point is that if you start with ESB, you end up with something other than SOA. You end up with a traditional middleware-driven architecture with service interfaces.

Now, that being said, there are some perfectly good ESB products on the market, and lot of them offer management capabilities as well. So, when you get to the point of considering what management infrastructure you need, you may turn to an ESB for that capability, but you don’t want to start there. You want to start with the architecture, the business problem, the business processes. Use those two to derive your services. Look at your infrastructure, solve the governance quality management problems, and at that point an ESB might come into the story.

Gardner: ESBs can be very powerful, and, as you are on the journey to SOA activities, integration has been problematic, brittle, expensive, and time consuming. Any productivity benefits that you can bring to integration to me makes sense to me, but an ESB can be more powerful in the management sense, because you can manage the ESB, the way you’ve been further managing your processes or your services or the integration of resources and assets that contribute to the production of your services.

So ESBs do play an important role in management, and we’ll need to see more flexibility and ease of managing ESBs and what they do in applying rules and policies to ESBs. It’s just another important part of the infrastructure.

We’ve had busing and messaging for some time. The idea of trying to make that inclusive of more transport protocols and technologies makes a lot of sense, but how do you manage them? So, again it’s about bringing intelligence from a higher abstraction across more systems. So, I think ESBs will be important, and I think managing ESBs will be important.

Milani: I would add that ESBs are a very powerful piece of this puzzle, and I believe that what they really do today is interconnect multiple types of different systems, and they facilitate and orchestrate the interaction and exchange of data. So, in many ways all they’re doing is exposing existing interfaces, and they facilitate interaction within those interfaces. But, I think what's missing from the ESB picture, from the monitoring and management perspective, is they don’t have deep visibility into the runtime of the interfaces that in fact they are exposing.

So, one piece that you don’t have visibility into is the runtime of those services within ESB, but I do think ESBs will play a major role going forward to marry passive management, which is to manage and monitor what you have, with active management which is constantly "correct and adapt." And I think that’s an area where an ESB could be extremely powerful in the next few years.

Butters: I thank everyone for joining us today. This has been a very enlightening discussion. Again, thank you to our panelists and thank you all for joining us here at the Harvard Club. Have a great day.

Gardner: Thank you.

Bloomberg: Thank you.

Listen to the podcast here. Sponsor: Tidal Software.

Edited transcript of SOA management trends and analysis discussion. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.