Interview: HP’s Tim Hall on Heightened Role of Governance in SOA, Cloud and Dynamic Business

Transcript of BriefingsDirect podcast with Hewlett-Packard on the expanding role that SOA governance plays across IT and business agility.

Listen to the podcast. Download the podcast. Find it on iTunes/iPod. Learn more. Sponsor: Hewlett-Packard.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today, we present a sponsored podcast discussion on services-oriented architecture (SOA) and the insurance that proper governance is providing as enterprises scale up of their use of SOA.

This insurance effect comes through deploying governance alongside and in sync with SOA development and deployment capabilities. The goal is to allow governance to give IT leaders a comprehensive ability to monitor, adjust, and enforce SOA best practices -- so that the productivity, agility, and business process refinements that SOA entails can be realized early.

Perhaps more important, proper governance ensures that SOA will grow without stumbling -- allowing companies to “crawl, walk, and run” to SOA without ever losing control. Done properly, SOA governance heightens the business benefits of services, increases IT efficiency returns, and reduces the risk that complexity could undermine the services lifecycle and hamper the adoption in large organizations.

To provide an in-depth look at how governance and SOA work in concert to empower SOA at scale, we welcome Tim Hall, Director of SOA Products for HP Software and Solutions. Welcome back to the show, Tim.

Tim Hall: Thank you, Dana.

Gardner: Tim, let's look at the context. Things have certainly changed rapidly in the world. We're seeing some uptake in the adoption of SOA. We have some reports and research that indicate that companies recognize the benefits. We're also seeing more economic concern, given the macro-economic situation across the world. At this point, both at the tactical and the strategic level, what makes SOA and its governance increasingly important in the top-of-mind for architects?

Hall: There are a few things, but first and foremost the adoption of services as a fundamental unit of commerce, if you will, within IT does something very fundamental to the way that people work together, and not so much technology. It runs counter to the way that we've been developing systems in the past.

Since the beginning, one of the purposes of SOA governance has been to set the architectural vision and direction, lay the ground rules under which those activities are going to take place, and then foster collaboration between architects, and other people who engage in the processes of building solutions for companies, be they consumer focused, or be they within enterprise IT.

The challenge is that the way that we have been taught to build systems for so many years is really about eliminating dependencies on other teams and other groups. Unfortunately, that's led us into the situation we have now with vast complexity, monolithic solutions and, in many cases, monolithic systems and stacks or silos. SOA is trying to undo all of that.

While, technologically speaking, it's very easy for us to undo some of that, culturally speaking, with the people who are involved, it's much harder to undo that dynamic. That's one of the key game changers about moving to SOA. Do you have the right kind of collaboration solutions fit underneath to support it, breaking down some of the these cultural barriers, or organizational dynamics that may exist within different companies?

Gardner: In addition to this economic climate, we're also hearing a lot more about services coming from a variety of sources and from hybrid scenarios. It sounds like that's even more important, when that's taken into consideration.

Mergers and Acquisitions

Hall: Absolutely. One of the driving use cases that we focus on, since the very early days of SOA, was about mergers and acquisitions. Many of the large financial institutions were already undergoing an SOA transformation internally. The proof of those investments is to see how rapidly some of these systems, teams, and organizations can come together to actually integrate.

They were originally independent organizations, but now, as they are coming together through consolidation, either forced or otherwise, those investments should start to pay off. It should be fairly easy for them to take a quick inventory of what capabilities they expose to services and then determine either how to rapidly assemble those or which ones are going to win out, as they continue down that path.

Gardner: As I mentioned, governance seems to imply more insurance against not only failure, but insurance that, at each stage along the way -- that crawl, walk and run scenario -- the pay offs are there, the return on value is there, and the ability to manage the people and the process is there. Tell us how governance works -- the technology and the people issues.

Hall: The whole thing is tracking your progress, where are you in this journey. It's not about installing a new pack of middleware and then declaring victory. You really have to measure along the way what you are doing, and how far you have gotten. Some measures that people start off looking at are things like reuse.

We have one particular company that has been engaged in an SOA transformation for about a year or a year-and-a-half. They've identified a particular function within their organization that they turned into a service. And now it's being reused by 11 different groups within their organization. They estimate that they have saved over a million dollars in redevelopment cost, or duplicate development costs. It's avoiding those costs by having them capitalize on the service that they've offered. And, they're able to measure that through their governance activities.

Further, they're able to have a single service catalog, where they can look and see what SOA-based services have been published by these different groups. They're able to review ownership to make sure that people aren't creating kingdoms of services that they shouldn't be responsible for and distributing that functionality based on their actual roles and responsibilities within the organization.

They're also able to apply architectural policies that they can use both to inspect the services and service artifacts for compliance against the architectural vision where they are going, as well as checking for best practices. This can be done in an automated fashion, which then frees up resources from having to desk check or to manually check those artifacts one-by-one.

Gardner: I suppose with any large scale and complex undertaking like SOA there might be a tendency to say, "Well, let's wait on certain things and let's test on a pilot basis or iterative basis." What's the rationale for bringing governance in early, part and parcel with just about any other SOA activities?

Hall: There's a real spectrum of responses to that question. We certainly had customers say, "You know what. I'm not going to be ready for this, until I have X number of services under my belt." And, we certainly have had other customers that say, "I don't even want to get started on this until I have the appropriate infrastructure put in place, because I know how my organization works, and without that supporting element, I fear for chaos on day one."

It's really a matter of mapping your organizational maturity and what you're trying to achieve with the appropriate tools. People shouldn't be running out and buying tools, unless they really understand what problems those tools are going to solve, and the fact that certain organizations can introspect what they have done in the past and say what problems they want us to solve and or avoid. With zero services, it's great.

Other organizations need to try it out within their four walls and get some hands-on experience, some organizational or collective learning, to project how they want to take things forward from there in a way that works for them.

HP is here to help either customer take those steps, but the key thing is looking at the organizational dynamics, the types of questions that you'd like to answer, the type of activities you'd like to automate, and then coming and working with the vendors to see how products can help mix and match to meet their specific needs.

Gardner: Now, you've done some research looking into how companies are actually putting these into practice -- these methods, technologies, and organizational approaches. Was there anything that surprised you, and was there anything that stood out that reinforces some of this "governance first and center" mentality?

Standards Drive Adoption

Hall: The thing that's surprising to me is that the adoption of SOA is kind of spread out. It's going on its eighth year, and I am not talking about just WS-*, Web services set of interoperable standards. In general, the concept has been around for a long time, but the current wave that we are talking about was really driven by these sets of standards.

What's interesting about it is that we're learning lots of interesting things about IT, and in particular, the ways that we can do things better. The whole notion of instilling an architectural vision to support change and flexibility; to give tools to the folks who are building composite systems, so they can better manage the roles and responsibilities for the various people that are participating in that; and better communicate with operations is something that we haven’t done very well.

So, the surprising thing for me is that the lessons that we're learning, that are specifically being applied to SOA right now, have more far-reaching implications. As we look at things, like the different compositional patterns for systems that are coming -- Web 2.0 technologies, Ajax, rich Internet applications (RIAs), putting front ends on some of these things, or cloud computing -- all of these things are interrelated. My question is, should we not be applying these fantastic concepts and activities that we have been establishing through SOA governance more broadly to support all of these different types of next-generation composition?

From HP's perspective the answer is absolutely. The question is at what point are we going to be talking about next-generation application lifecycle management, or next-generation application composition and stop talking about SOA by itself as an island.

Gardner: It really sounds as if we're not just talking about governing the SOA transition, but about governing IT transformation fundamentally.

Hall: That's right. The big issue is that we seem to be reaching this point of event sustainability, where IT has been focused on what we call "capability-centric IT." It's focused on servers, storage, CPUs, fan speeds, and all these things.

That's just not the language of business. The challenge is, when we have all this complexity we have to deal with, how do we hide it? How do we tune it, so that it's working in an appropriate manner, and aligned with what the business is trying to do? The answer is that the lessons are coming out of services.

The whole notion of providing a service is to hide the layers of abstraction and to hide the complexity behind layers of abstraction, so that we can make changes behind the scenes that don't necessarily disrupt or alter the offering of the service. There are a lot of examples of this in the real world. Why hasn't IT been able to do a better job of capitalizing on those things?

This is one of those transformation opportunities. We're not just talking about Web services. We're talking about different ways in which we need to be able to flexibly compose and offer capabilities back to the business through a channel called a service.

Gardner: So, the tools, technologies, and methods that we have in place and that we're starting to scale out for governance can cross some boundaries, right? For example, "development and deployment," not just "development and then throwing it over to deployment."

There needs to be more coordination there among architects, but also those focused on business processes, and those focused on the agility of the business, and how that relates. Tell us how what HP sees as SOA governance is able to cross these boundaries.

Hall: One of the things that we are seeing more and more of, as we're going deeper into the end of 2008 and looking forward into 2009 and the spread of adoption over the last seven years, is that new constituents come to the table. They ask, "What's the lifecycle of this service?” We've got this group of people who are now testing the service. How does that relate to its status for promotion into production environment? Shouldn't they get a say as to whether the service should or should not be promoted, based on the results, be it functional, performance, or security testing? They absolutely should.

On the flip side, maybe earlier upstream, you've got a group of business analysts, who are being told, "We need to offer a new product to the market. Go figure out how we are going to do that. What are the different channels of distribution? What does it mean in terms of the supply chain? What does it mean in terms of ordering off of the Website, and how can we facilitate that as rapidly as possible?"

And they're like, "Oh, gee, what do I have in my toolkit to be able to pull this off?" The first things they want to do are: A, understand the business requirements, but then B, look at what's available to them. Then, can they reasonably compose something out of what already exists. Or, can they work with folks in IT to say, “Hey, there is a gap here. We've got 80 percent of the parts we need, but we need somebody to fill in this 20 percent. How quickly can we get there?”

So, there are more people coming to the table, more constituents coming to say, “How can I connect to these governance activities that are going on for services, but really for the purpose of generating some new business outcomes?” That, to me, is tremendously exciting.

They want to link in to the control points for the service lifecycle, and clearly we can offer up where that happens. From HP's perspective, we are definitely trying to make sure that the collaboration between architects, quality assurance professionals, and operations personnel are there. That's kind of announcing that the various solution offerings that we're bringing to market are to make sure that none of these is an island. Those control points can reasonably be connected and allow for collaboration across all the different participants.

Gardner: That's what quite different about the SOA governance, compared to traditional IT management. It's, "Bring more people to the table, but get them there in a way that these inputs can be accepted, balances can be found and adjusted, and then automated over time." Those are the balances between too much control over what people can do, versus too little, but on a dynamic basis.

Tell us how the touch points for these different folks who have an impact, or role, and should have an ability to contribute and collaborate as to how these services evolve. Tell us how they relate to governance, at least in HP's philosophy. How do they engage with these tools? Is this a series of different inputs? Is there a methodological professional services approach?

Individual Tools

Hall: Everybody has their own set of tools currently. When you look across the IT landscape, are you going to try to drag people out of the tool set that they are currently using into something new, or you are going to keep them in their existing tool set and find the plug points that allow them to collaborate a little more naturally?

Gardner: I suppose we're at a point now, where we don't need to be a SQL-programmer, or a C++ programmer. Now, more of the folks who are involved with the business process are able to have the inputs into these governance functions.

Hall: That's exactly right. That's exactly right, and so everybody, whether they're using a modeling tool to define business-level artifacts, or whether it's an architect who is in an integrated development environment (IDE) looking at a particular artifact, they need to be able, in some way, shape or form, to plug that back into the system of record, or a system of record, that then helps facilitate communication across the various other teams.

One of the strategies that we have employed is to build specific plug-ins for the IDEs or the modeling tools. Then, the other portion of the strategy is to ask what standardized application programming interfaces (APIs) we can start to offer that allow us to connect to third-party systems that are responsible for quality assurance or establishing a configuration management database and operations, so that we can understand how to start connecting to these other systems and to systems that might exist within organizations that may not come from HP.

Gardner: I suppose that payoffs and return on investment are important. They always have been, but they're particularly important now. What examples do we have? How have companies benefited from governance and recognize that governance is part and parcel of SOA? If you have some companies, some anecdotes, or some case studies, I think that would help.

Hall: I mentioned one. This company recognizes that they saved a million dollars in the first 12 months, simply by having and establishing a service catalog and publicizing it. Before folks went down the path of building something custom, they looked to the catalog first, and saw that something existed that they could utilize immediately. They've got this particular capability now consumed 11 or so times now within their organization. That was huge.

We have another large telecommunication company in Europe that has had a 320-odd percent return on investment (ROI) in establishing their SOA governance and management solution and integrated solutions that include both of those parts. It crosses the spectrum of everything from customer retention, to time to market, to decreased downtime and increased availability. They did a fairly comprehensive job of looking at what they had before and what they were trying to get to, and they were pretty pleased with the results.

Gardner: Are there any other payoffs from governance that people might not be aware of that some of these organizations are finding as it become a bit more mature and a bit more scaled out when it comes to the SOA use?

Hall: A lot of it has to do with the cultural aspects. People are surprised to find that it's so difficult to change the people who are engaged in the activity of building systems. So, it's better that you can provide the tooling underneath them, so they have a standardized mechanism that they can utilize to understand what other people are doing. There is a huge benefit to that.

We have teams of architects that are plotting out what needs to be built and when. There are certain synergies that you can get from that by identifying, “Hey, wait a minute. We're about to start this project, and it looks like somebody has identified this particular service should exist in our lexicon, our enterprise architecture if you will. We should go and talk with them, and get joint requirements built out on this, and we could both take advantage of this more quickly." I think that's a huge hurdle to overcome, when most organizations operate on the “Not-Invented-Here” mentality.

Gardner: Let's look at the future. We mentioned earlier that the cloud and services from a hybrid or variety of sources seem to be appealing to more people for a variety of reasons. We're also seeing why it makes sense to balance governance across more than just IT functions, involving business process, management, and organizational issues. What's your take on the future when it comes to governance in SOA? Do we start to think about governance more broadly in SOA, in the sense that it becomes the underlying fabric of how companies balance IT innovation and management?

Hall: Absolutely. That's something that the SOA governance activities are teaching us. Establishing the vision for where you want to get to, and then trying to automate the checking of how you are doing towards that is definitely a desirable goal. But, I think one of the things you're going to see -- I'm not sure how far in the future, it's coming up more and more these days -- is an emphasis on understanding the business-to-business connections, or what some folks will call "federation."

I want to be very specific when I say "federation," because it is one of those overloaded terms that creates a lot of mystery. If we can take the wraps off of federation, what we're talking about is a pattern for how to expose the capabilities that I own within my domain to other domains. Those other domains could be within my organization, they could be elsewhere, or they could be third parties.

The good news is that SOA fundamentally supports that type of activity. The question is how well the tools support that activity today. HP has been at the forefront of this through the establishment of UDDI, a standardized protocol for sharing metadata across multiple environments, whether that's through the use of private UDDI, which is the most widely used UDDI registry today, or even in the early days of the public UDDI.

What you're going to see, especially because of the merger and acquisition activity we talked about, is the emergence of software-as-a-service (SaaS) offerings. As we move into a more comprehensive cloud set of offerings, we're going to need to federate the different instances of services, metadata, their ownership, the consumption of those pieces, and really formalizing the relationships of using tools between the consumers and providers of those things.

When I say establishing relationships, I think about trading-partner agreements that get put in place, or supply chain agreements. They get put between supply chain partners about what information they're going to share and in what context they can use that. We're really talking about doing the same kind of formalization with the consumption and providing of these various capabilities, in order for models like SaaS and cloud to scale up to the level that they need to in order to make a significant impact.

Gardner: It almost sounds as if the boundaries between the internal organizations inside companies, as well as between partners, supply chains, and other ecologies are becoming more permeable. That's important and that's good for a business reason, but it also needs to be managed, It needs to be balanced across risks, privacy, security, access, identity governance, and those sorts of things. So, governance really seems to be again at the forefront, not just of SOA, but of how companies will redefine themselves as not just a brick wall between them and the rest of the world, but as the sort of managed permeable membrane -- for lack of a better analogy.

Internal Governance is Necessary

Hall: That's absolutely the case, and I think the concern that everybody should have, is that you don't treat people outside your organization the same way that you treat people inside. In some ways, that's a good thing, and in some ways, it's a bad thing. As a specific example, you go through a lot of headache and heartache to put those trading partner agreements in place. There are lawyers and stacks of documents that go back and forth. The good news is, you have established the ground rules for who does what to whom, when, and where, including the worst case situations.

That's great, except that you don't treat the people within your organization the same way. Then what happens is that you're running on a set of informal agreements. When there's a problem, what happens? If that permeable membrane example is going to play out and be effective, we'd better start doing some formalization of those relationships internally, because you never know how long that relationship is going to last. It maybe internal today, and it maybe external tomorrow. You'd like to have the ground rules be relatively consistent, as you move from one model to the next.

Gardner: So, we'll need to have the ability to identify the rules, house the rules, share the rules, enforce the rules across these business activities, and SOA governance seems to be the best candidate at the moment, right?

Hall: Absolutely. The big deal is looking at how we can foster better collaboration through the formalization of these agreements. For example, a service provider needs to declare what roles and responsibilities they have to fill, as well as setting the expectations of what the consumer is responsible for doing, and do that in a flexible way that can be negotiated using the tools.

Gardner: And, importantly, the visibility is there because people need to examine whether these relationships are working or not, what may or may not be right or wrong with them, with the proper access that they would get by overseeing an SOA or services lifecycle? They get that into these business relationships, and it's "trust but verify" basically, when it comes to this level of governance.

Hall: That's exactly what I'm saying. At what point are we going to stop talking just the SOA aspects of this, and broaden this discussion and say, “Look what we learned from SOA governance. This can actually apply more broadly to a whole range of relationships, including application composition, be it internal, external, etc.”

Gardner: We can probably go on for another hour just talking about the data sharing implications of all this.

Hall: That's actually a really interesting one from a regulatory perspective. You start hearing different government organizations popping up and saying that we cannot put our medical records on a server in India, China, or anywhere other than within our borders. Those are going to be regulatory requirements that all customers have to operate under, and so they're going to need to look at those relationships. Even these SaaS and cloud providers may need to develop distributed mechanisms and instances of their technology, to ensure that they are able to do business and comply with those regulations as well.

Gardner: Just to toot your horn, I suppose HP has a number of these technologies, and areas of expertise in its quiver, be it IT management, SOA governance, or SOA infrastructure. There is the business technology optimization (BTO) through the lifecycle of development and deployment. There are the professional services, the understanding of these businesses. So you're seemingly in a pretty good position, given what we've been discussing.

Hall: HP has become the largest technology company on the planet by revenue, and there is a reason behind that. It's not just printers and ink. We're aggressively continuing to move forward on a number of these fronts, from investments that we make through our HP labs, which is the kind of the deep research that we see paying off between the five- to 10-year time horizon, to how do those things transition into specific product offerings and capabilities that come out of our hardware, software, and services groups.

Obviously, the acquisition of EDS allows us to scale up our service offerings as well. We have a big quiver, and we definitely pull all those pieces together to deliver comprehensive solutions to customers.

Gardner: I think we will leave it there. Obviously, it's a very large opportunity, but not without pitfalls. For those companies that do get governance right and can expand it beyond just Web services at a department level, and bring it from a tactical, strategic, and then extended-enterprise basis, there are perhaps some very important business benefits.

Hall: Absolutely, and it's critically important to look for trusted guides, people who have seen the last seven or eight years, and also have a vision for how to take this forward.

Gardner: Well, great. We've been discussing the importance of SOA governance and how it helps heighten business benefits. It can return higher efficiency and reduce risk of the complexity that can undermine services across the lifecycle. Helping us to understand these issues today has been Tim Hall, director for SOA products for HP Software and Solutions. Thanks for joining, Tim.

Hall: Thanks again, Dana.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions. You've been listening to a sponsored BriefingsDirect podcast. Thanks, and come back again next time.

Listen to the podcast. Download the podcast. Find it on iTunes/iPod. Learn more. Sponsor: Hewlett-Packard.

Transcript of a BriefingsDirect podcast with Hewlett-Packard on the expanding role that SOA governance plays across IT and business agility. Copyright Interarbor Solutions, LLC, 2005-2008. All rights reserved.