HCM SaaS Provider Workday's Advanced Architecture Brings New Business Agility Benefits to Enterprises

Transcript of BriefingsDirect podcast on how Workday's SaaS delivery model for human capital management applications provides better business intelligence and architectural advantages to end users.

Listen to the podcast. Download the podcast. Find it on iTunes/iPod and Podcast.com. Learn more. Sponsor: Workday.

Special offer: Download a new white paper on Workday's latest update to System 7.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you're listening to BriefingsDirect. Today, we present a sponsored podcast discussion on the virtues and paybacks from designing a strong IT architecture.

We'll examine how such architectural benefits promote business agility as a service while lowering total cost for both the deliverer -- and the receiver -- of pure application services. This perspective looks at IT architecture with a new twist, not just in terms of developing architectures on-premises, but ... for architectures that support the providers of services.

We'll take a look at how IT architectural best practices at a software-as-a-service (SaaS) provider help not only that provider's operations. We'll show how the users, the receivers, of those services can benefit in new ways as well.

By examining the experiences and approaches of Workday, a human capital management (HCM) SaaS provider, we can better understand the benefits of modern IT architecture and gaining new levels of business intelligence (BI), innovative search capabilities, and the ability to extend business processes out to mobile devices.

Here to help provide an in-depth look at how proper architecture allows the SaaS delivery model and business agility to come together, we're joined by Stan Swete, CTO of Workday. Welcome back to the show, Stan.

Stan Swete: Hi, Dana. How are you doing?

Gardner: I'm doing well. Workday has had an advantage in that you knew what your goals and objectives were when you started your architectural journey. You knew that you were going to go out as a service and you knew that there were some new modern technologies, approaches, and best practices to take advantage of.

That's a little different from a lot of enterprises that have, in many cases, had decades of IT to adjust and, in a sense, drag along with them. Let's first hear about the level set. When you started out from green field, what things went through your mind, and how was that refreshing, given that you were starting from scratch?

Swete: First, it definitely was refreshing to get the opportunity to start from scratch. I'm sure that if you talk to a lot of IT professionals, they'd all want that chance. At Workday, we had that chance and we started our company with a lot of background in what had gone before in terms of architectures to support enterprise resource planning (ERP).

We had that backlog of information, a list of what worked and what didn't work so well with previous client-server architectures. As you said, just like everyone else, we definitely had an appreciation for all the new developments in technology that make different approaches possible today.

Gardner: Today, enterprises are faced with a number of challenges. They're trying to adjust quickly for very dynamic business environments. They have to watch their costs. They'd like to modernize, but there's a significant time lag between how and when they can take advantage of these modern concepts and when they can't.

Do SaaS providers like yourself allow them to leapfrog by taking advantage of what you've been able to do and then bring those benefits into their business practice?

Complex environments

Swete: We absolutely think it does. You're right. In IT today, people are in a difficult spot. They have complex environments. The complexity has grown for a variety of reasons. Everyone sees the opportunity to modernize and to improve efficiencies, but how do you do that in the midst of a complex environment that is constraining just how aggressive you can be?

That's where SaaS can come in. If you've got a provider like Workday, or someone who's able to take a clean approach, there might be the opportunity to take the right portion of a certain set of your applications and, instead of having to deal with the complexity of managing all the multiple instances and different architectures you might have, use the unified SaaS service as a way to achieve some integration and a way to drive against cost. Today, it's all about cost.

A lot of the discussion around cost has driven IT to look at a variety of dimensions. Just the consolidation of some of the complexities and different instances of architectures that large companies all have is one area where they see opportunity, but they are bound by having to support where they have in place.

Gardner: What intrigues me, Stan, about what you're doing at Workday is that you have gone beyond the concept of just delivering an application or set of applications. You're positioning yourself as a partner for these businesses and how they can then relate to the outside world. You're extending the enterprise boundaries for them, and this notion of business agility as a service kicks in.

Explain for our audience what you mean by extending your value, not just as a provider of applications, but as an extension to enterprise architecture.

Swete: The space we're in is HCM, trying to be the core system that captures all the information about the workforce -- how it's organized, how it's compensated, and what work is actually done.

If you're trying to play that role at an enterprise, you can't be standalone and independent of the enterprise. You're going to have many, many ties into other things that the enterprise has going. You're going to have to be agile, in terms of how your solution can fit into different instances of different enterprises, because everyone has a slightly different picture and puzzle.

What that means to us is that there's a demand on the solutions to be agile, to be able to change shape and to be able to integrate to the variety of different scenarios you have. Otherwise, SaaS doesn't become a productive option for the IT professional to look at, if it's just one-size-fits-all and take-it-or-leave-it. They're going to have to make some trade offs that they can make.

So for us, if we can be viewed as just a solution that can meld itself to fit into whatever else it has got going on and to be an effective core that we can handle in the cloud for them. That's the best way for us to engage.

Gardner: You're also becoming sort of an adjunct and maybe even an accelerant to service-oriented architecture (SOA). If organizations have begun their journey toward SOA, you can provide a catalyst to that as a go-between, a services exchange, if you will, with a number of other providers.

Maybe it's payroll, maybe it's healthcare and insurance benefits, perhaps it's reaching out to partner with other organizations and the labor resources that they have. Do you see it that way -- that this is, in a sense, SOA but as a service?

Embracing Web services

Swete: Absolutely. You mentioned before about us having the ability to look at new and emerging technologies and approaches to architecture. Certainly, we've got the religion of SOA and firmly believe that the right way for us to tie into other systems in the cloud and other systems on-premise of our customers is via SOA and an embrace of Web services.

We embrace that and we think to some extent that it can accelerate SOA adoption within enterprises. Enterprises we talk to all see the technology the same way we do. They all see the appeal of newer SOA architectures, but I go back to the fact that they have the same issue. They have the whole other set of architectures that they've got to be concerned about maintaining.

So, if they can see a segment of their application stack -- in our case, the governance and control of the global workforce -- that absolutely can be an SOA project that we can jointly embrace and can get them down the road toward this new architectural style.

Gardner: For those of our listeners who might not be familiar, Workday is relatively young. You are only few years old. You're based in Pleasanton, CA. You’re focused on HCM, but why don't you fill that out a little bit. Then, let's also talk about the philosophy that you embraced when you started building out your services. Finally, we'll try to take a look under the hood. So, first, a little bit about Workday and then what you have got supporting your architectures?

Swete: I'm happy to introduce the company a little bit. Workday is about a four-year-old startup, as you said, based in Pleasanton. The company was founded by Dave Duffield, former founder of PeopleSoft, and co-founded by Aneel Bhusri. Basically, the company got together four years ago and began development of its products. We launched publicly in November of 2006 and then we had our first version of our HCM business services and two production customers.

Since then, we've grown the customer base up to north of 75 customers, with more than half of them in production. The idea behind Workday, in addition to being a SaaS company and in addition to focusing on HCM, was to focus on enterprises in a space that we define as the upper mid-market and we started with a focus on companies with between 1,000 and 5,000 employees.

In the past two years, we've had great success in selling not only to that target market, but we've been able to move up market and attract larger customers. Today, the average customer we're engaging is probably closer to 10,000 employees than 1,000. So, we'd say that we service companies in a range between 1,000 employees and 20,000-30,000 employees.

We've even attracted larger customers. Flextronics is the example of our largest customer. There are over 200,000 employees, and have selected Workday as a way to consolidate a number of HR instances that they have around the world.

Gardner: Given the fact that you needed to be modern, you needed to be flexible, but you also needed to scale, what were some of your requirements, and what did you end up with in general terms to make this possible?

Success at a high cost

Swete: Let me back up a little bit and just say the other bit of information to toss in about what was motivating us to start the company was just taking a look at the enterprise solution space and starting to identify some of the complexities of owning and implementing these applications.

It's our belief that enterprise applications have driven a lot of success and a lot of value in enterprises, but that success and value has come at a very, very high cost. Essentially the systems come down to being very hard to use, hard to change, and hard to integrate. Those were three thoughts in our heads as we started Workday. We wanted to go after the same space, which is a complex space. There are complex processing requirements and hence, the need to have a solution that’s going to scale.

So, it was taken as a given that we were going to have feature-rich applications that needed to scale to support large work forces, but we wanted to achieve that, while also attacking the issues of being hard to use, hard to change, and hard to integrate.

That's what led us to evaluate the new technologies in terms of how could we take an approach that would allow us to progress against those issues, while still being able to satisfy the enterprise-class functionality that you have to have to play in this game.

Gardner: And what did you decide on?

Swete: The approach we ended up taking, as we took it to the next level down, was that we started to investigate where some of these complexities and difficulties of use came from. As we evaluated the prevalent architecture enterprise systems, that of client-server, there were a couple of dimensions that we looked at.

From the point of view of being hard to use, the user experience for those applications typically was, first of all, designed for a highly trained back-office user and was deployed as a menu-driven application with tons of fields on each particular screen. Then, that user experience was migrated to the Web. So, these were not native Web applications. They're applications that found their way into the browser, but retained their old complexity and difficulty of use.

From the beginning at Workday, we knew we wanted another approach. Rich Internet application (RIA) experiences were emerging and supported by new technologies. We committed ourselves to being an application that was built first and foremost for the browser and one that was also built to consider the needs of not just the back-office, but the rest of the workforce. So, we started to look at technologies like Adobe Flex to give us an ability to be in the browser but to still deliver some of the rich experience that customers expect.

On the side of being hard to integrate, we took a look at the client-server architectures. The way these applications were written was to think about regulatory information that was required. Design a data model to meet that regulatory information and then build the transactions to feed the data model. That gave you a monolithic application that could generate the reporting you needed to satisfy your regulatory reporting, HCM, and financial management.

The integration of other systems to feed data into those systems or to get data out of those systems was left as an afterthought. Integration was not thought about upfront. Integration was not thought about in terms of the new Internet standards we have in the form of Web services, or the new architectural forms and approaches we have in SOA.

From the beginning, we thought about a system that would be able to deal natively with producing Web services to get data out of and back into the application and would treat the conversation with other systems as a first-class conversation, just like the conversation with individual users.

Turning to the ESB

We began an investigation of tools that could help us do that and really looked into the SOA space. We thought that what future state enterprise business applications needed was to embed some of the technologies you find in enterprise service bus (ESB) technologies. So, we've gone ahead and done that.

We do have some of the transformational and delivery options in multiple formats available to us in our data-center, so that the Workday applications can generate Web services. Beyond that, we can transform those Web services into other data formats that might be more meaningful to legacy applications or the other applications we need to tie to. We did a lot of work in that area and came up with the need to embrace Web services and embed in an ESB in our case.

Gardner: You also created what you call an "object management server." Why don't you explain what that is and why that makes sense?

Swete: That comes from the third area of these applications as being hard to change or just the general theme of this discussion today, which is this whole issue of agility. How do you make complex applications configurable, not only when you're initially implementing them but postproduction.

As you move forward, it's not like your business stops changing after you initially implement the enterprise application. Your business is constantly changing, and client-server based applications have shown a real inability to keep up with changing at the pace of anyone's business changes. They've shown a real high cost to be able to change to incorporate new functionality postproduction.

In working on that problem, we took a look at client-server architectures. It's our view that there's a lot of the rigidity in these architectures, and rigidity is what leads to a lack of agility. We think the rigidity in these architectures comes from the fact that you've got a complex logic layer.

Typically, it's multiple languages, but you have an executable that's built out of a lot of lines of program code. Millions of lines of code, in most cases, are backing the logic layer of enterprise systems. That layer has a complex conversation with the relational database, which also has its own complex structure -- typically thousands of relational tables to model all of the data.

Bringing change to that environment is difficult, because changes to the logic layer have to be synched up with corresponding changes to the data layer, and both layers are constantly changing. We decided to take an entirely new approach in this area and embrace an approach that leveraged the concept of encapsulating data with some of the logic into an object.

At Workday, the primary logic server is what we call our Object Management Server. It's a transaction processing system, but it's entirely based on an object graph, and that is just a class structure that represents not only the application and its data, but also the methods that process on that data.

The important difference is that we have that layer and we don't have a correspondingly complex and changing data layer. We have a persistent data store that is a simplified version of a relational database that can persist changes that happen from the object layer. But, as we're developing our applications and changing our applications, we don't need to constantly change the shape and form of the persistence layer. It's an unchanging relational schema that can persist, even as we make changes up in the object layer.

This frees us up to, one, develop our applications more rapidly, but two, to change those applications, even for our customers in production, without having to take the system down and make structural changes to the underlying database.

So, our two new approaches really reduce the coding you do up in the object layer. We try to define the application more as metadata and reduce the complexity of the relational model that you have.

Leveraging modern architectures

Gardner: So, we’ve recognized some of the handicaps of some of the older approaches, recognized the new set of requirements for the modern day, understood that SOA principles can be applied here quite advantageously, and recognized that rich Internet application interfaces are the way to go when the browser is the ultimate client target. Without getting into too much more detail, we've certainly established some improvement in modernization around the architecture. Let's get into a little bit of what that does for you.

We talked in general terms about agility, but there are some interesting add-ons here -- things that you couldn’t have gotten otherwise in terms of benefits. We're not just re-paving cow paths in terms of delivering applications and services. We're actually now able to do interesting things around BI, around scale and customization, and around different services federated to different users, but with more commonality under the hood. That brings more total cost reduction for the end user.

Let's get into what these modern architectures do not only for you in terms of cutting your costs, but advantageously creating new business benefits for your customers.

Swete: When you combine the architecture we talked about with the SaaS delivery model, you get some of these benefit categories. We get benefits in all the categories that you just mentioned, and you're absolutely right. Our view is that it's equal opportunity. There are definitely benefits for the customers that we're serving and, frankly, we think that in the approach there are tons of benefits for us, as a vendor, to take cost out of what we're doing and pass those savings on to our customers.

You named a lot of categories, so let me let me start with one area, which is the benefits we get out of doing more about integration. With an architecture that really facilitates integration and especially, if you combine that architecture with a cloud-based approach or delivery of SaaS, you get what we at Workday call "hosted integration" or "integration on demand."

We use our embedded ESB to do exactly what you just said. We take the ESB and package up integration so that it can be reused across a wide set of customers. The best example of packaged integration within what we call the "Workday Integration Network" is our benefits carrier network.

Here's a solution where Workday has used Web services to tie our HCM solution to a variety of benefits carriers and we offer customers the ability to sign up for this network. They would pay us for the use of the network just as they would pay us for any of other business service that we deliver. What we're able to do is offload the need for them to convey their benefits data out to the carrier and to get information back from the carrier into their human resource system.

This has been a very popular option for most of our customers, because most of our customers are large enough to have multiple carriers. All of the carriers inconveniently have multiple data formats, and the formats are always changing, and the mapping and testing of data access of those formats is always a cost. So, we're able to lift that off of them and just give them a service, which is a tie of all of the carriers they select into the Workday benefit system.

Gardner: Stan, one of the interesting things about cloud and this whole notion of centralizing allows for different things to be done with data. Now, the data is often in little nooks and crannies, in different formats and inside of different architectures. But, as we centralize the architecture, we're also getting more access to different types of data. In doing so, we can do joins, overlays, and comparisons in ways that hadn't been done before at a scale that hadn't been possible, at least at an acceptable price.

Let's get into this notion of BI. What can your architecture bring to the table, and what can your clients start to do in terms of gaining insights into what's going on inside their companies, but just as importantly, in conjunction with their business processes and extended business processes.

Built-in business intelligence

Swete: That's a big question. Built-in BI, as we call it, is absolutely an advantage of our offering. That is what we're offering today and the future that's possible. I can go through maybe a couple of levels on this, because as the customers that we've attracted look at the Workday solutions, they see unusually rich access to data as the first basic offering.

Having an object model that allows us to link more data attributes together than a classical relational database to establish relationship is a lot lighter weight than having to build the foreign key into another table. We're able to cross-link a lot of information that we're tracking inside the object model that we have, and so we're able to offer unusually rich reporting to the customers.

One example is that, just like many HR systems, we can give a straightforward headcount report, but with the Workday system, the headcount report isn't just a flat report in the system. We offer you the ability to give the headcount by organization to link to other information about each organizational entity, get its details if you want to, or see other reports related to that organization, probably more of the point for BI for the actual summary headcount that's in that particular organization, without the need of a third-party tool.

Workday is going to offer you the ability to drill down on that aggregate number, and take a look at the number in terms of all the dimensions that go into it. An HR professional could look at the headcount for an organization and analyze it in terms of gender by business site, for example, or job code by business site. Our transactional application is facilitating multi-dimensional analysis without the need to have to take the data, off load it into an OLAP cube, and then, by a third-party tool, query that cube.

The barrier that we have broken is -- again, going back to client-server systems -- you had these data models that were defined to provide one kind of reporting. Typically, it was regulatory financial reporting or the regulatory HR reports that you have to provide.

Workday provides all of those, but it crosses over into information that could be more interesting to the people who are not just back-office HR professionals, but maybe managers who wanted to get information about their workforce. That is all built into the application, and that's the level of increased BI we're delivering today.

As we look forward into other things we could do, you mentioned reaching across different elements of information. We do a bit of this today in terms of the automated business processing that we offer. We have built-in workflow in Workday and we have a system that is able to track all of the performance of the automated business processes we deliver.

Our BI today can reach across and look into their performance information and give the manager information, such as the average time it took you to do a hire, which department had the longest time to do it, and which department had the shortest time to do it. We'll continue to improve this kind of analytical information over your business performance. We see this as a very valuable area going forward and we'll get richer analytics supported in that space. That's an area that we're verging into.

The third opportunity, which you’ve also alluded to, is the benefits carrier network. There's an ability to drive intelligence across the population of users who are engaging in this network. For example, statistical information about the most widely used carriers might be interesting for any company in the network, whether they have those carriers or not. Usage information about the relative number of people using a carrier inside the company would obviously be interesting.

There is just a large world of opportunity to expand into, but the important thing is that we think as a transaction processing systems, which is what these systems are classically looked at as, we're growing out of that to also provide BI without the need to buy third-party tools to do it.

Social networking

Gardner: We're talking about people here -- workers, productivity, habits. People don't just live in the workplace, they have lives as well, and we have this phenomenon now that's going on around social networking and the ability for people to connect in new and different ways. It seems to me that this offers the potential for yet another large data source to perhaps be compared, contrasted, brought-in, and in some ways leveraged, vis-à-vis what's going on in the HCM apparatus inside the organization.

Swete: You're absolutely right. That actually even widens beyond social networking to the proliferation of productivity tools that get called broadly Web 2.0. For me, what that means is that to be an enterprise application that's relevant in a world where those applications are gaining increased usage, you not only have to have the great system-to-system integration that I talked about before, but you have to be an application that, with good security, is mashup-able, if I can use that non-word.

That's the second dimension of integration that you really can deliver on, if you’re an application that’s built for the Internet, just like the Web 2.0 applications, and certainly Workday is that. The application appears to our users as a Website, and the data in the application is accessible via the Web services I talked before, but you don't always have to construct or communicate in a heavy-duty Web service.

Workday has the ability to have RSS feeds of our data, the ability to instantly tie to emailing systems, the ability to link out to make calls to a phone numbers in Workday. The most popular mashup is always mapping locations of, let’s say, business sites. Workday is wide-open to that type of integration, and we expect that to explode, as the use of our applications reaches out into the workforce.

Gardner: I certainly expect that the ability to integrate to the social tier is becoming all the more important, and can be extremely valuable. I don't think people have plumbed the depths of what productivity benefits are inherent in that.

Swete: People are thinking about that a lot, and there is huge opportunity for a certain class of social network apps to bond with enterprise apps and deliver real value. A great example of that is LinkedIn as an application that really can facilitate a new way to do recruiting, for example. They certainly see that as an opportunity, and it's incumbent upon modern enterprise applications to be able to tie into services such as that, so you can get that kind of benefit.

Gardner: Another important tier to integrate to, or to reach, is the mobile tier. You guys have apparently put some thought into that. Tell us what a SaaS provider like yourself can bring to the table for an enterprise that would love to be able to get more data, more applications, and more business processes extended out to mobile devices across these mobile network.

Widening access

Swete: That's extremely important for us. I think I said this earlier. With our applications, in order to overcome the application being hard to use, you want to work on your native user experience, but you also want to consider that the wider user population is just not always going to use your native user interface (UI). You're going to have people who want to use your application without getting into the pages that your application actually renders. Mobile is a great example of that. We absolutely see widening out access to Workday on the mobile devices.

We're just taking our first step in that direction right now, and this is improving the benefits of the modern architecture. We've been very quickly able to extend the business-process framework that we have. This is the framework that delivers the automated workflow that I spoke about before. We've been able to extend that out so that approvals that are done within that framework can now be completely processed on a mobile device.

We’ve picked the iPhone as the first starting point and we'll be expanding out to other devices. The benefit here is that you get access to the Workday solution without having to be at the mercy of a browser on a tiny device running the rich UI that Workday generates.

We're able to mark-up a subset of our data and have that appear in a native client on the iPhone that you can get on the App Store, just like you get any other iPhone application. Then, with security, you're just utilizing a native app, which is acting on Workday data. We use that for manager approvals, the management of to-do lists, and for enterprise search of the workforce. That's been a successful example of leveraging this modern architecture. We didn't have to go in and rewrite our applications.

Our applications developers merely extended the existing application to say, "for small profile devices here's the data that's relevant to show" and they stopped there. We're able to use the separate UI layer and extend that UI layer to generate a completely different view of what the approval would look like for the mobile device. Then, process that just as if it were a transaction coming in from our own native user experience.

Gardner: I can see where a line-of-business manager could really benefit from this. They've got some ideas about what they want to do with their business. They, of course, have to bring the employees on line. There are going to be approvals, there's going to be a necessity for dealing with the HR department on that.

If they could find a way of entering into that process or workflow around approvals through the mobile device, through these interfaces, do it quickly, and get it automated, I think that might be a very interesting opportunity.

Critical opportunity

Swete: More than interesting. It's critical. If you look at a lot of the managers I know, it's just binary. It's the difference between the using the system and not using the system. Even though we have a very accessible and very user-friendly user experience, we're talking about busy people who use what they use.

They use their portals of information to get access to information on the Web and they use their intelligent cellphones now to get access to information and even to update information.

If we can provide this functionality on the devices that they use, then they will use this. If we can't, then they will get someone else to enter their information into the system, and that's not the way we want it. So, absolutely, it's tying in another theme of Workday as wider access to our applications. We think it's absolutely essential to increase the value that you get from your enterprise applications.

Gardner: One of my favorite sayings is that convenience is the killer application and I think that's what you bring in here, right?

Swete: Absolutely. Mobile is a good place to start with that, but we won't stop there. There is a lot of information that is currently presented well within Workday, but it could be presented just as well within a gadget and someone else's portal. We'll be looking for that opportunity as well. It's a way to put information in front of the people who need to see it without having to draw them into your user experience.

Gardner: Let's move into some examples, perhaps some anecdotes, about how this is being used in the real world and then we'll also look into the crystal ball and see what you might have in mind for the future. Do you have any case studies or anecdotes of how any of your customers have actually put into use, or are getting returns and paybacks from, some of the benefits that we have talked about that have their underpinnings in the architecture?

Swete: I have a couple of examples I think might help. I can go from general to specific. The one general example that I always like to quote is a real payoff and testament to the new architectural approach and combination of the new architectural approach and the use of SaaS in terms of delivering the business services. All Workday customers are always on the current version, and this is very, very different for the world of business enterprise application.

Workday delivers new updates three times a year, and our growing customer population whether they are in implementation or in production, comes up on each of those updates within the six weeks following the initial release of the update.

We did four updates last year. We will do three this year. Just this past month, we have been bringing our user population up to Update 7. If we have had this call two months ago, the only update that anyone would have known about in the customer population was Update 6. Sitting here today, Update 6 is a complete memory, and the only update that anyone knows about, whether they are just implementing on Workday or whether they've been in production for three years, is Update 7. That just brings a whole new opportunity to the customer conversation.

Special offer: Download a new white paper on Workday's latest update to System 7.

When we're talking to customers, we're all looking at the same code line and the same set of functionalities. We don't have to think about what version, what tech stack are you on, what version of your database are you on, or what version of Workday are you on? When they are asking a question about a new feature they want, we're all looking at the same feature set. It really helps to facilitate the conversation about what new features might be appropriate for the coming update, to say nothing of what it does for the cost profile of supporting these guys.

We have our stack, and that's it. They all run on it and we're able to keep them current on it. That's the benefit from the vendor side. From the customer side, the benefit they get is new functionality delivered to them, and all the manual work and data conversion work that is still necessary is done by Workday and not by them. They can focus on how they want to implement this new functionality on what timetable.

That's data point number 1 for some of the payback to the new architecture. A lot of us who came from the enterprise space are really impressed and pleasantly surprised at how rapidly we can move production customers and implementing customers forward.

The psychological effect

Gardner: I suppose there's a psychological effect there as well on the receiving end of these services. All of a sudden, you're getting more and better, but you didn't have to pay more and you didn't have to go through the pain of implementation and debugging. In adopting new things in the past there was an actual penalty for adoption, whereas the SaaS model gives you all rewards. It's like "hit me again," right?

Swete: It's absolutely that. As a customer, you avoid the high cost of having to set up QA environments and duplicate environments on-premise, having to deal with data conversions, or having to deal with installation instructions for the software that you've got on your premise. That's all handled by the vendor.

The other thing that's not widely talked about that's also valuable, though, is this notion of chunking up how new features come. We're on a regular clock here. So, three times a year, you're going to get a set of new functionality from your vendor and you are going to get converted to it.

In the on-premise world, where the delivery is deferred, what's building up is just a larger set of functionality that's going to have to get consumed some time at very high cost. Part of the way we look at this is that the incremental approach is just much more cost effective for our customers and for us. We know that, because even in our brief history here, we've had an on-going conversation with our customers about how long we want the update periods to be.

Our first update period was seven months and we delivered a ton of functionality. It was harder for them to consume and harder for us to support the upgrade to it. Now, we’re on a really good balance, where we can have meaningful functionality, but come out in a way that can be readily consumed. I don't think -- well, I don't think, I know the on-premise world just does not work that way.

Gardner: Okay, a quick look to the future before we wrap up, Stan. What are some of the implications from what we've been discussing, perhaps in terms of BI, perhaps in terms of extended business processes, or more of this integration agility?

Swete: Let's focus on the last two. We talked about a lot of the value of leveraging new technology to deliver enterprise applications in a new way and then combining that with doing it from the cloud. That combination is going to profoundly change things going forward.

Today, it's a new option for enterprises to look at in terms of offloading some of the applications that they're trying to support in their existing environment. It's a vehicle for consolidating some of the complexity that you have into a single instance that can be managed globally if you have architected globally, as Workday has done.

As I see that playing out going forward, you'll see more vendors taking this approach and you'll see those vendors partnering. If you think about the combination of modern architectures and cloud-based modern architectures, what will happen when two vendors that have taken that similar approach start to partner in terms of integrated business processing is that the bar will get raised significantly for how tight that integration can become, how well supported it can be, and how it can functionally grow itself forward, without causing high cost and complexity to the consuming enterprise that's using both sides.

As I look in the future, I think enterprises will see an ecosystem of their major application providers be cloud-based and be more cohesive than a like group of on-premise vendors. Instead of having a collection of different architectures and different vendors all in their data center, what they will see is an integrated service from the set of providers that are integrating with Web services in the cloud.

Gardner: So, when the cloud model becomes the common denominator, it allows for a lot more, I don't know, co-existence collaboration but I suppose really just integrated processes.

Swete: It allows for a lot more integrated processes. The key thing with enterprise functionality is you're never done. The requirements are always changing, because business is always changing. What it allows for us is not only cloud-based integration, but the ability to change that integration without placing additional cost on your customer.

That's what's the key is, that you will be able to deliver enhancements to the integration between vendors without getting caught up in how that integration might have been deployed at customer A, versus customer B, versus C. That will allow the same kind of agility we've been talking about. That will allow integrated solutions, the cross-vendor integrated solutions, to keep pace with the change of business, which is absolutely not the case today.

Gardner: I want to thank the sponsor of this discussion, Workday, for underwriting its production and a special thanks to our guest, Stan Swete, CTO of Workday. I really appreciate your inputs, Stan.

Swete: Dana, thanks a lot.

Gardner: We've been learning about how IT architecture is destiny and how a SaaS provider's operations can mean more to its customers and simply lower costs and baseline delivery of services as a Web application. We have seen a multiplier effect, if you will, in terms of how new and additional productivity and agility benefits are gained from a modern architecture regardless of its location and ownership.

This is Dana Gardner, principal analyst at Interarbor Solutions. Thanks for listening and come back next time.

Listen to the podcast. Download the podcast. Find it on iTunes/iPod and Podcast.com. Learn more. Sponsor: Workday.

Special offer: Download a new white paper on Workday's latest update to System 7.

Transcript of BriefingsDirect podcast on how Workday's SaaS delivery model for human capital management applications provides better business intelligence and architectural advantages to end users. Copyright Interarbor Solutions, LLC, 2005-2009. All rights reserved.

Enterprises Seek Ways to Exploit Web Application Mashups and Lightweight Data Presentation Techniques

Transcript of BriefingsDirect podcast on data mashups with IBM and Kapow.

Listen to the podcast here. Sponsor: Kapow Technologies.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today, a sponsored podcast discussion about the state of choice in the modern enterprise around development and deployment technologies.

These days, developers, architects and even line-of-business managers have many choices. This includes things like Web applications, software-as-a-service (SaaS), Services Oriented Architecture (SOA), RESTful applications, mashups, pure services off the Web, and pure services from within an Intranet or even the extended enterprise. We’re talking about RSS and Atom feeds, and, of course, there is a traditional .NET and Java going on.

We also see people experimenting with Ruby and a lot of use around PHP and scripting. The good news is that there are a lot of choices. The bad news is also that there are a lot of choices.

Some of these activities are taking place outside the purview of IT managers. People are being innovative and creative, which is good, but perhaps not always in the way that IT would like in terms of security and access control. These newer activities may not align with some of the larger activities that IT needs to manage -- which many times these days includes consolidation, unification, and modernization of legacy applications.

To help us weed through some of the agony and ecstasy of the choices facing application development and deployment in the enterprise, we have on the call, Rod Smith. Rod is Vice President of Internet Emerging Technologies at IBM. Welcome to the show, Rod.

Rod Smith: Thank you very much. It’s nice to be here.

Gardner: We also have Stefan Andreasen, the Founder and CTO of Kapow Technologies. Welcome to the show, Stefan.

Stefan Andreasen: Thank you.

Gardner: Let’s go first to Rod. We spoke last spring about these choices and how there are, in effect, myriad cultures that are now involved with development. In past years, development was more in a closed environment, where people were under control … white coats, raised floors, and glass rooms come to mind. But now it’s more like the Wild West. What have you been finding in the field, and do you see this as chaos or opportunity?

Smith: A little of both. In times of innovation you get some definite chaos coming through, but IT, in particular, and line of businesses see this as a big opportunity. Because of SOA and the other technologies you mentioned, information is available, and line of business is very interested in capturing new business opportunities.

Time to market is getting shorter, and getting squeezed all the time. So you’re seeing line of business and IT coming together around what they have to do to drive more innovation and move it up a couple of notches, from a business perspective.

Open standards now are very important to IT. Line of business, with mashups in particular, can use those types of services to get the information and create solutions they couldn’t do in the labs, when the propeller heads and others had to be involved five or 10 years ago.

Gardner: So we have dual or maybe multiple tracks going on. I suppose what’s missing is methodological and technical management. That’s an area where IBM has been involved for some time. Does IBM look at this as an opportunity?

Smith: A big opportunity. And you hit it on the head. The methodology here is very different from the development methodology we’ve been brought up to do. It’s much more collaborative, if you’re line of business, and it’s much more than a set of specifications.

Here is where we’re seeing people talk about building mashups. Usually they have a really good idea that comes to mind or something that they think will help with a new business opportunity.

Often the second question -- and we’ve seen a pattern with this -- is “Where is the data? How do we get to the data? Can IT open it up for us? Do line-of-business people have it in spreadsheets?” Typically, when it’s valuable to the business, they want to catalog it and put it together, so other people can share it. Finally, they do a mashup.

So methodology is one of the things we call a self-service business pattern. It starts with the idea, from a developer standpoint. "I really need to understand the business. I need to understand the time to market and the partnerships, and how information can be exposed." Then, they get down into some of the details. "I've got to do it quickly."

What we are seeing from an opportunity standpoint is that many businesses, when they see an opportunity, want a vendor to respond in 30 days or less, [and do more] within six months down the road. So that’s a challenge, and it is an opportunity. We think about tooling and middleware and services. How can we help the customer?

Gardner: Let’s go to Stefan. When you see these activities in the enterprise around mashups, SOAP, REST, HTML and XML, there’s an opportunity for bridging the chaos, but I suppose there’s also a whole new type of development around situational applications.

That is to say that, an opportunity exists to access content that hadn’t really been brought into an application development activity in the past. Can you tell us a little bit about what you’re seeing in the enterprise and how these new types of development are manifesting themselves?

Andreasen: Let me comment on the chaos thing a little bit. It’s important to understand the history here. At first, central IT worked with all their big systems. Line of business really didn’t have any access to IT or tools themselves, until recently when they got desktop tools like Excel.

This current wave is really driven by line of business getting IT in their own hands. They’ve started using it, and that’s created the chaos, but chaos is created because there is a need.

Now, with the Web 2.0 and the mashup wave, there is an acknowledgement of a big need here, as Rod also said. So it’s necessary to understand why this is happening and why it is something that’s very important.

Gardner: These end-users, power users, these line of business folks, they’ve been using whatever tools have been available to them, even if it’s an Excel spreadsheet. I suppose that gives them some productivity, but it also leaves these assets, data and content, lying around on hard drives in a fairly unmanaged perspective.

Can we knock two birds down with one stone in terms of managing this chaos in terms of the data, but also bring together some interface and application development benefits?

Andreasen: The worst thing would be to shut it down, of course. The best thing that’s happening now is acknowledging that line-of-business people need to do their own thing. We need to give them the tools, environments and infrastructure so they can do it in a controlled way -- in an acceptable, secured way -- so that your laptop with all of your customer data doesn't get stolen at the airport.

When we talk about customer data, we leap back to your earlier question about data. What are line-of-business people working with? Well, they’re working with data, analyzing data, and finding intelligence in that data, drawing conclusions out of the data, or inventing new products with the data. So the center of the universe here for this IT work is really dealing with data.

Gardner: SOA is one of the things that sits in the middle between the traditional IT approaches and IT development and then these newer activities around data, access, and UIs and using Web protocols.

I wonder if you think that that’s where these things meet. Is there a way to use an enterprise service bus (ESB) for checking in and out of provisioned or governed services? Is there a way that mashups and the ERP applications meet up?

Smith: The answer is yes. Without SOA we probably wouldn't have gotten to a place where we can think about mashable content or remixable content.

What you are seeing from customers is the need to take internal information and transform it into XML or RESTful services. There’s a good match between ESB things … [and] thinking about security and other pieces of it, and then building the Rich Internet Application (RIA) type of applications.

The part you touched on before is interesting, too. And I think Stefan would agree with me. One thing we learned as we opened up this content is that it isn't just about IT managing or controlling it. It’s really a partnership now.

One thing Stefan has with Kapow that really got us talking early was the fact that for Stefan’s content they have a freshness style. We found that same thing is very important. The line of business wants to be involved when information is available and published. That’s a very different blending of responsibility than we've seen before on this.

So thinking forward you can imagine that while you are publishing this, you might be putting it into a catalog repository or into services. But it also has to available for line of business now to be able to look at those assets and work with IT on when they should be available to business partners, customers and others.

Gardner: It’s interesting you mentioned the word "publish," and it’s almost as if we are interchanging the words "publishing" and "application development" in the sense that they are munging or overlapping.

Does that fit with what Kapow has been seeing, Stefan, that publishing and syndication are now a function of application development?

Andreasen: There are several sides to this question of which data you need, how to access it, how it is published, etc. One thing you are talking about is line of business publishing their data so other people can use it.

I split data into several groups. One is what I call core data, the data that is generally available to everybody in the company and probably sits in your big systems. It’s something everybody has. It’s probably something that's service-oriented or is going to be very soon.

Then there is the more specialized data that’s sitting out in line of business. There's a tendency now to publish those in standard formats like RSS, RESTful services, etc.

There's is a third group, which I call intelligence data. That's hard to find, but gives you that extra insight, extra intelligence, to let you draw a conclusion which is different from -- and hopefully better than -- your competitors’.

That’s data that’s probably not accessible in any standard way, but will be accessible on the Web in a browser. This is exactly what our product does. It allows you to turn any Web-based data into standard format, so you can access what I call intelligence data in a standard fashion.

Gardner: This is the type of data that had not been brought into use with applications in the past?

Andreasen: That is correct. There is a lot of information that’s out there, both on the public Web and on the private Web, which is really meant to be human-readable information. You can just think about something as simple as going to U.S. Geological Service and looking at fault lines of earthquakes and there isn't any programmatic API to access this data.

This kind of data might be very important. If I am building a factory in an earthquake area, I don’t want to buy a lot that is right on the top of a fault line. So I can turn this data into a standard API, and then use that as part of my intelligence to find the best property for my new factory.

Smith: When we talk of line of business, it’s just not internal information they want. It's external information, and we really are empowering these content developers now. The types of applications that people are putting together are much more like dashboards of information, both internally and externally over the Internet, that businesses use to really drive their business. Before, the access costs were high.

Now the access costs are continuing to drop very low, and people do say, "Let’s go ahead and publish this information, so it can be consumed and remixed by business partners and others,” rather than thinking about just a set of APIs at a low level, like we did in the past with Java.

Gardner: How do we bring these differing orbits into alignment? We've got people who are focused on content and the human knowledge dimension -- recognizing that more and more great information is being made available openly through the Web.

At the same time, we have this group that is API-minded. I guess we need to find a way of bringing an API to those folks who need that sort of interface to work with this data, but we also need for these people to take this data and make it available in such a way that a developer might agree with it or use it.

How does Kapow work between these constituencies and make one amenable to the other? We're looking for a way to bind together traditional IT development with some of these “mashupable” services, be it internal content or data or external services off of the Web.

I wonder what Kapow brings to the table in terms of helping these two different types of data and content to come together -- APIs versus content?

Andreasen: If you want to have automatic access to data or content, you need to be able to access it in a standard way. What is happening now with Web Oriented Architecture (WOA) is that we're focusing on a few standard formats like RESTful services and on feeds like RSS and Atom.

So first you need to be able to access your data that way. This is exactly what we do. Our customers turn data they work with in an application into these standard APIs and feeds, so they can work with them in an automated way.

It hadn’t been so much of a problem earlier, maybe because there wasn’t so much data, and people could basically cut and paste the data. But with the explosion of information out there, there's a realization that having the right data at the right time is getting more and more important. There is a huge need for getting access in an automated way.

How do line-of-business people work with the data? Well, they work with the data in the application interface. What if the application interface today is your browser?

Kapow allows the line-of-business people to automatically access data the way they worked with it in their Web browser.

That’s a very powerful way of accessing data, because you don't have to have an extra level of IT personnel. You don't have to first explain, "Well, this is the data I need. Go find it for me." And then, maybe you get the wrong data. Now, you are actually getting the data that you see the way you want.

Gardner: Another aspect to this is the popularity of social networking and what's known as the "wisdom of crowds" and wikis. A lot of contributions can be brought into play with this sort of gray area between content and publishing, different content feeds, and exposure and access and the traditional IT function.

Wikis have come into play, and they have quite a bit of exposure. Maybe you have a sense of how these worlds can be bridged, using some of what's been known as social networking?

Smith: Software development now is much more of a social networking activity than an engineering activity. At IBM, we have Blog and Wiki Central, where people use wikis to get their thoughts down and collectively bring an idea about.

Also at IBM, we have Innovation Jam, which we hold every year, and which brings in hundreds of thousands of people now. It used to be just IBM, but we’ve opened it up this last year to everyone, friends and family alike, to come up with ideas.

That part is great on the front end. You then can have a much better idea of what the expectations are, and what a user group wants. They're usually very motivated to stay in the loop to give you feedback as you do development.

The big part here is when it comes to doing mashups. It's the idea that you can produce something relatively quickly. With IBM’s QEDWiki, we like the idea that someone could assemble an application, wire it together in the browser, and it has the wiki characteristics. That is, it's stored on the server, it’s versioned as to enterprise characteristics, and it’s sharable.

It’s a key aspect that it has to be immediately deployable and immediately accessible by the folks that you are networking with.

That relates to what Stefan was saying and what you were asking about on how to bridge the two worlds of APIs and content. We're seeing now that as you think about the social networking side, people want the apps built into dashboards.

The more forward-thinking people in IT departments realize that the faster they can put together publishable data content, they can get a deeper understanding in a very short time about what their customers want.

They can then go back and decide the best way to open up that data. Is it through syndication feeds, XML, or programmatic API? Before, IT had to guess usage and how many folks might be touching it, and then build it once and make it scalable.

We’re doing things much more Agile-wise and building it that way, and then, as a flip, building the app that’s probably 80 percent there. Then IT can figure out how they could open up the right interfaces and content to make it available broadly.

Gardner: Stefan, could you give us some examples of user scenarios, where Kapow has been brought in and has helped mitigate some of the issues of access to content and then made it available to traditional development? Is there a way for those folks who are perhaps SOA-minded, to become a bit more open to what some people refer to as Web-Oriented Architecture?

Andreasen: One example that was mentioned in The Wall Street Journal recently in an article on mashups. It was on Audi in Germany. They are using our product to allow line of business to repurpose existing Intranets.

Let’s say that a group of people want to take what’s already there, but tweak it, combine it, and maybe expose it as a mobile application. With our tool, they can now do that in a self-service way, and then, of course, they can share that. What’s important is that they published this mini-mashup into their WebSphere portal and shared it with other people.

Some of them might just be for individual use. One important thing about a mashup is that an individual often creates it. Then it either stops there, because only that individual needs it – or it can also grow into company-wide use and eventually be taken over by central IT, as a great new way to improve performance in the entire company. So that shows one of the benefits.

Other examples have a lot to do with external data -- for example, in pricing comparisons. Let’s say I'm an online retailer and suddenly Amazon enters the market and starts taking a lot of market share, and I really don’t understand why. You can use our product to go out and harvest, let’s say, all the data from digital cameras from Amazon and from your own website.

You can quickly find out that whenever I have the lowest price, my product is out of stock -- and whenever I have a price that's too high, I don’t sell anything. Being able to constantly monitor that and optimize my prices is another example.

Another very interesting piece of information you can get is vendor pricing. You can know your own profit margin. Maybe it’s very low on Nikon cameras. You see that eBay is offering the Nikon cameras below even your cost as the vendor. You know for sure that buyers are getting a better deal with Nikon than you can offer. I call this using data to create intelligence and improve your business.

Gardner: All this real-time, updated content and data on the Web can be brought into many aspects of what enterprises do -- business processes, purchasing, evaluation, and research.

I suppose a small amount of effort from a mashup could end up saving a significant amount of money, because you’re bringing real-time information to those people making decisions.

How about you on your side, Rod? Any examples of how these two worlds -- the peanut butter and chocolate, if you will -- come together for a little better snack?

Smith: I’ll give you a good one. It’s an interesting one we did as a technology preview with Reuters and AccuWeather. Think about this again from the business perspective, where two business folks met at a conference and chit-chatted a bit.

AccuWeather was talking about how they offer different types of services, and the Reuters CTO said, "You know, we have this commodity-shipping dashboard, and folks can watch the cargo go from one place to another. It’s odd that we don’t have any weather information in there.” And the question came up very quickly: "I wonder how hard it would be to mash in some weather information."

We took one of their folks, one of mine, and the person from AccuWeather. They sat down over about three or four hours, figured out the scenario that Reuters was interested in and where the data came from, and they put it together. It took them about two weeks, but altogether 17 hours -- and that’s over a beer.

So it was chocolate and nuts and beer. I was in pretty good shape at that point. The interesting thing came after that. When we showed it to Reuters, they were very thrilled with the idea that you have that re-mixibility of content. They said that weather probably would be interesting, but piracy is a lot more interesting. "And, by the way" -- and this is from the line of business person -- "I know where to get that information."

Gardner: Now when you say "piracy," you mean the high seas and the Jolly Roger flying up on the mast -- that kind of thing?

Smith: That’s it. I didn’t even know it existed anymore. In 2006, there were 6,000 piracy events.

Gardner: Hijackings at sea?

Smith: Yes.

Gardner: Wow!

Smith: I had no idea. It turned out that the information was a syndication feed. So we pulled it in and could put it on a map, so you could look at the different events.

It took about two hours, but that’s that kind of dynamic now. The line-of-business person says, "Boy, if that only took you that much time, then I have a lot of ideas, which I’ve really not talked about before. I always knew that if I mentioned one more feature or function, IT would tell me, it takes six more months to do."

We've seen a huge flip now. Work is commensurate with some results that come quickly. Now we will see more collaboration coming from IT on information and partnerships.

Gardner: This networking-collaboration or social interaction is really what’s crafting the new level of requirements. Instead of getting in line behind 18 six-month projects, 12 to 20 hours can be devoted by people who are perhaps on the periphery of IT.

They're still under the auspices of what’s condoned under IT and make these mashups happen, so that it’s users close to the issues, close to where the creativity can begin that create a requirement, and then binds these two worlds together.

Smith: That’s correct, and what is interesting about it is, if you think about what I just described -- where we mashed in some data with AccuWeather -- if that had been an old SOA project of nine or 18 months, that would have been a significant investment for us, and would have been hard to justify.

Now, if that takes a couple of weeks and hours to do -- even if it fails or doesn’t hit the right spot -- it was a great tool for learning what the other requirements were, and other things that we try as a business.

That’s what a lot of this Web 2.0 and mashups are about -- new avenues for communication, where you can be engaged and you can look at information and how you can put things together. And it has the right costs associated with it -- inexpensive.

If I were going to sum up a lot of Web 2.0 and mashups, the magnitude of drop in “customization cost” is phenomenal.

Gardner: And that spells high return on value, right?

Smith: That’s right.

Gardner: How do you see this panning out in the future? Let’s look in our crystal ball. How do you see this ability of taking intelligence, as you call it, around the content, and then the line-of-business people coming in and making decisions about requirements, and how they want to tune or see the freshness of the content?

What’s going to happen in two or three years, now that we are bringing these things together?

Andreasen: There will be a lot more of what Rod just described. What Rod just mentioned is an early move, and a lot of companies aren't even thinking along these lines yet. Over the next one or two years, more people will realize the opportunity and the possibility here, and start doing it more. Eventually, it’s going to explode.

People will realize that getting the right data and the right content at the right time, and using that to create more intelligence is one thing. The other thing they’ll realize is that by networking with peers and colleagues, they'll get ideas and references to new data. All of these aspects -- the social aspects, the data aspect and the mashup aspect -- will be much more realized. I think it’s going to explode in usage.

Gardner: Any last thoughts, Rod, from where you see these things going?

Smith: Well, as we see in other technologies moving through from an SOA perspective, this is a great deal about cultural change within companies, and the technology barriers are coming down dramatically.

You don’t have to be a Java expert or a C# expert. I'm scary enough to be able to put together or find solutions for my own needs. It’s creating a way that line-of-business people are empowered and they can see business results quickly.

That also helps IT, because if the line of business is happy, then IT can justify the necessary middleware. That’s a fundamental shift. It's no longer an IT world, where they can promise a solution to the line of business 12 to 18 months down the road.

It’s much more of, "Show me something quickly. When I’ve got the results in my hand -- the dashboard -- then you can explain what I need to do for IT investments and other things."

It’s more collaboration at that point, and makes a lot of sense on governance, security, and other things. I can see the value of my app, and I can actually start using that to bring value to my company.

Gardner: I suppose another important aspect culturally is that part of SOA’s value is around reuse. These mashups and using this content in association with other different activities, in a sense promotes the notion of reuse.

You're thinking about, "How can I reuse this mashup? How can I extend this content, either off the Web or internally, into new activities?" That, in a sense, greases the skids toward more SOA, which I think is probably where IT is going to be heading anyway.

Smith: Well, what’s fun about this, and I think Stefan will agree, is that when I go to a customer, I don’t take PowerPoint charts anymore. I look on their website and I see if they have some syndication feeds or some REST interfaces or something.

Then I look around and I see if I can create a mashup of their material with other material that hadn’t been built with before. That’s compelling.

People look and they start to get excited because, as you just said, they see business patterns in that. "If you could do that, could you grab this other information from so-and-so?"

It’s almost like a jam session at that point, where people come up with ideas. That’s where we will see more of these examples. Actually, a lot of our stuff is on YouTube, where we had a retail store that wanted to see their stores on Google Maps and then see the weather, because weather is an important factor in terms of their businesses.

In fact, it’s one of the most important factors. What we didn’t realize is that very simple pattern -- from a technology standpoint it didn’t take much -- held up over and over again. If it wasn’t a store, it was banking location. If it wasn’t banking locations, it was ships. There were combinations in here that you could talk to your businessperson about.

Then you could say to the technologist or a developer, "What do I have to do to help them achieve that?" They don’t have to learn XML, Web objects, or anything else, because you have these SOA interfaces. It helps IT expand that whole nature of SOA into their enterprise.

Andreasen: One thing that's going to happen is that line-of-business people are getting a lot of great ideas. If I am working with business problems, I constantly get ideas about how to solve things. Usually, I just brush it away and say, "Well, it will be cool to have this, but it’s impossible."

They just don’t understand that the time from idea to implementation is dramatically going to go down. When they start realizing this, there is hidden potential out on the edge of the business that will now be cut loose and create a lot of value. It’s going to be extremely interesting to see.

Smith: One of the insights we have from customers is that mashups and this type of technology help them to visualize their SOA investments. You can’t see middleware. Your IT shop tells you what’s good, they tell you they get flexibility, but they want to be shown results -- and mashups help do that.

The second part is people say it completes the "last mile" for SOA. It starts to make a lot of sense for your IT shop to be able to show how the middleware can be used in ways it wasn’t necessarily planned for.

The big comment we hear is, "I want my content to be mashable or re-mixable." We figured out that it’s very much a SOA value. They want things to be used in ways they weren't planned for originally. Show me that aggressive new business opportunity, and you make me a very happy person.

Andreasen: Probably one thing we will see in companies is some resistance from the technologists, from central IT, because they are afraid they will lose control. They are afraid of the security issues etc., but it will probably be like what we've seen with company wikis.

They're coming in the back door in line of business and eventually the companies buy the company-wide wiki. I think we'll see the same thing with mashups. It will be starting out in line of business, and eventually the whole company understands, "Well, we have to have infrastructure that solves this problem in a controlled way."

Some companies have very strict policy today. They don’t even allow their line-of-business pros to write macros in Excel. Those companies are probably the ones that will be the last ones discovering the huge potential in mashups.

I really hope they also start opening their eyes that there are other roles for IT, rather than just the big, central system that run your business.

Gardner: Well, great -- thanks very much for your insights. This has really helped me understand better how these things relate and really what the payoff is. It sounds compelling from the examples that you provided.

To help us understand how enterprises are using Web applications, mashups, and lightweight data presentation, we’ve been chatting today with Rod Smith, Vice President of Internet Emerging Technologies at IBM. I really appreciate your time, Rod.

Smith: Thank you.

Gardner: And Stefan Andreasen, the Founder and CTO of Kapow Technologies. Thanks for joining, Stefan.

Andreasen: It’s been a pleasure, Dana.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions, and you've been listening to a BriefingsDirect. Thanks for listening and come back next time.

Listen to the podcast here. Sponsor: Kapow Technologies.

Transcript of BriefingsDirect podcast on data mashups with IBM and Kapow. Copyright Interarbor Solutions, LLC, 2005-2008. All rights reserved.

Analysts Debate IT Management and Monitoring Needs for the SOA Era

Edited transcript of SOA management trends and analysis discussion with Interarbor Solution's Dana Gardner and ZapThink's Jason Bloomberg, recorded before a live audience at the Harvard Club of Boston on Sept.14, 2007.

Listen to the podcast here. Sponsor: Tidal Software.

Welcome to a special BriefingsDirect presentation, an IT industry analyst panel podcast created before a live audience at the Harvard Club of Boston. Our sponsored discussion centers on the role of management on Service Oriented Architecture (SOA) use and operations.

Our panelists consist of Jason Bloomberg, managing partner and senior analyst at ZapThink, and Dana Gardner, president and principal analyst at Interarbor Solutions. Moderating the discussion is Martin Milani, chief technology officer at Tidal Software. The in-depth presentations from the analysts are followed by questions from the live audience.

Listen as these SOA experts explore how IT management will evolve in the world of service-based applications. They delve into issues of new standards, how SOA demands that performance management and change management augment and elevate the role of systems management, and how the integrity of services delivery requires a deep and wide approach to management in total across a services lifecycle.

Now, let's hear from our moderator, Tidal Software's CTO Martin Milani.

Martin Milani: Thank you. I guess it's no secret to anyone that SOA has finally arrived, and that SOA deployments are increasing rapidly -- and far more mission critically -- in the past couple of years. It's one of the fastest-growing segments of the software industry as a whole.

So, with that I want to see if the analysts could share some of your thoughts on the industry, and then some of the challenges with SOA in general. Jason?

Jason Bloomberg: Well, thanks a lot, Martin. It’s great to be here. I’d like to start with the definition of SOA for a level-set. SOA is essentially an approach to organizing IT resources to better meet the changing needs of the business. Fundamentally, it’s an architectural approach, a set of best practices, for leveraging IT in a flexible way.

The core business motivation for SOA in most organizations is business agility, to be able to respond to changing business requirements and to leverage change for competitive advantage as well. As a result, one of the key challenges, if you are looking to architect your IT organization in terms of flexible services to meet this business agility benefit, is being able to create, manage and evolve these services over time.

One of the core challenges we’ll be talking about today is loose coupling, where you want to build services that you can control and manage independently of the consumers of those services. That’s a key part of the business agility benefit of SOA. That’s how you would actually achieve business agility in practice.

In turn, the way you achieve loose coupling is through management, and that’s something we’ll be talking about as well. Management then becomes the critical enabler for loose coupling, which is the critical enabler for business agility. That’s how it all fits together.

Milani: Dana?

Dana Gardner: Thanks Martin, and thanks to Tidal Software. I agree with Jason. I think also that SOA has some catalytic implications for companies that begin a journey toward this architecture. It can foster change, and perhaps also benefit agility, but if they have not done their homework, if they are an organization that’s very much in silos, both in technology as well as practices, there could be a lot of risk involved.

There's going to be a period of time, where people look at SOA and find that the opportunities are going to depend on how they’ve done their preparation. We've had a lot of work toward service enablement of data, cleansing data, and putting it into a form in which it can be delivered across multiple applications and processes.

We’ve seen companies begin their journey toward better integration that’s open, that fosters interoperability, and we’ve seen management, but mostly on the level of "speeds and feeds," of how to make sure that the trains run on time, when it comes to delivery of data, whether transactions are going to be fulfilled in a timely manner and if application performance is maintained.

But management for SOA needs to go a step further and take into consideration many systems and interdependencies -- perhaps involving services coming from outside the organizational boundaries, be it partners across the supply chain, even hosting organizations and commercial-services providers.

So management is going to be, as a topic, very important to SOA in a traditional sense, but also management in a new sense. If SOA benefits technology, as well as the business goals and objectives and agility, then business management, IT management, and SOA management need to have some commonality.

Relying on individual people -- "wetware," if you will -- to bind these together and to hand off one to another isn’t going to scale. It’s going to be expensive. So I'm hoping that management is propelled through SOA into a more advanced concept that bridges a gap between organizational management, IT systems management, and the management of services themselves.

Milani: It sounds like SOA is affecting the application assistance management landscape as we know it. Would you agree with that?

Bloomberg: It definitely is. The key thing to keep in mind about SOA in this context is that services are an abstraction. That is, they help to provide flexibility to the business, but they don’t actually simplify the underlying technology. Many architects are a bit surprised by this, that SOA doesn't make their jobs easier or make the job of IT any easier. If anything, it’s more complex. There's more of a challenge for IT to meet the business requirements for flexible, agile, composable, and loosely coupled services. As a result, you have this need for the IT organization to rise to the challenge of services.

This is especially true in the management area because the services essentially have to behave as advertised. These are contracted interfaces to various application functionality of data across the organization. They have to do what they're supposed to do.

That's the core of the loose coupling. So, any consumer can come along and leverage a service, and it does what it is supposed to do. If there is a problem with the service, then it’s not loosely coupled. The core challenge is essentially saying, "Well, we want to make sure these services do what they're supposed to do." That’s fundamentally a management challenge.

Gardner: The performance of applications has been problematic, particularly the transition from mainframe to client-server and then to the Web. Waiting for an application performance issue to crop up and then chasing down the core problem has unfortunately been the norm. I'm not sure that that’s going to continue to be possible.

It’s never been the preferred way -- firefighting as the means to application performance. When you take this to the step of decoupled applications or services, you recognize that so many different systems are now supporting processes. It's not just a single model of a stack of support and platform beneath a specific application. You need to start working toward a predictive, analytical approach to the management of performance.

The implications for those dealing with applications is that you are going to service-enable those applications, decouple, and decompose them into essential core services, and then repurpose them by cross-compositing processes. What is that going to do to you, if you think you are going to go to firefighting mode when you have performance issues? It’s simply not going to work.

You need to rethink management and support, and you need to try to get proactive in how systems will be supported to head off performance issues and create insurance policies against blackouts, brownouts or other snafus. SOA is really a catalyst toward a different approach to the management and support of the services.

Bloomberg: That’s an important point. In the context of SOA, we're thinking in terms of business processes that are implemented by composing services. What’s happening here is that the definition of an application is beginning to shift, especially from the perspective of the business.

The business doesn’t want to think of an application as some big huge piece of software that costs them a lot of money and introduces a lot of risk, that doesn’t directly meet business needs, that’s not what they are asking for, but is what they’ve had to buy because it's what was available.

In the context of SOA, though, an application is what you do with the services. You compose them into composite applications or service-oriented business applications that implement processes in a flexible way to meet the needs of the business as they continue to evolve.

From the business perspective, this notion of application is whatever you are applying the services to. The root word of "application" is to apply something, and, as a result, the management challenge, especially the application management challenge, in particular, is an entirely different set of challenges.

It’s not saying, "Well I have this big enterprise application, which is some monolithic suite. I need to manage that." If you have those, you still need to manage those. So, that’s not going away, but SOA introduces this whole new concept of flexible, composable business process-based applications, which now have to be managed as compositions of services.

Services are obstructing functionality across various systems and various applications, and the management challenge becomes immensely more complex. If anything stops working, then the whole thing falls apart. Management now becomes even more important. You can’t just have a firefighting approach, and when there's a problem, you send an alert to the poor systems administrator, who has to wake up in the middle of the night to fix it. That just not going to cut it anymore.

Gardner: One more factor in this is that we are not just talking about SOA in a vacuum. We're not just moving toward SOA or services-enablement. We are also dealing with application modernization, pulling them off older more expensive platforms and putting them on standards-based or commodity platforms. We're talking about virtualization where we are going to create containers and try to get higher utilization and efficiency rates off of our underlying investment for the support of these services.

We're talking also about business continuity issues where we are going to try to have replication of services in the event of disruptions, be it natural disasters or otherwise.

So when you think about SOA, it’s happening at a time when there are many other IT mega trends under way, and the management needs to be considered within that context as well.

Milani: Obviously, SOA is a disruptive event. There is a radical change in the architecture of the applications, as we know them. There are far more points of failure, far more pieces of the application that could reside across separate systems, separate technologies and perhaps across enterprise silos.

So the traditional management approach has been used for the past 15-20 years, and is very client-server centric. Can these traditional monitoring and management approaches handle the SOA deployments of today and tomorrow? Dana?

Gardner: You're not going to throw out the baby with the bath water. You're going to continue to leverage your previous investments. Just as we’ve seen with storage and application deployment strategies, it’s about elevating to a higher abstraction -- perhaps through metadata, perhaps through standards -- so that you can increase outputs qualitatively and quantitatively from your systems.

Therefore, you can offer a management overview, whether it’s a dashboard, or even the equivalent of screen scraping one management view with another. At least, you can assimilate them in a such way that you can start to get a total view, a comprehensive view of systems.

I do think that you are going to continue to leverage existing management approaches. I'm sure that the older-line management vendors are going to continue to augment and support more advanced processes or more complex interactions between elements of infrastructure and applications.

It needs to be looked at not just as management of discrete parts, not just trees within the forest that each stand on their own -- but the forest itself. I'd like to see that get to the point where it becomes something that can be assimilated further than just the systems -- with the business objectives as well.

Furthermore, one of the things that’s been unfortunate about systems management up until now is that it tends to be binary -- on or off, green- or red-light, it’s working or not working. SOA is going to require more of a blended approach to management.

That is to say, you are going to want to tune how your applications and services are delivered, perhaps to live up to service level agreements, or perhaps so that you can give priority to certain data, application, or services streams over others. You're going want to be able to fine tune, rather than just throw a switch on or off. That’s going to require a different level of management. It’s really about leveraging the old, finding ways to assimilate and then put a more operator- or policy-driven -- perhaps even automated -- approach on top of it.

Bloomberg: It's interesting that you say SOA is disruptive, because SOA is disruptive -- but not in the way you might think. It’s not really that disruptive on the technology side. In fact, from the technology perspective, SOA helps leverage existing technologies. It has a heterogeneity story that says, "Well, you don’t have to replace. You can leave and abstract as needed."

So, if SOA is architected properly, it doesn’t have to be disruptive on the technology side, but it is most definitely disruptive on the organizational and political-cultural side, on the human side, because what SOA tells the IT shop in particular, is that we can’t do business as usual.

We can’t simply have a siloed approach -- here are the operations people that run this, here are the Windows people that do this, here are the application people that do this. That’s not going to work anymore. In order to think about services properly and to get the benefit of SOA at all, you have to think in a more cross-functional, comprehensive, architected, best-practices way about how IT does what it's supposed to do. How IT is going to meet with the needs of the business is shifting in a broad way, and that’s how SOA is disruptive.

When it comes time to think about management at all levels -- whether it’s networks, systems, application management or the business service management -- these are being reworked, because you can’t think of any of them in the narrow silo.

It’s like saying, "Well, I have this application; I have to manage this application. I have my TCP/IP network; I have to manage the network." You still have those technologies and you still have to manage them, but now we also have this whole notion of services cutting across different parts of the IT shop, meeting different needs of different parts of the business.

A single service might abstract multiple databases, multiple underlying applications, and multiple platforms. That same service might serve multiple lines of business and might serve the needs of internal as well as external users. So, in that context, what it means to do management is being entirely disrupted by the service-oriented approach.

Milani: Traditionally, the way people have looked at the Web services in SOA management today, has been to manage or monitor the interfaces, which are actually an interface to a runtime environment that holds some sort of service.

What do you think is required to deeply understand how to monitor these services beyond what the interfaces are doing, and finding out what’s under the tip of the iceberg? Dana?

Gardner: Well, with SOA you need to gather information about your systems both deeply and broadly: deep and wide. You can already get a fire-hose of data from your systems, log files, and agent- and agentless-based approaches already on the market. You get a ton of data.

It’s working with that data in the context of a horizontal business process that’s the hard part. The type of applications that we’re going to be seeing in the future, as Jason mentioned, cutting across different aspects of the organization, are going to require redundancy. If one aspect of a process goes down, that’s the weak link in that chain, the whole chain could be at disadvantage.

In the past, you might have one application down, but people could go off and do another task, because that mainframe would be backup at two o’clock. You can live with that. If your entire supply chain is disabled for a period of time, that’s a higher price to be paid. So, we're looking at a different level, and I don’t think we’ve seen the solution yet.

Bloomberg: It’s important to note that in the context of SOA, monitoring is just not good enough anymore. It’s not good enough to simply say, "We need to find out when we have a problem." It’s like, "Great, okay, now we know that Titanic has sunk." That’s not going to do it for you. You need to have preventative and corrective management. For a service to be loosely coupled, it has to behave the way it’s supposed to behave, regardless of what’s going on beneath the cover.

So it’s not good enough simply to say, "My management tool is telling me my service is down." You need to have a way of understanding that problems are brewing, that they are in the works. You have to be able to identify a potential problem before it actually impacts the behavior of the service and you need to take corrective action, ideally before the consumers of the service are impacted.

This is a high bar to set, an enormous challenge, and it’s not likely anybody is going to be able to do this perfectly. There is nothing perfect in this world and there will always be instances where there are some problems that a preventative, active management tool won’t be able to catch, but it’s critical for the architect to plan for this level of management, preventative corrective management.

That’s something the architect has to think of ahead of time, when they plan how they are going to build and implement those services. You can’t let it go for later. It’s not like you say, "We're going to launch our software and then let operations deal with management." There has to be something that’s part of your initial thinking when you are putting together the plan for your SOA.

Gardner: Fortunately, the way we’ve seen SOA evolve in the marketplace has been more on the crawl, walk, run basis. People aren't going out there saying, "We're going to switch over to SOA on Monday and all systems will be services." That’s just not the way.

There is an opportunity to learn as you go to encounter problems and then to put into place the management feedback and create the feedback loop into the remediation. Another important aspect of this is to start finding commonality between pre-production and post-production when it comes to applications and services development.

We're beginning to see some products in the market that try to take data that’s collected in the development phase and make it available to post-production to start feeding a loop of communication. So, when something is amiss in production, you can not only look to what is at issue in the systems and operation, but also look to how could we make this service or application better. So, the requirements for service and application are being impacted by operations.

For a long time, there was a significant wall between these facets of IT. It’s still very problematic, but if we can create a management feedback loop, so that -- as we get into faster iterations of development, when the test-debug cycle and build cycle is faster due to agile and lean practices -- we can start saying, "Let’s find out what’s going on in the field when something is wrong."

Do we just throw more hardware at it? Do we just add more servers? Or, do we say, "Can we actually design and engineer the service to be better," and then make that happen with a matter of weeks or months?

Milani: It sounds like the playing field again is far more complicated and complex with SOA, and is going to get more complicated from here on. You have business processes that go across multiple technology silos, multiple enterprise silos, and probably to other enterprises. So, you have different constituents, different organizations, different groups, and many different types of technologies.

This area was pretty hard to monitor and manage in even a simple three-tier architecture. Now, you have an N-tier architecture which you could almost call an "N-to-the-N-tier" architecture. Now, you have different types of services that you could be calling on, depending on the quality of service and security, trust and policy, and so on. So, it’s a different game. If you were to fast-forward five years or seven years, what do you think monitoring a management system, which could address all the issues we just talked about, would look like?

Bloomberg: It’s important to understand that management is not a practice in isolation. It’s really part of a family of capabilities that includes governance and quality, as well. You mentioned policy, security, and a variety of issues. All these are part of the SOA infrastructure story. So, to answer the question where things are going over the next several years, it’s really maturing the family of SOA infrastructure capabilities, broadly speaking.

We like to call that the governance quality management, or GQM, suite, which handles design time as well as runtime issues, creation and communication of policies in the context of governance, management in the context of runtime. It's not just thinking of runtime, in and of itself, because runtime is only part of the story. We have the full lifecycle now with design time creation of services, as well as publication and discovery of services.

The runtime part, which is the current focus of management, as well as the change time part, where you reconfigure and recompose services essentially in runtime context without any underlying co-changes, that’s where we see a lot of the activity going on in a mature SOA environment, is at that change-time configuration, composition level. So more and more of what has to happen there, from the infrastructure perspective, is this pulling together of the quality management in governance roles into full lifecycle quality management governance suite and that’s what we see happening already today, with a lot of the products in the space.

Gardner: In five to seven years, we're going to continue to see an increase in the pressure on businesses to adapt in markets. We’ve got globalization well under way and it’s going to continue. We have lower barriers of entry to companies, where digital assets are made available. We are going much more from bricks to clicks. Therefore, companies need to adapt readily and they can’t go to their IT departments and say, “We need to change; how long will it take?” They need to say, “Here’s the change; implement it.”

Let’s take an example of something from recent business history. Let’s say you’re a manufacturer of toys and you're told that a portion of these toys are now in the field with toxic paint that is an violation of the regulations in your country. You need to take quick action. You need to find out through your supply chain where the problem is. You need to find new suppliers. You need to go back out to the field and conduct a recall, and coordinate this with your marketing and your PR department and your investor relations department, so that you don’t lose the entire value of your company overnight.

How are you going to do this? You might do this through your IT systems. So you’re going to need to examine what’s going on in your supply chain and find out where the problem is and stop it. You’re going to say, “We don’t want to just keep the trains running on time. We want to pick up the tracks and move them somewhere else.”

IT is going to have a great opportunity to become far more valuable to their parent organizations, to be the real partnership that they should be, through the exploitation of SOA and through the proper management of IT and processes.

To answer your question: The value of IT here can be much greater. It can be an enabler, not a cost center. It can be the way in which not only is information relayed about what’s going on, but can determine what we want to happen. We want to change that supply chain. We want to change that distribution, recall these products, get a list of every single product and every serial number, and we want to relay that to our sales force.

That sounds straightforward, but if you try to do that with a lot of IT systems today, you’re going to find yourself up there with the equivalent of mimeograph and crayons, doing it by hand -- and that’s just not acceptable. So in the future, a company’s very existence could be at stake if they don’t have agility in these processes.

Bloomberg: This is a really important point. SOA is really not optional. Companies that don’t get this right will suffer the consequences. They will suffer lawsuits and suffer a competitive disadvantage. They are going to go out of business. This is an important thing to keep in mind. IT is not playing around here. You can't say, "Maybe we will do SOA, if we can figure it out, or maybe we won’t. We’ll just do things the old way, where we are siloed and we keep on going."

If you keep doing things the old way, you are going to lose out one way or the other, whether it’s some sort of regulatory or competitive pressures, some disaster like lead paint, or credit card numbers getting out on the Internet. Whatever the problem is, it’s going to happen, and you have to be prepared. Organizations that don’t get this right are not going to survive.

Milani: SOA, again, is very business-driven. One could say it’s totally business-driven. It gives business agility and adaptability. Often people call it agile enterprise, real time enterprise. Inherently, SOA means real time integration across separate applications and separate technologies.

Something we touched on a little earlier was preventative automation and correction. Can you guys elaborate on that a little bit, why it’s very important. If you’re talking about an agile, active enterprise or real-time enterprise, then that automatically means that the management cannot be reactive and the management cannot be an afterthought. Dana?

Gardner: I recently read a book on SOA that I found very useful. Dr. Paul Brown is the author, and it’s called, Succeeding with SOA: Realizing Business Value Through Total Architecture. A lot of the book has to do with this concept of "Total Architecture."

It’s the architecture of the business and the architecture of IT having some relationship to one another. We can borrow from this concept and say that this should be a "total management" approach as well, so that we have this ability to manage processes and infiltrate the systems in such a way that it becomes a two-way street.

Preemption is really about latency. You can be reactive, if you can do it in a nanosecond. But, preemptive means that you’re going to come to a time where the lights are blinking and there is something wrong, and, before you’re totally out of business or some process is brought to its knees, you want to take remediation. It’s really about balancing the latency of reaction toward a point of preemption. That's very hard to do, however.

When we have system redundancy, we have all this log data. There are probabilistic approaches. There is looking at performance states that are normal, when you compare that over time, but when we get into this area, there are many unknowns, many more variables across the supply chain. When we are dealing with services that are coming from organizations that you don’t have authority or control over, it’s going to be a difficult approach.

I don’t have a quick answer to it. I do think that the latency issue needs to be addressed -- the amount of information that's shared. As we get toward this concept of total management, we need to bring in what incentives are being applied to people and systems.

Are we going to pay for systems based on a steady stream, are we going to start putting in incentives that penalize people for down performance, while paying them higher prices for greater performance? As we see services that are acquired on a subscription basis or a pay-as-you-drink basis, we are also going to start seeing a lot more monetized incentives around performance.

So whether the systems react or not, the price could be so high that you’re going to have to make the investments. I think economics and the concept of total management need to be brought to this.

Bloomberg: There is no magic wand here. Systems aren’t perfect. Software is never a 100 percent bug free, computers are never 100 percent operational, and networks are never up 100 percent of the time. There are always problems. There are problems on the business side as well. Secrets sometimes do get out, and products sometimes are defective and whatever it is, there are always problems. This is just the way of the world, the reality of life on earth.

How do businesses deals with this traditionally? Well, you hope there aren’t any problems, and you have certain plans in place, but fundamentally, when there are problems, you step away from your technology and you deal with it however the people can deal with it. So, you run around and try to fix the problem.

We are looking at SOA, saying, "We can do a bit better with our technology in terms of dealing with the reality that there will be problems." As far as agility, we want to be able to respond to change, including changes we don’t want. That include problems with systems, with the business, with regulation, competition, or global disasters -- whatever it is.

Instead of just saying, "We can’t rely upon our technology, when problems come along," we want to say, "Yes, we have a more flexible way of dealing with problems, even though we can’t predict what they are.” That’s part of the benefit of SOA, part of the agility benefit. We are dealing with unexpected change, including various problems.

Instead of just running around and not being able to use technology, we want to have a governance plan in place, saying “This is how we’re dealing with problems. Here is how the technology will rise to these challenges." And, we make it a matter of policy. So now, instead of just having to wing it when you have some sort of issue, there is an infrastructure in place for helping you deal with issues as they come about.

A key to this is the management challenge. As management technology improves, it is less and less about just monitoring stuff, and more and more about being able to deal with issues as a matter of policy, where your policy is in place for dealing with problems that you can’t predict -- and those are the most challenging ones. That’s what we see happening over time.

Technology is getting better at dealing with these unpredictable situations. A core part of what we need from agile IT is being able to deal with the unpredictable.

Milani: Great point. It dovetails to my next point, which is businesses are moving to standard policies and standard practices in the form of a business process. More and more, they are moving the way they do business into a well-defined practice of business processes. So, using technology such as BPEL and business process management systems, why wouldn’t the management of some of these technology and software resources look like how processes are handled on the business side of the equation?

Bloomberg: Actually, they might be. You want to think about business process in the context of whatever the business does, and that applies to IT as well as it does to the business. IT should be run as a business, as part of the business. So, it’s not like IT is some different world. It’s just part of the business, one of many divisions, and it has a certain role within the context of the business.

Management, to the business side, means a whole range of things, of which technology is only a portion. It means making sure that you have a management hierarchy within the organization. It means that your business is running properly and that the business process is running properly. All of this is part of what the business means by management.

Part of that is IT enabled and part of that is specifically IT centric, but from the business perspective, it’s not like they are drawing a line and saying, "This is the stuff we mean by management in the business context, and this is the stuff we mean by management in IT context," as though they were different things. There is a spectrum, and as we move to SOA, we’re seeing that IT is becoming more of an enabler of the business in many more flexible ways. So, the term "management" now becomes much more of a business-centric context, of which IT is now an enabler, as opposed to two different kinds of management, one for the business and one for IT.

Gardner: If you could meaningfully and successfully divorce IT from the organization, you would have seen a lot more outsourcing. Companies tried to outsource lots of aspects of their IT. Some are successful, but in many cases it was brought back in.

IT and business are intrinsically bound. It’s a competitive differentiator: The way in which you do IT better than your competitors. I don’t think we’re going to see a divorce here. We’re going to see more assimilation. But the management of IT is relatively new, compared to the practice of business management.

The modern corporation is over 100 years old. Mercantile economic activities are 600 years old. You can find other aspects even older than that as to how people are governed and operate as teams or individuals.

If we have things like Balance Scorecard, Six Sigma, Design for Manufacturability, Simultaneous Engineering -- these concept have evolved on the business side, and on the manufacturing side. We should expect to see the same aspects for IT, and perhaps have the same overview of a Balanced Scorecard on how the business is operating, as well as how the IT is operating -- someday, maybe even together.

What we're really talking about here is the maturation and the natural process for IT to become more like business, and not be off in its own corner. It takes time. These were complex things that happened very, very quickly. We used to call it Internet Time. It’s probably, in some respects, even faster now, when you think about Enterprise 2.0 Time.

These IT systems have evolved so quickly, and companies have implemented them in such a haphazard way with lots of different heterogeneity involved, that it’s natural that it’s going to take time. Ultimately, we're going to get to a maturation where IT catches up to the business. Then they can be governed very similarly.

Milani: Are there any standards required sooner than later? Is there any specific standard that you think can enable the deployment of these SOA environments?

Bloomberg: There are standards in the works; there is Web Services Distributed Management (WSDM) and WS-Management and a variety of others. There are some that are datacenter centric, others that are more network centric. But, by and large, I agree with Dana that this is still the early days in terms of management standards.

What’s happening in the management standards world is a pissing match between the big vendors. You have the Java guys wanting to this and then Microsoft guys wanting to do that, and nobody listens to them, because they can't agree with each other. So, they'll realize, "Hey, all of our customers are ignoring us. We'd better get our act together." It’s become this big political thing that’s just slowing whole thing down.

From the enterprise perspective, you don’t have to wait around for the vendors to grow up. You can get stuff done today. This isn’t going to stop you from being successful with SOA initiatives today. It might mean that two products you buy off the shelf might not interoperate as well you like. That has to be part of your plan. It might mean you have to come up with your own internal standards for the time being.

Lots of organizations are doing that as well, because the open standards are just not mature enough to do everything you wanted to do. But that doesn’t mean you can’t be successful. Interoperability is part of the story, but a lot of what you need to get SOA to work are other challenges that you can work on. You can get SOA business value.

Gardner: We need to see standards at a higher level. It's not just about systems level, and it’s not just about framework level, like Java or .NET. It’s really at the level of how the people who are responsible for a business process get a view into that entire process.

Ultimately in these organizations, we're not going to have people responsible for a server farm, a database, or an SAP implementation. They're going to be responsible for a order-to-bill capability. For those people to get a view into all of the aspects for which they are responsible, they're going to need a management view at a higher elevation. That’s where standards need to be applied, so that they can get that view, so that there is interoperability, sharing of data, and a canonical view of management.

Milani: We're almost out of time. I want to thank you for your time and quickly ask you if you want to close and sum up what you think people should consider, as they deploy more and more SOA implementations, and what they should consider with respect to management.

Bloomberg: SOA is not just about the technology that you can't buy from a vendor. It’s not something you buy. It’s something you do. It’s a set of best practices that are still relatively loosely defined.

You can be successful with SOA by taking it a step at a time, achieving real business value. You don’t have to set the bar so high that it becomes impossible. But, that being said, even if your SOA initiative is relatively modest and you have a few services in the context of that architecture, management is something that you have to think about very early on. In fact, you need to think about management with services even before you get to SOA.

If you just have a few services and they are not managed, then they could be redundant or they could be incompatible. Worst of all, they could be rogue services, services that are not on the radar of anybody who is running these services. They could expose confidential information or bring systems down. Who knows what the problem is?

A rogue service is one that you just have no idea what sort of damage it can cause. This is a real risk, because, if you’re building a Web service, in particular, it might just run over Port 80. It might just expose information over the firewall, and you have to think about this very early on in your services initiative, even before you get SOA.

As you get SOA, you have to think about all the services you already have and plan for loosely coupled services in the context of your architecture. All of that has to be done in the context of management, as well as governance and quality as well.

Gardner: As companies move closer to SOA, it forces them to grow up. It forces them to think across boundaries. In the past, complexity has forced companies to divvy up issues into small compartments, put a box around them, and assign people to that item of complexity.

But that has stifled the ability of interoperability and of addressing things holistically, of being fleet and agile. It’s made them brittle, has made them slower, and has made things expensive. SOA forces companies to start binding what happens in pre-production to post-production, what happens in an application with what happens in an infrastructure, and what happens on a service level from an outside provider to what happens as a shared service internally.

There are great risks if you try to do SOA without growing up, but there are super opportunities if it’s done properly. It can elevate IT as a function within the organization from being an inhibitor to the absolute enablement for new business and growth and opportunity.

If you’re inside an IT department, if you are a vendor, if you’re a consultant -- consider that if you do the diligence, if you come up some standardization, it’s the methodological approaches that work. SOA will make the company better and will make IT indispensable in the best possible way.

Rod Butters: I want to thank all of you for the discussion. It’s actually ranged all the way from the value of SOA to the business and the kinds of business opportunities that enables how management keys into that and feeds into that from a technology standpoint, and the interesting ideas you shared today about how management ties with the actual SOA architecture to provide agility to the business.

So, with that, thanks very much for this great discussion today and thanks to everyone here who has attended our live debate and discussion on SOA management. At this point, we’re going to open this up for questions.

Question: I'm from Bank of America, and I was really glad the way you guys started to talk about the management top-down approach to addressing the changes that the organization will face. You also talked about agility and business demands and bringing IT management bit closer to the business management side. Business, for all intents and purposes, doesn’t really care what’s underneath the covers, right?

They have a business value proposition, they have business ideas and perhaps a very cool looking interface, if one may just look at the Internet only, a Web 2.0 experience. For them, agility and latency has a lot more meaning than what we are talking about with respect to SOA. With so much organizational change, there’s going to be some time before these things are going to get settled. How do we really achieve agility in terms of business demands and SOA, which is lot more organizationally heavy it seems? That’s one question.

Second, there is one notion of introducing and implementing SOA, which is going to take some time, and we are talking about management needs to get together into a common consensus of implementing SOA in a most efficient fashion. Now again, talking about agility in terms of maintenance and regular changes to the environment, from the business perspective, they want change to be implemented faster.

The owners define pretty much for the entire organization enterprise wide. How does business still achieve agility into their maintenance mode, when they want to make small change to an application which is now going to require a lot more changes on the back-end side?

Bloomberg: You’re quite right; the business doesn’t have visibility into the inner workings of things, but they also don’t want to know. They want SOA stuff to work the way it’s supposed to work. They will have a change. They want it to happen. They don’t want to hear about any problems. They don't want to hear a bunch of tech-speak in response to some request that they have. That’s one of the core challenges of SOA, thinking about services in the context of the business. It’s a role of IT to build and support business services that abstract the underlying technical complexity to provide the business with that flexibility that they need.

If you can achieve this with individual services, then you can achieve this with the compositions of services. You have to build services to be composable. You’re trying to enable the business to build and evolve business processes by composing, recomposing, and reconfiguring services. If you do this right, if you get the architecture and the implementation correct -- it's a big “if” because it’s a challenge -- then you can build big services out of little ones, because you can compose services into composite applications and expose that as a service as well. So, this big exposed composition of services as a service gives you two core things.

One, it’s recursive, if I can build big ones out of little ones. Second, if you were to see some sort of business service, you don’t know if it’s actually abstracting a process or not. So, in telco, you could have a user-provisioning service, where it’s actually in multiple steps. Or, in banking, you could have an open-account service. That might involve six or eight steps, whatever it is. From the business perspective, from the customer prospective, they don’t want to know.

They want to open an account. If they can push a button and the account is open, that’s great. If you can build loose coupling, not only into individual services, but into how you compose services as well, then you’re able to build flexibility into the processes themselves. So, it’s a challenge, but that's the goal -- to build this level of flexibility into the processes, because that gives the business now the ability to have the agility at that business level.

Gardner: If the business side of house wants to have buttons and levers that they can push and pull, then it's up to IT organization to take those inferences and those instructions and then make that into something that changes. It’s a change management function. And, as Jason said, if the processes are composed of individual services, you can rearrange those services, and you could not only rearrange your service within a process, but you can create new services rapidly.

If you have a service that needs to be changed, you don’t necessarily have to rip and replace. You don’t have to shut things down. It’s not a three-year replacement cycle. You can actually recreate that service, make the changes you wanted and then slowly bring that service into production across more processes.

While one of the benefits of SOA/services enablement is reusability, you can also get redundancy. You can create services that are rather similar. You might want to not let anyone be that prolific and write too much, because then you’ve got multiple slices and dices of services. But, it allows for the ability for moving functional sets without having to recreate the entire application.

You’re going to have more iterations, smaller changes within services. You can have services that are similar, and you might well combine them to be a fuller set of requirements within a single service. Then, you can bring that into production across more and more of the organization. It’s just a more flexible approach. You cannot do this when you have brittle applications on a single silo.

Bloomberg: From the business perspective, what you’re saying is exactly right, but, from the business perspective, you can abstract a set of redundant services or revolving services. Think of those that are at a lower level. From the business perspective, they’re all just one service. You have your account-opening service, it’s just the one service, and it works the way it supposed to work.

Behind the scenes, you have redundancy. You have versioning policies. You have management infrastructure. You have all the stuff going on behind the scenes, from the perspective of the business, to get that service to be flexible, so as business requirements change, it does what it’s supposed to do.

From the business perspective, it’s an account-opening service. It opens accounts, and it works for all my lines of business, for all my different kinds of accounts, and it continues to work even if the requirements for that service change. It’s not easy, right? But, you can make it simple for the business by taking these infrastructure and architecture steps within the context of IT.

Question: You talked a lot about the active aspects of SOA management. How is that different from an enterprise service bus (ESB)?

Bloomberg: If you noticed, we didn’t talk about ESBs. Normally, we don't talk about ESBs. We didn’t talk about integration infrastructure at all. We didn’t talk about middleware at all, and that was intentional. When I talk about SOA infrastructure, ESBs are not high in the list. Middleware, in general, is not high in the list. I talk about governance quality and management as the core infrastructure capabilities that SOA requires. From the context of middleware, most organizations already have a lot of middleware. They don’t necessarily need a lot more.

Now, if you’re getting into SOA implementation and you have your architecture, you have your service design, you have your governance, and you’ve thought about governance quality and management, you may find that you need middleware at some point, because whatever middleware you have is not to the task. Then, there may be a requirement for an ESB, or other middleware solution.

But, you don’t want to start there. If you start with the ESB, because some vendor came and said, "Well, to do so you need to buy an ESB," you’re not going to end up with SOA. You’re going to end up with an ESB, actually what the blogosphere is trying to call an ESB-oriented architecture. The point is that if you start with ESB, you end up with something other than SOA. You end up with a traditional middleware-driven architecture with service interfaces.

Now, that being said, there are some perfectly good ESB products on the market, and lot of them offer management capabilities as well. So, when you get to the point of considering what management infrastructure you need, you may turn to an ESB for that capability, but you don’t want to start there. You want to start with the architecture, the business problem, the business processes. Use those two to derive your services. Look at your infrastructure, solve the governance quality management problems, and at that point an ESB might come into the story.

Gardner: ESBs can be very powerful, and, as you are on the journey to SOA activities, integration has been problematic, brittle, expensive, and time consuming. Any productivity benefits that you can bring to integration to me makes sense to me, but an ESB can be more powerful in the management sense, because you can manage the ESB, the way you’ve been further managing your processes or your services or the integration of resources and assets that contribute to the production of your services.

So ESBs do play an important role in management, and we’ll need to see more flexibility and ease of managing ESBs and what they do in applying rules and policies to ESBs. It’s just another important part of the infrastructure.

We’ve had busing and messaging for some time. The idea of trying to make that inclusive of more transport protocols and technologies makes a lot of sense, but how do you manage them? So, again it’s about bringing intelligence from a higher abstraction across more systems. So, I think ESBs will be important, and I think managing ESBs will be important.

Milani: I would add that ESBs are a very powerful piece of this puzzle, and I believe that what they really do today is interconnect multiple types of different systems, and they facilitate and orchestrate the interaction and exchange of data. So, in many ways all they’re doing is exposing existing interfaces, and they facilitate interaction within those interfaces. But, I think what's missing from the ESB picture, from the monitoring and management perspective, is they don’t have deep visibility into the runtime of the interfaces that in fact they are exposing.

So, one piece that you don’t have visibility into is the runtime of those services within ESB, but I do think ESBs will play a major role going forward to marry passive management, which is to manage and monitor what you have, with active management which is constantly "correct and adapt." And I think that’s an area where an ESB could be extremely powerful in the next few years.

Butters: I thank everyone for joining us today. This has been a very enlightening discussion. Again, thank you to our panelists and thank you all for joining us here at the Harvard Club. Have a great day.

Gardner: Thank you.

Bloomberg: Thank you.

Listen to the podcast here. Sponsor: Tidal Software.

Edited transcript of SOA management trends and analysis discussion. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.