Transcript of BriefingsDirect podcast on peak season shipping efficiencies and UPS retail solutions with Alibris and QVC.
Listen to the podcast here. Sponsor: UPS.
Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions and you're listening to BriefingsDirect.
Today, a sponsored podcast discussion about the peak holiday season for retail shopping -- online and via television -- and the impact that this large bump in the road has logistically and technically for some major retailers.
We’re going to discuss how Alibris, an online media and bookseller, as well as QVC, a global multimedia shopping network, handles this peak demand issue. The peak is culminating for such shippers as UPS this week, right around Dec. 19, 2007.
We’re going to talk about how the end-user in this era of higher expectations is now accustomed to making a phone call or going online to tap in a few keystrokes, and then -- like Santa himself -- having a package show up within a day or two. It's instant gratification, if you will, from the logistics point-of-view.
Helping us understand how this modern miracle can be accomplished at such high scale and with such a huge amount of additional capacity required during the November and December shopping period, we’re joined by two guests. We’re going to be talking with Mark Nason, vice president of operations at Alibris, and also Andy Quay, vice president of outbound transportation at QVC. I want to welcome you both to the show.
Mark Nason: Thank you, Dana.
Gardner: Tell us a little bit about what’s different now for Alibris, given the peak season demands, over just a few years ago. Have the expectations of the end-user really evolved, and how do you maintain that sort of instant gratification despite the level of complexity required?
Nason: What we strive for is a consistent customer experience. Through the online order process, shoppers have come to expect a routine that is reliable, accurate, timely, and customer-centric. For us to do that internally it means that we prepare for this season throughout the year. The same challenges that we have are just intensified during this holiday time-period.
Gardner: For those who might not be familiar, tell us a little about Alibris. You sell books, used books, out-of-print books, rare media and other media -- and not just directly, but through an online network of independent booksellers and retailers. Tell us more about how that works.
Nason: Alibris has books you thought you would never find. These are books, music, movies, things in the secondary market with much more variety, and that aren’t necessarily found in your local new bookseller or local media store.
We aggregate -- through the use of technology -- the selection of thousands of sellers worldwide. That allows sellers to list things and standardize what they have in their store through the use of a central catalogue, and allows customers to find what they're looking for when it comes to a book or title on some subject that isn’t readily available through their local new books store or media seller.
Gardner: Now, this is a very substantial undertaking. We're talking about something on the order of 70 million books from a network of some 10,000 booksellers in 65 or more countries. Is that right?
Nason: Roughly, that’s correct. Going in and out of the network at any given time, we've got thousands of sellers with literally millions of book and other media titles. These need to be updated, not only when they are sold or added, but also when they are priced. Prices are constantly changing. It’s a very dynamic market.
Gardner: What is the difference in terms of the volume that you manage from your slowest time of the year compared to this peak holiday period, from mid-November through December?
Nason: It’s roughly 100 percent.
Gardner: Wow!
Nason: In this industry there are actually two peak time periods. We experience this during the back-to-school season that occurs both in January and the latter-half of August and into September.
Gardner: So at the end of the calendar year you deal with the holidays, but also for those college students who are entering into their second semester?
Nason: Exactly. Our peak season associated with the holidays in December extends well into January and even the first week of February.
Gardner: Given this network and the scale and volume and the number of different players, how do you manage a consistent response to your customers, even with a 100 percent increase at the peak season?
Nason: Well, you hit on the term we use a lot -- and that is "managing" the complexity of the arrangement. We have to be sure there is bandwidth available. It’s not just staffing and workstations per se. The technology behind it has to handle the workload on the website, and through to our service partners, which we call our B2B partners. Their volume increases as well.
So all the file sizes, if you will, during the transfer processes are larger, and there is just more for everybody to do. That bandwidth has to be available, and it has to be fully functional at the smaller size, in order for it to function in its larger form.
Gardner: I assume this isn’t something you can do entirely on your own, that you depend on partners, some of those B2B folks you mentioned. Tell us a little bit about some of the major ones, and how they help you ramp up.
Nason: In the area of fulfillment, we rely heavily on our third-party logistics partners, which include carriers. At our distribution centers, typically we lease space, equipment, and the labor required to keep up with the volume.
Then with our B2B partners -- those are the folks that buy from us on a wholesale or distribution basis -- we work out with them ahead of time what their volume estimates might be and what their demands on us would be. Then we work on scheduling when those files might come through, so we can be proactive in fulfilling those orders.
Gardner: When it comes to the actual delivery of the package, tell us how that works and how you manage that complexity and/or scale.
Nason: Well, we have a benefit in that we are in locations that have scalable capacity available from the carriers. That includes lift capacity at the airport, trucking capacity for the highway, and, of course, railheads. These are all issues we are sensitive to, when it comes to informing our carriers and other suppliers that we rely on, by giving them estimates of what we expect our volume to be. It gives them the lead time they need to have capacity there for us.
Gardner: I suppose communication is essential. Is there a higher level of integration handoff between your systems and their systems? Is this entering a more automated level?
Nason: It is, year-round. For peak season it doesn’t necessarily change in that form. The process remains. However, we may have multiple pick-ups scheduled throughout the day from our primary carriers, and/or we arrange special holiday calendar scheduling with those carriers for pick-up, perhaps on a Saturday, or twice on Mondays. If they are sensitive to weather or traffic delays, for example, we know the terminals they need to go through.
Gardner: How about returns? Is that something that you work with these carriers on as well? Or is that something you handle separately?
Nason: Returns are a fundamental part of our business. In fact, we do our best to give the customer the confidence of knowing that by purchasing in the secondary market, the transaction is indemnified, and returns are a definite part of our business on a day-to-day basis.
Gardner: What can we expect in the future? Obviously this volume continues, the expectations rise, and people are doing more types of things online. I suppose college students have been brought up with this, rather than it being something they have learned. It’s something that has always been there.
Do you see any prospects in the future for a higher level of technology need or collaboration need, how can we scale even further?
Nason: Constantly, the improvements in technology challenge the process, and managing the complexity is what you weigh against streamlining even further what we have available -- in particular, optimizing inter-modal transport. For example, with fuel costs skyrocketing, and the cost of everyone's time going up, through the use of technology we look for opportunities on back-haul lanes, or in getting partial loads filled before they move, without sacrificing the service interval.
These are the kinds of things that technology allows when it's managed properly. Of course, another layer of technology has to be considered from the complexity standpoint before you can be successful with it.
Gardner: Is there anything in the future you would like to see from such carriers as UPS, as they try to become your top partners on all of this?
Nason: Integration is the key, and by that I mean the features of service that they provide. It’s not simply transportation, it’s the trackability, it’s scaling; both on the volume side, but also in allowing us to give the customer information about the order, when it will be there, or any exceptions. They're an extension of Alibris in terms of what the customer sees for the end-to-end transaction.
Gardner: Fine, thanks. Now we’re going to talk with Andy Quay, the vice president of outbound transportation at QVC.
QVC has been having a very busy holiday peak season this year. And QVC, of course, has had an illustrious long-term play in pioneering, both retail through television and cable, as well as online.
Welcome Andy, and tell us a little bit about QVC and your story. How long you have been there?
Andy Quay: Well, I am celebrating my 21st anniversary this December. So I can say I have been through every peak season.
Although peak season 20 some years ago was nothing compared to what we are dealing with now. This has been an evolutionary process as our business has grown and become accepted by consumers across the country. More recently we’ve been able to develop with our website as well, which really augments our live television shows.
Gardner: Give us a sense of the numbers here. After 21 years this is quite a different ball game than when you started. What sort of volumes and what sort of records, if any, are we dealing with this year?
Quay: Well, I can tell you that in our first year in business, in December, 1986 -- and I still have the actual report, believe it or not -- we shipped 14,600 some-odd packages. We are currently shipping probably 350,000 to 450,000 packages a day at this point.
We've come a long way. We actually set a record this year by taking more than 870,000 orders in a 24-hour period on Nov. 11. This led to our typical busy season through the Thanksgiving holiday to the December Christmas season. We'll be shipping right up to Friday, Dec. 21 for delivery on Christmas.
Gardner: At QVC you sell a tremendous diversity of goods. Many of them you procure and deal with the supply chain yourselves, therefore cutting costs and offering quicker turnaround processing.
Tell us a little about the technology that goes into that, and perhaps also a little bit about what the expectations are now. Since people are used to clicking a button on their keyboard or making a quick phone call and then ... wow, a day or two later, the package arrives. Their expectations are pretty high.
Quay: That’s an excellent point. We’ve been seeing customer expectations get higher every year. More people are becoming familiar with this form of ordering, whether through the web or over the telephone.
I’ll also touch on the technology very briefly. We use an automated ordering system with voice response units that enable my wife, for example, to place an order in about 35 seconds. So that enables us to handle high volumes of orders. Using that technology has allowed us to take some 870,000 orders in a day.
The planning for this allows the supply chain to be very quick. We are like television broadcasts. We literally are scripting the show 24-hours in advance. So we can be very opportunistic. If we have a hot product, we can get it on the air very quickly and not have to worry about necessarily supplying 300 brick-and-mortar stores. Our turnaround time can be blindingly quick, depending upon how fast we can get the inventory into one of our distribution centers.
We currently have five distribution centers, and they are all along the East Coast of the U.S., and they are predominantly commodity driven. For example, we have specific commodities such as jewelry in one facility, and we have apparel and accessories as categories of goods in another facility. That lends itself to a challenge when people are ordering multiple items across commodities. We end up having to ship them separately. That’s a dilemma we have been struggling with as customers do more multi-category orders.
As I mentioned, the scripting of the SKUs for the broadcast is typically 24 hours prior, with the exception of Today's Special Value (TSV) show and other specific shows. We spend a great deal of time forecasting for the phone centers and the distribution carriers to ensure that we can take the orders in volume and ship them within 48 hours.
We are constantly focused on our cycle-time and in trying to turn those orders around and get them out the door as quickly as possible. To support this effort we probably have one of the largest "zone-jumping" operations in the country.
Gardner: And what does "zone-jumping" mean?
Quay: Zone jumping allows me to contract with truckload carriers to deliver our packages into the UPS network. We go to 14 different hubs across the country, in many cases using team drivers. This enables us to speed the delivery to the customer, and we’re constantly focused on the customer.
Gardner: And this must require quite a bit of integration, or at least interoperability in communications between your systems and UPS’s systems?
Quay: Absolutely, and we carefully plan leading up to the peak season we're in now. We literally begin planning this in June for what takes place during the holidays -- right up to Christmas Day.
We work very closely with UPS and their network planners, both ground and air, to ensure cost-efficient delivery to the customer. We actually sort packages for air shipments, during critical business periods, to optimize the UPS network.
Gardner: It really sounds like a just-in-time supply chain for retail.
Quay: It's as close as you can get it. As I sometimes say, it's "just-out-of-time"! We do certainly try for a quick turnaround.
Coming back to what you said earlier, as far as the competition goes it is getting more intense. The customer expectations are getting higher and higher. And, of course, we are trying to stay ahead of the curve.
Gardner: What's the difference between your peak season now and the more regular baseline of volume of business? How much increase do you have to deal with during this period, between late-November and mid- to late-December?
Quay: Well, it ramps up considerably. We can go from a 150,000 to 200,000 orders a day, to literally over 400,000 to 500,000 orders a day.
Gardner: So double, maybe triple, the volume?
Quay: Right. The other challenge I mentioned, the commodity-basis distribution that we operate on -- along with the volatility of our orders -- this all tends to focus on a single distribution center. We spend an inordinate amount of time trying to forecast volume, both for staffing and also planning with our carriers like UPS.
We want to know what buying is going to be shipping, at what distribution center, on what day. And that only compresses even more around the holiday period. We have specific cutoff times that the distribution center operations must hit in order to meet the customers' delivery date. We work very closely on when we dispatch trucks ... all of this leading up to our holiday cutoff sequence this week.
We try to maximize ground service versus the more expensive airfreight. I think we have done a very good job at penetrating UPS’s network to maximize ground delivery, all in an effort to keep the shipping and handling cost to the customers as low as possible.
Gardner: How about the future? Is this trend of that past 21 years sustainable? How far can we go?
Quay: I believe it is sustainable. Our web business is booming, with very high growth every year. And that really augments the television broadcast. We have, honestly, a fair amount of penetration, and we can still obtain more with our audiences.
Our cable broadcast is in 90 million-plus homes that actually receive our signal, but a relatively small portion actually purchase. So that’s my point. We have a long way to go to further penetrate and earn more customers. We have to get people to try us.
Gardner: And, of course, people are now also finding goods via Web search. For example, when they go to search for a piece of apparel, or a retail item, or some kind or a gift -- they might just go to, say, Google or Yahoo! or MSN, and type something in and end up on your web site. That gives you a whole new level of potential volume.
Quay: Well, it does, and we also make the website very well known. I am looking at our television show right now and we’ve have our www.qvc.com site advertised right on it. That provides an extended search capability. People are trying to do more shopping on the web, in addition to watching the television.
Gardner: We have synergies on the distribution side; we have synergies on the acquisition, and of using information and how to engage with partners. And so the technology is really in the middle of it all. And you also expect a tremendous amount of growth still to come.
Quay: Yes, absolutely. And it’s amazing, the different functions within QVC, the synergies that we work together internally. That goes from our merchandising to where we are sourcing product.
You mentioned supply chains, and the visibility of getting into the distribution center. Our merchants and programmers watch that like a hawk so they can script new items on the air. We have pre-scripted hours that we’re definitely looking to get certain products on.
The planning for the television broadcast is something that drives the back end of the supply chain. The coordination with our distribution centers -- as far as getting the operation forecast, staffed and fulfilled through shipping to our customers -- is outstanding.
Gardner: Well, it’s very impressive, given what you’ve done and all of these different plates that you need to keep spinning in the air -- while also keeping them coordinated. I really appreciate the daunting task, and that you have been able to reach this high level of efficiency.
Quay: Oh, we are not perfect yet. We are still working very hard to improve our service. It never slows down.
Gardner: Great. Thanks very much for your input. I have learned a bit more about this whole peak season, what really goes on behind the scenes at both QVC and Alibris. It seems like quite an accomplishment what you all are able to do at both organizations.
Nason: Well, thank you, Dana. Thanks for taking the time to hear about the Alibris story.
Gardner: Sure. This is Dana Gardner, principal analyst at Interarbor Solutions. We have been talking with Mark Nason, the vice president of operations at Alibris, about managing the peak season demand, and the logistics and technology required for a seamless customer experience.
We’ve also been joined by Andy Quay, vice president of outbound transportation, at the QVC shopping network.
Thanks to our listeners for joining on this BriefingsDirect sponsored podcast. Come back and listen again next time.
Listen to the podcast here. Sponsor: UPS.
Transcript of BriefingsDirect podcast on peak season shipping efficiencies and UPS retail solutions. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.
Showing posts with label integration. Show all posts
Showing posts with label integration. Show all posts
Wednesday, December 19, 2007
Thursday, August 16, 2007
Apache Camel Addresses Need for Discrete Infrastructure for Services Mediation and Routing
Edited transcript of BriefingsDirect[tm/sm] podcast with Dana Gardner, recorded July 27, 2007.
Listen to the podcast here. Podcast sponsor: IONA Technologies.
Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today, a sponsored podcast discussion on a top level Apache Software Foundation project, known as Apache Camel.
It’s a way to help developers do better integration and mediation for routing in software-intense environments, and moving lots of objects and component services around, but also relying on rules and patterns to do so.
To help us understand more about Apache Camel, we’re joined today by James Strachan, technical director of engineering at IONA Technologies. Welcome, James.
James Strachan: Hi.
Gardner: Tell us a little bit about Apache Camel. It seems to be a subset of many other integration and infrastructure projects in the open-source community. Tell us a little bit about the history of how Apache Camel came to be.
Strachan: Earlier this year, Apache Camel grew organically from code and ideas from a bunch of other Apache projects, particularly Apache ActiveMQ and Apache ServiceMix. We found that people wanted to create and use patterns from the "Enterprise Integration Patterns" book in many different scenarios.
Some people wanted to use these patterns inside an Enterprise Service Bus (ESB), some people wanted to use these patterns inside a message broker, and other people wanted to use these patterns inside an application itself or to talk between messaging providers. Still other people wanted to use them inside a Web services framework or some other communication platform. So, rather than tie this routing code to a particular message broker or ESB, we tried to extract this code to be a standalone framework that can be used in any project.
Gardner: I assume that this is to simplify it and make it easier for developers? Is that right?
Strachan: Definitely. What we tried to do with Camel was give it the smallest footprint possible, so that it can be reused anywhere, whether in a servlet, in the Web services stack, inside a full ESB, or a messaging application.
Gardner: And, this is a plain old Java object (POJO), I believe, also built in Java. Is that correct?
Strachan: Absolutely.
Gardner: Tell us a bit about the team behind this and perhaps a little background for yourself?
Strachan: I'm an Apache committer. I’ve been in Apache for over five years now, and I am an Apache member for a few years too. I can’t remember how many. The Camel team is comprised mostly of people from the ActiveMQ team and also from ServiceMix and Apache CXF communities. Plus, a bunch of other people have joined since then.
Gardner: Tell us a little bit about your relationship with Apache. You’re involved with a number of activities there. Is that right?
Strachan: I've been around Apache for many years now, and I am involved in all kinds of different projects -- ActiveMQ, ServiceMix are the big ones -- but also on a bunch of others, like Jakarta Commons, Maven, and many others.
Gardner: There are a number of terms we’ve been bandying around. Let’s assume that not everyone listening is that deeply into technology. When we talk about mediation and patterns, let’s flesh some of these out. Mediation and routing, how is that different from what people might understand generally under enterprise integration?
Strachan: I definitely recommend people read Gregor Hohpe's book "Enterprise Integration Patterns." He offers a really good patterns catalog of how people should do mediation and integration. Rather like the original Gang of Four "Design Patterns" book, which describes low-level programming things, Gregor’s book describes very well how enterprise integration patterns (EIPs) can work and gives us a language for describing them.
Aggregate, re-sequencer, message filter, and message translator generally describe patterns people commonly want to do within an integration solution. One of the simplest patterns is "content-based router," where you want to route messages around your network, based on the content. It might be, for example, that for gold-level customers you want to use the big, shiny, fast machines in your data center, but for bronze-level customers you use the crappy, old, worn Linux box in the corner of the basement.
Gardner: You mentioned earlier ESB, enterprise service bus. Explain to me how this works in terms of other ESBs. This is ESB-agnostic. Is it something that people will use to complement ESBs or perhaps replace them in some instances? What’s the relationship between Camel and an ESB?
Strachan: That’s a good question. In some ways the term "ESB" is a little bit like the terms "object" and "component," in that ESB is used for many different things. It has kind of lost a lot of its meaning. We typically don’t tend to think of Camel as being an ESB. We think of it as a rules-based routing and mediation framework that you may deploy inside an ESB.
We work very closely with the ServiceMix community at Apache which has created a complete Java Business Integration (JBI)-compliant container of JBI components. Camel can be deployed within the ServiceMix ESB among JBI components, but some people don’t use JBI and they may just use Java Message Service (JMS) or they may just use Web services or they may just be JAX-WS clients or whatever. So, we try to make Camel agnostic to technologies. You can use it within patterns like Spring, or JBI or OSGi, or you can use it within any application.
Gardner: I’ve also heard you refer to this as a domain-specific language project. What do you mean by that?
Strachan: Programming languages are often very general purpose. They can be used to solve any kind of problem. Domain-specific languages have become popular lately, and they’re basically tied to typically a business domain, such as investment banking, telco, or whatever. In Camel’s case, we’ve made a domain-specific language that focuses purely on the EIPs.
The language itself describes how to do the various things you typically do in the patterns, such as wire tap, aggregator, content-based router. These are first-class concepts within the language itself. We’ve implemented this domain-specific language in both Java and XML, so you can stick to a Java IDE if you prefer, or, if you’d rather, you can use XML to describe this language and deploy it inside your Spring XML.
Gardner: So, we’ve had vertical industry domain-specific languages. This seems to be more of a functionally domain-specific language. Is that a correct statement?
Strachan: Yes. The EIP is the functional domain, or rather the business domain.
Gardner: And, this is all about getting more granularity, more specialization for integration that nonetheless can play across a more general or horizontal capability. Is that fair?
Strachan: Definitely. Across all of IT we’re seeing increased specialization in many different areas, where the specialization helps us solve a problem at a higher level. If you spend your life solving problems, with, say, JavaBeans everywhere, you have a lot of code to comprehend and understand, and things can get quite complicated. The higher level of abstraction helps us solve problems easily.
Gardner: And, because we’re doing it an open-source environment where there’s a community involved, the more likelihood that this will be applied across many other different types of platforms and technology?
Strachan: Exactly. The other benefit of being with Apache is that we have a very liberal license, so this can be embedded in any commercial product very easily. We've seen lots of partners using this technology.
Gardner: Right. A few moments ago, we talked about ESBs. There has been, as you say, some divergence in the term ESB. There are also a number of products and approaches in the field. How about for Camel as a rules-based routing and mediation engine? Are there many others of those? Who else is trying to solve the same problem, or can we safely say that the Camel is unique?
Strachan: It depends on how wide you cast that net. Camel is unique in a number of ways. What we’re doing with Camel is defining a high-level language to describe EIPs, which I don’t think anybody else has done before. The other thing that’s unique is that this language very closely maps to components that work inside Spring 2.
We use a lot Spring 2 features, such as declarative transactions, inversion of control configuration, and various utility classes for working, with such things as JMS and JDBC and Java Persistence API (JPA). What we’re doing is raising the abstraction level to make things very simple, reducing the amount of XML we have to write, but still exposing the wire-level access if you need to do the really hard stuff and roll your sleeves up and get down and dirty.
Gardner: Let’s move this a bit closer to the business side. I wonder what sort of a problem set it is that we’re facing here. Is this a new approach to older problems or a new approach to newer problems? What's the problem solution fit here in the market?
Strachan: The problem we’re trying to address is the routing and mediation problem, which lots of people have. They're taking data from various components and sources -- whether it’s files, databases, message queues, Web services, instant messaging, or other data systems, integrating them together, formatting them, and connecting them to the systems. From a higher level perspective, this could be for legacy integration of systems, for smart routing, performance, monitoring, or testing or monitoring transaction flows.
In general, integration is a very old problem. As soon as we relate to couple of different computer programs, we already have an integration problem and that just grows with time. The thing that’s new is the approach that’s used. We're increasing the abstraction level by making it very easy to work at the EIP layer.
People don’t have to worry about the low-level details of how to use JMS, how to use JBI, or how to wire together the Spring components correctly, and so forth. We're giving people a nice, simple, high-level abstraction, but yet we are exposing all the power of frameworks like Spring and still exposing the low-level details if you need them.
Gardner: Over the last couple of years, because of the popularity and use of Web services, the business side wants to expand the interactions across not only applications, but organizations, and perhaps in a supply chain setting. It sounds like this helps solve the issue of inclusiveness across domains and businesses, and then adds the opportunity to start moving toward event-driven computing. Does that sound fair?
Strachan: Definitely. And, as soon as people start moving toward distributed systems, service oriented architecture (SOA), event-driven architectures, and all these other terms, the one thing that keeps happening is that people need a pervasive technology they can apply anywhere. Camel is lightweight, and can be used anywhere, in a smart end point, in an ESB, or a message broker.
ActiveMQ5 is coming out in [August, 2007]. That's going to have EIPs integrated in the client and the broker by Camel. ServiceMix has integration with Camel and CXF, as a project at Apache, which implements the JAX-WS standard that’s got EIP integration via Camel. We’re seeing a large number of projects that have EIP integration via Camel. It’s already proven in the open-source world that it’s very easy to reuse this routing and mediation technology anywhere. Hopefully, we'll see this just grow and grow.
Gardner: It’s curious to me. It seems like yesterday, but I suppose we're going on 10 years now, that a lot of these functions -- singular, distributed, perhaps even proprietary -- were combined into what became conceptually the application server. Now, it seems like we’re decoupling things. That one large application server approach might not be the best fit, as we get more types of routing and messaging in these integration patterns.
Can you expand on that? What's been the larger trend, in terms of architecture?
Strachan: There’s been a definite move away from J2EE apps, the big application servers, and people are looking for small, lightweight solutions that solve only the problem they have and not problems they don’t have. A lot of software vendors suffered from the kitchen-sink syndrome, where they tried to have one of everything in a big box. That just led to lots of complexity and redundancy.
People really want small and simple-to-use components that solve the problems they have. I've seen that throughout all of our customers. People ask for very specific solutions. They don’t say, "Give me a SOA." They say, "I need a message router," "I need a message bus," "I need an ESB," or "I need a services framework." Often, people have very specific requirements and are very much cherry-picking the best-of-breed components from the open-source tool set.
Gardner: So, it’s the Swiss Army Knife approach.
Strachan: Exactly.
Gardner: That works well for developers. I wonder over time how Camel might find itself being used in terms of operators, specifiers, and architects on the operational side of things. Do you see this moving in that direction? What’s the short-, medium-, and long-term implementation roadmap that you foresee for Camel?
Strachan: Today it's very much developer-focused. The current Camel is aimed at developers who are savvy with Java, XML, or Spring. What we want then is to gradually raise the abstraction level, so the architects and operational people can monitor services, monitor business processes, perform operational behavior, plus be able to design and deploy EIPs in a simple way.
For example, building tools and making it easier to design EIPs, as well as to choose a monitoring tool, are important.
Gardner: How would the specifying roadmap shake out? Do you think this is something that would get bundled in or integrated? How would this be something easily rationalized in production environments for those people that are going to be looking at large data center implementations?
Is this something that would be part of another grouping of products, other Apache kind of projects? How would this combine to make a whole larger than the sum of the parts?
Strachan: What we’ve seen over the last few years is a huge diversity in how people actually deploy products. Maybe five years ago, people assumed the world was J2EE app servers and everything would deploy in a J2EE deployment unit. These days, we see people make their own deployment units. We're seeing a lot of people move towards OSGi bundles. So we see a huge difference in the ways in which people deploy software now.
One of the reasons we’ve taken extreme steps to split Camel up and make it a very simple and easy to use bundle is that we know that there is a huge range of deployment platforms. Some people want to use Camel within their own smart end points. Some people want to host it inside the message broker, in the ESB and so forth. So, we see it going into production in many guises and many different forms.
Gardner: In IONA’s case, how would this be used in the context of its offering? I'm assuming that, as with other IONA offerings, it will be the commercial open-source support approach.
Strachan: Yes. Camel’s pretty much the core foundation for smart routing and mediation within an entire product suite. We have a product called FUSE, an open-source SOA backplane that provides a services framework, a message broker, an ESB, and a mediation and routing engine. The mediation and routing engine is based very heavily on Apache Camel, and we're using Apache Camel throughout other projects as well.
Gardner: Are there forums or community sites? How are some of the dialog and collaboration around this taking place?
Strachan: If you go to http://open.iona.com, there is a whole raft of documentation online forums, a wiki, and so forth.
Gardner: I know this is fairly early, but maybe this will be a good time to lay out some of the milestones. Are we expecting some significant developments and/or visions within the overall Camel project? Can you give us a bit of a timeline?
Strachan: It’s still fairly early for the timeline, but the next version of Camel 1.2 should be out fairly soon, and it's going to offer improved monitoring and management tools and particularly improved visualization and tooling.
Toward the end of the year, we’re hoping to improve various other capabilities we’ve added recently for extract, transform and load (ETL). It’s a way of loading data into systems and load testing systems with data. Plus, another area we’ve been developing what we call business activity monitoring (BAM), which helps monitor transaction flows across different systems.
For example, you might want to track that a purchase order in a system maps through to an invoice that comes out of another system. You might be processing a 100,000 of those a day, but there is one that goes along every few hours, and you want to find out which one that is within a small time window. That’s another area where we are extending the reach of Camel to do very complex and powerful monitoring.
Gardner: That's interesting, because this provides the actual routing mechanisms that provide a window into what's good, and also what's not good about how those processes are being built and used.
Strachan: Definitely. We’ve found that as soon as you have this very flexible and reusable integration patterns, it’s very easy to build on that, to build higher level abstractions, whether this is powerful testing frameworks, stimulation frameworks, ETL or BAM. We’re seeing the levels of abstraction increase to remove layers and layers of complexity.
Gardner: Is this something that could be applied to the ongoing -- and I imagine increasingly arduous -- task of dealing with semantic issues in terms of data across applications? That is to say, helping to define a mediation and a rules-based approach, such as this help in how companies can deal with that diversity and then come up with more common approaches to data and content that can then be injected into a business process.
Strachan: Definitely. This is a very large and complex topic. There are just a few bullet points on the top, and we’ve already got a range of transformation and validation capabilities in Camel, but we’re looking to improve this in various ways.
IONA recently acquired a company, called C24, which has a whole range of technologies for doing semantic mapping and transformation of different data formats. For example, they've got models for pretty much all the standard financial formats, like SWIFT, FpML, FIX, and all these other things. And, they've got other models for telcos. We’re looking to open source some of that and integrate that into Camel for doing lots of data service type integration.
Another area that’s kind of similar, but slightly different, is in the business activity monitoring ground, where you sometimes want to just check the effect of systems. So, it’s a little bit like I am doing real time reconciliation systems, and you’re doing all this transformation between systems, but you want to make sure of the end results, like that final purchase order or that final settlement instruction does contain the values that you expect. For example, we might want to check that the purchase order amount matches the invoice amount, or whatever.
Gardner: As you say, it’s early in the evaluation of Camel, but are there instances where we can see how it's being used or how developers have identified this as a productivity benefit, or are there metrics of productivity? How is it helping in terms of getting a better job done?
Strachan: We’ve got a bunch of different customers using Camel in different ways. We seem to have a few different kinds of customers finding Camel for the first time. Some come from an ActiveMQ background. They're doing lots of messaging, and they want something a bit more powerful to do more complex routing within the message broker itself.
Some come from the ServiceMix background. They are very ESB focused. They love JBI. They like the idea of standards-based ESB. They’ve got a bunch of standards-based integration components, but they’re working together, and they just want a slightly more powerful routing engine to work with.
Some of our customers have been using Mule. They found Camel a little bit easier, and they’ve migrated across. Finally, there are people who’ve just found Camel, that's all they know, and they just dive straight into it.
One of the things we’re seeing from customers is that using APIs, like JMS and JBI in particular, and to a lesser extent JAX-WS, could often become complicated, but people find that wiring things together with Camel is almost trivial. People get things going really quickly without having to read pages and pages of specifications for things like JMS or JBI, or dealing with the lower-level details of Spring.
We see quite a rapid productivity gain, and it means that people can then use lots of different technologies, whether it’s JMS or JBI without having to spend a long time figuring out things in the Spring manuals.
Gardner: Among these early users are there any vertical industries popping out, where it seems to be making sense first, perhaps financial services?
Strachan: Yes, finance and telcos definitely are big markets, but we’re also fairly strong in the e-tailers. Those areas will be our big three markets right now.
Gardner: Can you explain the business problems that they're using it for?
Strachan: Pretty much all of our customers have similar kinds of problems. They have lots of silos in applications. There are lots of messages being exchanged between systems, whether those are messages via flat files, email, JMS, legacy message brokers, or web services. They generally want to do one or more of the patterns, whether counter-based routing, or message transformations. We see more people starting to investigate a business activity monitoring feature of Camel, and that’s something I'm quite excited about.
Lots of companies, once they realize they can tie together messages to business processes, start running rules off the back of that. That has been extremely useful for people. One thing about SOA that’s quite difficult is knowing what’s really happening in your system.
Today, people have often done such things as monitor message queues or use a JMX-based management console to look at numbers and statistics. Being able to tie together all the different things flowing through your SOA to actual business processes is extremely useful and valuable for customers.
Gardner: There seems to be some murkiness in how feedback will occur, when you have events-driven architectures, and how to create common approaches to runtime and design time with this BAM emphasis that we are seeing associated with Camel. Do you expect that Camel could fill some of that role in terms of a balancing act between what takes place in operations and what needs to be brought into a service level agreement of some kind?
Strachan: Definitely. I have worked with quite a few customers on various BAM-related problems, and it does seem that BAM solves those problems. It’s quite hard to have an off-the-shelf tool to do everything people need. Almost as soon as people start doing any kind of BAM, they find that they want to put their own code in there. You might be in financing, and maybe there is a trade in U.S. dollars. You want to take that trade to the settlement instruction, but you want to do a foreign exchange conversion in there.
Almost as soon as you start doing the “hello world,” BAM-type scenarios, you tend to want to roll your sleeves up and actually do some coding of some kind, to do custom reconciliation, custom calculations, custom visualizations. BAM is very much a framework-driven thing, rather then a black-box tool. Certainly, we see people take on the Camel framework itself and then expand and extend it to do more powerful things.
Gardner: I suppose that’s part and parcel of open source, community-driven projects. They tend to be pushed out in a variety of directions. Do you have any sense of what other functional parts or functional capabilities we might see brought into Camel?
Strachan: A big area is visualization. It’s incredibly hard for people to follow what’s happening in general. Once you can create a wire tap into any system, it’s very easy to glean information from it and find out what’s happening, what its throughput is, what it is actually doing. You can then do things like trace messages around the system, correlate messages to business processes, reverse engineer whole business flows from legacy systems.
That whole area is a visualization and tooling problem. I see that as a very large area of research with the Camel project. We hope to be to be rolling out in Q4 the first batch of visualization tools. It's very much a work in progress in the next few years to get even better, more powerful operational controls and visualization engines.
Gardner: By getting more visual, it seems like we can bring more people into the process, and that maybe elevates us beyond BAM into business intelligence. Is that fair?
Strachan: Definitely. Absolutely.
Gardner: I think a lot of business people would like to get more of a real-time sense of what’s going on among these processes to provide more confidence, as they move processes from manual and older message into this new SOA approach.
Strachan: Definitely. Plus people want early warning notice of any kind of failure. One of the nice things is that you can be alerted that maybe something's going on that hasn't been found yet. Maybe it's running a bit slow today, or maybe it’s just a heavy trading day. But, it might not be, and maybe something is really wrong. So, rather than waiting for some extreme failure that results in a fine or extra interest being paid, you can respond more rapidly than that.
Gardner: Let’s just circle back quickly. Developers seem to be the core short- to medium-term audience for this. What sort of developers specifically should be more aware of what Camel’s capable of?
Strachan: I’d say anyone who’s doing any kind of SOA or distributed programming. They should at least take a good look. Definitely, anybody who ever read the EIP book. If you read that book, you’d probably just see Camel and you’ll immediately just figure out exactly what it’s doing. It’s very simple.
If you're doing Web services, you might need mediation. If you're doing any kind of messaging using multiple databases, if you have any kind of integration problems, and certainly if you’re using any kind of ESB, then definitely take Camel for a ride.
Gardner: Now, when we look toward getting more information on this, we have the resources at Apache Foundation, and also, as you mentioned, http://open.iona.com?
Strachan: Yes.
Gardner: Any other resources? You mentioned the book. Are there any other resources on this that people should be aware of?
Strachan: Those are the big three. There’s also my blog: http://macstrac.blogspot.com/.
Gardner: Very good. Well, we’ve been discussing a budding Apache Foundation project known as Apache Camel. It’s a rules-based routing and mediation engine for POJO-based implementation of EIPs.
Discussing this with us today, we had James Strachan, technical director of engineering at IONA Technologies.
I'm your host and moderator Dana Gardner, principal analyst at Interarbor Solutions. Thanks for listening and learning more about Apache Camel.
Listen to the podcast here. Podcast sponsor: IONA Technologies.
Transcript of Dana Gardner’s BriefingsDirect podcast on Apache Camel. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.
Listen to the podcast here. Podcast sponsor: IONA Technologies.
Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect. Today, a sponsored podcast discussion on a top level Apache Software Foundation project, known as Apache Camel.
It’s a way to help developers do better integration and mediation for routing in software-intense environments, and moving lots of objects and component services around, but also relying on rules and patterns to do so.
To help us understand more about Apache Camel, we’re joined today by James Strachan, technical director of engineering at IONA Technologies. Welcome, James.
James Strachan: Hi.
Gardner: Tell us a little bit about Apache Camel. It seems to be a subset of many other integration and infrastructure projects in the open-source community. Tell us a little bit about the history of how Apache Camel came to be.
Strachan: Earlier this year, Apache Camel grew organically from code and ideas from a bunch of other Apache projects, particularly Apache ActiveMQ and Apache ServiceMix. We found that people wanted to create and use patterns from the "Enterprise Integration Patterns" book in many different scenarios.
Some people wanted to use these patterns inside an Enterprise Service Bus (ESB), some people wanted to use these patterns inside a message broker, and other people wanted to use these patterns inside an application itself or to talk between messaging providers. Still other people wanted to use them inside a Web services framework or some other communication platform. So, rather than tie this routing code to a particular message broker or ESB, we tried to extract this code to be a standalone framework that can be used in any project.
Gardner: I assume that this is to simplify it and make it easier for developers? Is that right?
Strachan: Definitely. What we tried to do with Camel was give it the smallest footprint possible, so that it can be reused anywhere, whether in a servlet, in the Web services stack, inside a full ESB, or a messaging application.
Gardner: And, this is a plain old Java object (POJO), I believe, also built in Java. Is that correct?
Strachan: Absolutely.
Gardner: Tell us a bit about the team behind this and perhaps a little background for yourself?
Strachan: I'm an Apache committer. I’ve been in Apache for over five years now, and I am an Apache member for a few years too. I can’t remember how many. The Camel team is comprised mostly of people from the ActiveMQ team and also from ServiceMix and Apache CXF communities. Plus, a bunch of other people have joined since then.
Gardner: Tell us a little bit about your relationship with Apache. You’re involved with a number of activities there. Is that right?
Strachan: I've been around Apache for many years now, and I am involved in all kinds of different projects -- ActiveMQ, ServiceMix are the big ones -- but also on a bunch of others, like Jakarta Commons, Maven, and many others.
Gardner: There are a number of terms we’ve been bandying around. Let’s assume that not everyone listening is that deeply into technology. When we talk about mediation and patterns, let’s flesh some of these out. Mediation and routing, how is that different from what people might understand generally under enterprise integration?
Strachan: I definitely recommend people read Gregor Hohpe's book "Enterprise Integration Patterns." He offers a really good patterns catalog of how people should do mediation and integration. Rather like the original Gang of Four "Design Patterns" book, which describes low-level programming things, Gregor’s book describes very well how enterprise integration patterns (EIPs) can work and gives us a language for describing them.
Aggregate, re-sequencer, message filter, and message translator generally describe patterns people commonly want to do within an integration solution. One of the simplest patterns is "content-based router," where you want to route messages around your network, based on the content. It might be, for example, that for gold-level customers you want to use the big, shiny, fast machines in your data center, but for bronze-level customers you use the crappy, old, worn Linux box in the corner of the basement.
Gardner: You mentioned earlier ESB, enterprise service bus. Explain to me how this works in terms of other ESBs. This is ESB-agnostic. Is it something that people will use to complement ESBs or perhaps replace them in some instances? What’s the relationship between Camel and an ESB?
Strachan: That’s a good question. In some ways the term "ESB" is a little bit like the terms "object" and "component," in that ESB is used for many different things. It has kind of lost a lot of its meaning. We typically don’t tend to think of Camel as being an ESB. We think of it as a rules-based routing and mediation framework that you may deploy inside an ESB.
We work very closely with the ServiceMix community at Apache which has created a complete Java Business Integration (JBI)-compliant container of JBI components. Camel can be deployed within the ServiceMix ESB among JBI components, but some people don’t use JBI and they may just use Java Message Service (JMS) or they may just use Web services or they may just be JAX-WS clients or whatever. So, we try to make Camel agnostic to technologies. You can use it within patterns like Spring, or JBI or OSGi, or you can use it within any application.
Gardner: I’ve also heard you refer to this as a domain-specific language project. What do you mean by that?
Strachan: Programming languages are often very general purpose. They can be used to solve any kind of problem. Domain-specific languages have become popular lately, and they’re basically tied to typically a business domain, such as investment banking, telco, or whatever. In Camel’s case, we’ve made a domain-specific language that focuses purely on the EIPs.
The language itself describes how to do the various things you typically do in the patterns, such as wire tap, aggregator, content-based router. These are first-class concepts within the language itself. We’ve implemented this domain-specific language in both Java and XML, so you can stick to a Java IDE if you prefer, or, if you’d rather, you can use XML to describe this language and deploy it inside your Spring XML.
Gardner: So, we’ve had vertical industry domain-specific languages. This seems to be more of a functionally domain-specific language. Is that a correct statement?
Strachan: Yes. The EIP is the functional domain, or rather the business domain.
Gardner: And, this is all about getting more granularity, more specialization for integration that nonetheless can play across a more general or horizontal capability. Is that fair?
Strachan: Definitely. Across all of IT we’re seeing increased specialization in many different areas, where the specialization helps us solve a problem at a higher level. If you spend your life solving problems, with, say, JavaBeans everywhere, you have a lot of code to comprehend and understand, and things can get quite complicated. The higher level of abstraction helps us solve problems easily.
Gardner: And, because we’re doing it an open-source environment where there’s a community involved, the more likelihood that this will be applied across many other different types of platforms and technology?
Strachan: Exactly. The other benefit of being with Apache is that we have a very liberal license, so this can be embedded in any commercial product very easily. We've seen lots of partners using this technology.
Gardner: Right. A few moments ago, we talked about ESBs. There has been, as you say, some divergence in the term ESB. There are also a number of products and approaches in the field. How about for Camel as a rules-based routing and mediation engine? Are there many others of those? Who else is trying to solve the same problem, or can we safely say that the Camel is unique?
Strachan: It depends on how wide you cast that net. Camel is unique in a number of ways. What we’re doing with Camel is defining a high-level language to describe EIPs, which I don’t think anybody else has done before. The other thing that’s unique is that this language very closely maps to components that work inside Spring 2.
We use a lot Spring 2 features, such as declarative transactions, inversion of control configuration, and various utility classes for working, with such things as JMS and JDBC and Java Persistence API (JPA). What we’re doing is raising the abstraction level to make things very simple, reducing the amount of XML we have to write, but still exposing the wire-level access if you need to do the really hard stuff and roll your sleeves up and get down and dirty.
Gardner: Let’s move this a bit closer to the business side. I wonder what sort of a problem set it is that we’re facing here. Is this a new approach to older problems or a new approach to newer problems? What's the problem solution fit here in the market?
Strachan: The problem we’re trying to address is the routing and mediation problem, which lots of people have. They're taking data from various components and sources -- whether it’s files, databases, message queues, Web services, instant messaging, or other data systems, integrating them together, formatting them, and connecting them to the systems. From a higher level perspective, this could be for legacy integration of systems, for smart routing, performance, monitoring, or testing or monitoring transaction flows.
In general, integration is a very old problem. As soon as we relate to couple of different computer programs, we already have an integration problem and that just grows with time. The thing that’s new is the approach that’s used. We're increasing the abstraction level by making it very easy to work at the EIP layer.
People don’t have to worry about the low-level details of how to use JMS, how to use JBI, or how to wire together the Spring components correctly, and so forth. We're giving people a nice, simple, high-level abstraction, but yet we are exposing all the power of frameworks like Spring and still exposing the low-level details if you need them.
Gardner: Over the last couple of years, because of the popularity and use of Web services, the business side wants to expand the interactions across not only applications, but organizations, and perhaps in a supply chain setting. It sounds like this helps solve the issue of inclusiveness across domains and businesses, and then adds the opportunity to start moving toward event-driven computing. Does that sound fair?
Strachan: Definitely. And, as soon as people start moving toward distributed systems, service oriented architecture (SOA), event-driven architectures, and all these other terms, the one thing that keeps happening is that people need a pervasive technology they can apply anywhere. Camel is lightweight, and can be used anywhere, in a smart end point, in an ESB, or a message broker.
ActiveMQ5 is coming out in [August, 2007]. That's going to have EIPs integrated in the client and the broker by Camel. ServiceMix has integration with Camel and CXF, as a project at Apache, which implements the JAX-WS standard that’s got EIP integration via Camel. We’re seeing a large number of projects that have EIP integration via Camel. It’s already proven in the open-source world that it’s very easy to reuse this routing and mediation technology anywhere. Hopefully, we'll see this just grow and grow.
Gardner: It’s curious to me. It seems like yesterday, but I suppose we're going on 10 years now, that a lot of these functions -- singular, distributed, perhaps even proprietary -- were combined into what became conceptually the application server. Now, it seems like we’re decoupling things. That one large application server approach might not be the best fit, as we get more types of routing and messaging in these integration patterns.
Can you expand on that? What's been the larger trend, in terms of architecture?
Strachan: There’s been a definite move away from J2EE apps, the big application servers, and people are looking for small, lightweight solutions that solve only the problem they have and not problems they don’t have. A lot of software vendors suffered from the kitchen-sink syndrome, where they tried to have one of everything in a big box. That just led to lots of complexity and redundancy.
People really want small and simple-to-use components that solve the problems they have. I've seen that throughout all of our customers. People ask for very specific solutions. They don’t say, "Give me a SOA." They say, "I need a message router," "I need a message bus," "I need an ESB," or "I need a services framework." Often, people have very specific requirements and are very much cherry-picking the best-of-breed components from the open-source tool set.
Gardner: So, it’s the Swiss Army Knife approach.
Strachan: Exactly.
Gardner: That works well for developers. I wonder over time how Camel might find itself being used in terms of operators, specifiers, and architects on the operational side of things. Do you see this moving in that direction? What’s the short-, medium-, and long-term implementation roadmap that you foresee for Camel?
Strachan: Today it's very much developer-focused. The current Camel is aimed at developers who are savvy with Java, XML, or Spring. What we want then is to gradually raise the abstraction level, so the architects and operational people can monitor services, monitor business processes, perform operational behavior, plus be able to design and deploy EIPs in a simple way.
For example, building tools and making it easier to design EIPs, as well as to choose a monitoring tool, are important.
Gardner: How would the specifying roadmap shake out? Do you think this is something that would get bundled in or integrated? How would this be something easily rationalized in production environments for those people that are going to be looking at large data center implementations?
Is this something that would be part of another grouping of products, other Apache kind of projects? How would this combine to make a whole larger than the sum of the parts?
Strachan: What we’ve seen over the last few years is a huge diversity in how people actually deploy products. Maybe five years ago, people assumed the world was J2EE app servers and everything would deploy in a J2EE deployment unit. These days, we see people make their own deployment units. We're seeing a lot of people move towards OSGi bundles. So we see a huge difference in the ways in which people deploy software now.
One of the reasons we’ve taken extreme steps to split Camel up and make it a very simple and easy to use bundle is that we know that there is a huge range of deployment platforms. Some people want to use Camel within their own smart end points. Some people want to host it inside the message broker, in the ESB and so forth. So, we see it going into production in many guises and many different forms.
Gardner: In IONA’s case, how would this be used in the context of its offering? I'm assuming that, as with other IONA offerings, it will be the commercial open-source support approach.
Strachan: Yes. Camel’s pretty much the core foundation for smart routing and mediation within an entire product suite. We have a product called FUSE, an open-source SOA backplane that provides a services framework, a message broker, an ESB, and a mediation and routing engine. The mediation and routing engine is based very heavily on Apache Camel, and we're using Apache Camel throughout other projects as well.
Gardner: Are there forums or community sites? How are some of the dialog and collaboration around this taking place?
Strachan: If you go to http://open.iona.com, there is a whole raft of documentation online forums, a wiki, and so forth.
Gardner: I know this is fairly early, but maybe this will be a good time to lay out some of the milestones. Are we expecting some significant developments and/or visions within the overall Camel project? Can you give us a bit of a timeline?
Strachan: It’s still fairly early for the timeline, but the next version of Camel 1.2 should be out fairly soon, and it's going to offer improved monitoring and management tools and particularly improved visualization and tooling.
Toward the end of the year, we’re hoping to improve various other capabilities we’ve added recently for extract, transform and load (ETL). It’s a way of loading data into systems and load testing systems with data. Plus, another area we’ve been developing what we call business activity monitoring (BAM), which helps monitor transaction flows across different systems.
For example, you might want to track that a purchase order in a system maps through to an invoice that comes out of another system. You might be processing a 100,000 of those a day, but there is one that goes along every few hours, and you want to find out which one that is within a small time window. That’s another area where we are extending the reach of Camel to do very complex and powerful monitoring.
Gardner: That's interesting, because this provides the actual routing mechanisms that provide a window into what's good, and also what's not good about how those processes are being built and used.
Strachan: Definitely. We’ve found that as soon as you have this very flexible and reusable integration patterns, it’s very easy to build on that, to build higher level abstractions, whether this is powerful testing frameworks, stimulation frameworks, ETL or BAM. We’re seeing the levels of abstraction increase to remove layers and layers of complexity.
Gardner: Is this something that could be applied to the ongoing -- and I imagine increasingly arduous -- task of dealing with semantic issues in terms of data across applications? That is to say, helping to define a mediation and a rules-based approach, such as this help in how companies can deal with that diversity and then come up with more common approaches to data and content that can then be injected into a business process.
Strachan: Definitely. This is a very large and complex topic. There are just a few bullet points on the top, and we’ve already got a range of transformation and validation capabilities in Camel, but we’re looking to improve this in various ways.
IONA recently acquired a company, called C24, which has a whole range of technologies for doing semantic mapping and transformation of different data formats. For example, they've got models for pretty much all the standard financial formats, like SWIFT, FpML, FIX, and all these other things. And, they've got other models for telcos. We’re looking to open source some of that and integrate that into Camel for doing lots of data service type integration.
Another area that’s kind of similar, but slightly different, is in the business activity monitoring ground, where you sometimes want to just check the effect of systems. So, it’s a little bit like I am doing real time reconciliation systems, and you’re doing all this transformation between systems, but you want to make sure of the end results, like that final purchase order or that final settlement instruction does contain the values that you expect. For example, we might want to check that the purchase order amount matches the invoice amount, or whatever.
Gardner: As you say, it’s early in the evaluation of Camel, but are there instances where we can see how it's being used or how developers have identified this as a productivity benefit, or are there metrics of productivity? How is it helping in terms of getting a better job done?
Strachan: We’ve got a bunch of different customers using Camel in different ways. We seem to have a few different kinds of customers finding Camel for the first time. Some come from an ActiveMQ background. They're doing lots of messaging, and they want something a bit more powerful to do more complex routing within the message broker itself.
Some come from the ServiceMix background. They are very ESB focused. They love JBI. They like the idea of standards-based ESB. They’ve got a bunch of standards-based integration components, but they’re working together, and they just want a slightly more powerful routing engine to work with.
Some of our customers have been using Mule. They found Camel a little bit easier, and they’ve migrated across. Finally, there are people who’ve just found Camel, that's all they know, and they just dive straight into it.
One of the things we’re seeing from customers is that using APIs, like JMS and JBI in particular, and to a lesser extent JAX-WS, could often become complicated, but people find that wiring things together with Camel is almost trivial. People get things going really quickly without having to read pages and pages of specifications for things like JMS or JBI, or dealing with the lower-level details of Spring.
We see quite a rapid productivity gain, and it means that people can then use lots of different technologies, whether it’s JMS or JBI without having to spend a long time figuring out things in the Spring manuals.
Gardner: Among these early users are there any vertical industries popping out, where it seems to be making sense first, perhaps financial services?
Strachan: Yes, finance and telcos definitely are big markets, but we’re also fairly strong in the e-tailers. Those areas will be our big three markets right now.
Gardner: Can you explain the business problems that they're using it for?
Strachan: Pretty much all of our customers have similar kinds of problems. They have lots of silos in applications. There are lots of messages being exchanged between systems, whether those are messages via flat files, email, JMS, legacy message brokers, or web services. They generally want to do one or more of the patterns, whether counter-based routing, or message transformations. We see more people starting to investigate a business activity monitoring feature of Camel, and that’s something I'm quite excited about.
Lots of companies, once they realize they can tie together messages to business processes, start running rules off the back of that. That has been extremely useful for people. One thing about SOA that’s quite difficult is knowing what’s really happening in your system.
Today, people have often done such things as monitor message queues or use a JMX-based management console to look at numbers and statistics. Being able to tie together all the different things flowing through your SOA to actual business processes is extremely useful and valuable for customers.
Gardner: There seems to be some murkiness in how feedback will occur, when you have events-driven architectures, and how to create common approaches to runtime and design time with this BAM emphasis that we are seeing associated with Camel. Do you expect that Camel could fill some of that role in terms of a balancing act between what takes place in operations and what needs to be brought into a service level agreement of some kind?
Strachan: Definitely. I have worked with quite a few customers on various BAM-related problems, and it does seem that BAM solves those problems. It’s quite hard to have an off-the-shelf tool to do everything people need. Almost as soon as people start doing any kind of BAM, they find that they want to put their own code in there. You might be in financing, and maybe there is a trade in U.S. dollars. You want to take that trade to the settlement instruction, but you want to do a foreign exchange conversion in there.
Almost as soon as you start doing the “hello world,” BAM-type scenarios, you tend to want to roll your sleeves up and actually do some coding of some kind, to do custom reconciliation, custom calculations, custom visualizations. BAM is very much a framework-driven thing, rather then a black-box tool. Certainly, we see people take on the Camel framework itself and then expand and extend it to do more powerful things.
Gardner: I suppose that’s part and parcel of open source, community-driven projects. They tend to be pushed out in a variety of directions. Do you have any sense of what other functional parts or functional capabilities we might see brought into Camel?
Strachan: A big area is visualization. It’s incredibly hard for people to follow what’s happening in general. Once you can create a wire tap into any system, it’s very easy to glean information from it and find out what’s happening, what its throughput is, what it is actually doing. You can then do things like trace messages around the system, correlate messages to business processes, reverse engineer whole business flows from legacy systems.
That whole area is a visualization and tooling problem. I see that as a very large area of research with the Camel project. We hope to be to be rolling out in Q4 the first batch of visualization tools. It's very much a work in progress in the next few years to get even better, more powerful operational controls and visualization engines.
Gardner: By getting more visual, it seems like we can bring more people into the process, and that maybe elevates us beyond BAM into business intelligence. Is that fair?
Strachan: Definitely. Absolutely.
Gardner: I think a lot of business people would like to get more of a real-time sense of what’s going on among these processes to provide more confidence, as they move processes from manual and older message into this new SOA approach.
Strachan: Definitely. Plus people want early warning notice of any kind of failure. One of the nice things is that you can be alerted that maybe something's going on that hasn't been found yet. Maybe it's running a bit slow today, or maybe it’s just a heavy trading day. But, it might not be, and maybe something is really wrong. So, rather than waiting for some extreme failure that results in a fine or extra interest being paid, you can respond more rapidly than that.
Gardner: Let’s just circle back quickly. Developers seem to be the core short- to medium-term audience for this. What sort of developers specifically should be more aware of what Camel’s capable of?
Strachan: I’d say anyone who’s doing any kind of SOA or distributed programming. They should at least take a good look. Definitely, anybody who ever read the EIP book. If you read that book, you’d probably just see Camel and you’ll immediately just figure out exactly what it’s doing. It’s very simple.
If you're doing Web services, you might need mediation. If you're doing any kind of messaging using multiple databases, if you have any kind of integration problems, and certainly if you’re using any kind of ESB, then definitely take Camel for a ride.
Gardner: Now, when we look toward getting more information on this, we have the resources at Apache Foundation, and also, as you mentioned, http://open.iona.com?
Strachan: Yes.
Gardner: Any other resources? You mentioned the book. Are there any other resources on this that people should be aware of?
Strachan: Those are the big three. There’s also my blog: http://macstrac.blogspot.com/.
Gardner: Very good. Well, we’ve been discussing a budding Apache Foundation project known as Apache Camel. It’s a rules-based routing and mediation engine for POJO-based implementation of EIPs.
Discussing this with us today, we had James Strachan, technical director of engineering at IONA Technologies.
I'm your host and moderator Dana Gardner, principal analyst at Interarbor Solutions. Thanks for listening and learning more about Apache Camel.
Listen to the podcast here. Podcast sponsor: IONA Technologies.
Transcript of Dana Gardner’s BriefingsDirect podcast on Apache Camel. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.
Saturday, August 11, 2007
SOA Insights Analysts Discuss Likely Future of SOA at Open Group Conference
Edited transcript of weekly BriefingsDirect[TM] SOA Insights Edition podcast, recorded July 23, 2007.
To learn more about BriefingsDirect B2B informational podcasts, or to become a sponsor of this or other B2B podcasts, contact Interarbor Solutions at 603-528-2435.
Transcript of Dana Gardner’s BriefingsDirect Podcast on The Future of SOA. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.
Listen to the podcast here. To learn more about BriefingsDirect B2B informational podcasts, or to become a sponsor of this or other B2B podcasts, contact Interarbor Solutions at 603-528-2435.
Dana Gardner: Hello, and welcome to a special presentation of BriefingsDirect SOA Insights Edition, an ongoing series of podcast discussions on Services Oriented Architecture (SOA)-related news and events with a panel of industry analysts and guests. I'm your host and moderator, Dana Gardner, principal analyst at Interarbor Solutions.
We are recording this special podcast on "The Future of SOA" before a live audience at the July 23rd, 2007 plenary session of the Open Group’s Enterprise Architecture Practitioners Conference in Austin, Texas. Welcome to you all.
Our panel today consists of Eric Knorr, executive editor-at-large at InfoWorld. Also, Tony Baer, principal at onStrategies; Todd Biske, principal architect at MomentumSI, based here in Austin, and, Beth Gold-Bernstein, vice president of ebizQ Learning Center. Welcome to the panel.
Our topic today is “The Future of SOA.” We want to take a look at what might happen if a lot of the things we’ve been considering for SOA actually happen.
One of the interesting things we heard this morning from Dave Linthicum was that he's anticipating that, in as few as five years, the role of enterprise architect and the role of SOA architect will meld.
I thought that was a little ambitious, and I wonder if I could put that to our panel. Before we get into the future of SOA, we should actually determine when the future of SOA will be important.
Beth, what do you think about five years? Do you think we are going to see SOA actually become part and parcel of all enterprise architecture activities?
Beth Gold-Bernstein: Five years is definitely ambitious. All the polls that we’ve done at ebizQ have shown that the market is really in the early stages of adoption. From my point of view, the sooner the SOA architect’s role is rolled into enterprise architecture in terms of governance the better. It is a subpart. And it is, as Dave said, best practice in architecture. We’ve known that for a couple of decades, actually.
We are getting there. Standards have gotten in. But there is more to that than just architecture. It fundamentally changes the way we create applications. That means developers need to change the way they are architecting applications, and that’s very different. It's going to take quite a while until we build up the different levels of services.
Gardner: I’ve always thought that SOA was a journey, in which you never actually get to the destination. Todd, what do you think? Is this an ongoing journey, or is there a future point for SOA when we can determine, “We’ve made it; it’s working”?
Todd Biske: I definitely agree that it's a journey. One of the things that I have seen in my work as a consultant, and prior to that as a practicing enterprise architect in a major enterprise, is that you're always going to have these new initiatives come along that may start out from a point solution.
Just as was said this morning, if solves a small percentage of my problem, if you're only looking at a narrow scope, and you’ve got boundaries around it, you are not seeing the whole enterprise.
So, in enterprise architecture practice the enterprise is not going to go away. It will continue. It’s always a journey that we have to integrate, whether it’s SOA, BPM, or any of these efforts into those long-term initiatives that keep going on.
If the company goes away, well your SOA probably will go away with it or at least be folded into someone else's. But definitely it’s a journey that involves culture change, and that takes a long time.
Gardner: There's an interesting whitepaper being delivered at this event, and it’s part of this notion of "boundaryless information flow." It appears to me that we're all going to be creating new information on an ongoing basis throughout our organizations. So, the boundaries will always be created, and therefore will need to be torn down.
Let’s go to you, Tony. What do you think about this notion of "people" as an important element about the future of SOA?
Tony Baer: There’s no question about it. You and I have been bouncing off each other a new acronym -- and I’ll have to credit you on it -- human-oriented architecture (HOA). Those of you who have been looking at the headlines over the last few weeks, have seen that the usual suspects on Oasis -- IBM, SAP, Oracle and others -- have formally said, "We are going to submit a proposal to incorporate human workflow into BPEL" -- actually into SOA in general.
It’s a recognition on their part that very few processes are completely automated. That’s one side of the story. The other side is that, when you are looking at defining processes, you have to take a look at the context of people's roles, whether the process is automated or not. So, it goes both ways. You can’t have SOA without people, whether you call it human-oriented architecture or whatever.
Gardner: Alright, Eric, to you now. We might actually drop the words, "Services Oriented Architecture." We might still call it generally, "architecture," but it does seem as if this is always going to be a chicken-and-egg relationship.
As we get more kinds of processes, we're going to come up with different standards to try to tear down different walls around siloed platforms or information stores. If we don’t call it SOA in five years, what do you think we should call it?
Eric Knorr: Dave had it exactly right this morning. It’s something that will be subsumed within enterprise architecture, as a whole. It’s part of the ongoing saga of making IT more efficient.
If you listened to Dave’s keynote this morning, he also mentioned that out of the implementations that he had seen, 80 percent were in trouble. Adoption, on a broad level, is still relatively at an early stage right now.
So there is a certain danger right now in SOA in terms of the acronym and its impact on the industry. It maybe another one of these initiatives that looks like it’s going to be very successful. You go through the hype curve, and then it begins to fall away. When that happens, I think it will be absorbed into enterprise architecture. The basic principles won't go away.
Everybody is gravitating toward service orientation within the enterprise, and there are all sorts of reasons why that makes sense from management and architecture reasons, redundancy development, and other things. That will continue to go on, but as a trend, it may have a definite life span -- and that may be only a couple of more years.
Gardner: An interesting observation from the last presentation from Deutsche Bank was that the costs seemed to be going down, according to their forecast over the next several years, and that benefits are going up, at least in terms of one subset of SOA activities. That would lead us to believe that, at least from Deutsche Bank’s perspective, SOA is going to be an ongoing productivity benefit.
If we are able to create a more boundaryless enterprise, if we can get people to work more harmoniously with processes, and agility and change become de facto attributes of IT, then we really will be coming up with something different.
So, let’s look at that. Let’s assume that much of what we’ve heard today is prologue to the future, whether it’s five, seven, 10 years -- and whether we call it SOA or not is not so important.
How is this going to first affect the IT department? Then how is it going to affect the business? Then, take a step further and ask how this might change the characteristics of what we consider companies and how they relate to one another.
Let’s start with the IT department. Todd, you’ve worked in an IT department. If you had a boundaryless information flow, if you had agility, and you could have IT work at the pace of business, what do you think the impact would be on how IT departments actually behave?
Biske: It would be a dramatic shift from what we see today. As Beth pointed out, and other people have said, adopting SOA is a fundamental change in the way that IT operates. It’s a cultural change, and an example that I point to is the notion of a service provider.
During the Austin Energy presentation today, I asked, “A power company is a service provider. They are used to that concept. How does IT act as a service provider, whether they are doing internal services used by their colleagues on another application or are producing services that are being used by customers, business partners?"
We're used to building a solution, putting it into production, and going on to the next project. IT is a very project-based culture.
If you move to SOA, you have to shift more toward a product-based culture, where you have a life cycle that goes on over multiple versions and doesn’t end until you take that service out of production.
You have to deal with customer management, looking at the different scope -- one of consumers and how they utilize those services. How you leverage that information in building new services is where it may tie into your business intelligence (BI) efforts and technologies associated with that. This move from a project-based culture to a product-based culture will be the biggest shift.
If you want a good example, look at companies that practice product management, and at the things that they sell, and you will probably gain a good idea on how IT needs to operate.
Gardner: Beth, you’ve mentioned that you think there are a number of technologies that are going to be important in terms of how they relate to SOA activities. What’s your take on how this agility that we’re anticipating will affect the adoption of technology within IT departments?
Gold-Bernstein: There are related technologies on this. BI is one of them. It’s very important to get the business value out of the SOA infrastructure. BPM is another that is separate technology, but related to SOA.
Very directly it can be used to create the overall end-to-end process. While underlying services are delivering the functionality of the overall application, business process management (BPM) will give you visibility and monitoring.
Also, event-driven architecture (EDA) processing is a separate discipline, but an event-driven SOA delivers higher agility. It provides even predictability, and the technologies are coming forward.
So once you have the SOA infrastructure down, the discipline of creating very loosely coupled components, and layering on top of that BPM and event processing -- then you will be able to deliver to the business the value, the intelligence, the visibility, the monitoring, the predictability that will make the business more agile.
Gardner: Are you anticipating that, as SOA is adopted successfully, it acts as a catalyst, accelerant, or a multiplier to the adoption of other technologies that can then be brought to bear on business issues?
Gold-Bernstein: Absolutely.
Gardner: What’s your take on that, Eric? Are you as optimistic, or do you have any cynicism on that?
Knorr: I wouldn’t say I am cynical about that, but what Beth says does make sense. It’s interesting, too, to see what happens to applications like BI that right now have their own proprietary integration infrastructure in the enterprise.
That’s a big part of what the BI players offer. What’s left? Well, the analytics. As you see SOA methodology spreading through the organization and the ISVs, you'll begin to see a more component-oriented way of developing applications that will permeate the commercial software vendors.
That’s a very long-running project. SAP has been talking about that since 2000. Oracle seems to have jumped in a little bit further with the Fusion platform, while that remains to be seen in the way it plays out. But I do think it will penetrate those areas, and the event-driven angle is very important.
Gardner: Before we leave the implications for the business, we also need to step back and revisit what Dave Linthicum said this morning about other mega trends. He mentioned Software As a Service (SaaS). We have heard other blanks or X’s "as a service," whether it’s integration, platform, or infrastructure.
What do you think will be the impact on IT and the business, as SOA not only opens the floodgates to some of these other technologies, but also opens the floodgates to more ways of acquiring and consuming services outside the organization?
Tony?
Baer: In a way, that’s déjà vu all over again. I remember that during the bubble, when we were all having a lot of fun, we were talking just B2B exchanges. It was the idea of, “Well, you can find something.” It was like an eBay for the supply chain.
You can see how far that went, because it went against established practice in the supply chain. Twenty years of supply chain management theory was: Consolidate your partners, get to know them better, and don’t partner anonymously.
My sense is that what has to come of all this is not just a random coupling. Ten years ago we said all these components out there are not going to just couple randomly. There are going to be partnerships. We were starting to talk the other day about semantic integration, but behind every successful semantic integration is a successful human partnership.
Gardner: So part of a SOA success trajectory would be the ability to consume, as an enterprise, services in a marketplace, where such pressures or forces as competition and a drive for lower costs and higher benefits could play quite nicely.
Baer: And partnerships …
Gardner: And partnerships, so that you find areas where you might overlap with someone in your supply chain. And you must start handing off some of your services, and you might adopt some of theirs.
Baer: Or, perish the thought, start within your own enterprise -- if you can get beyond all the political roadblocks.
Gardner: There are political roadblocks inside of your departments and your companies. Imagine the political roadblocks you’d fight trying to partner with others companies entirely?
Baer: Exactly.
Gardner: So we do come back to this "people" and "what I learned in kindergarten"-mentality.
All right, so let’s not think SOA is all just peaches and cream. You mentioned this semantic issue, Tony.
If I've got a Tower of Babel inside my enterprise, or in a department, where different types of doing things or identifying them are scattered across whatever someone puts in an Excel spreadsheet, then we have this ongoing battle around how we identify that which we want to use, consume, or produce within services.
Todd, let’s get back to you. Let’s discuss what I believe is an inhibiting factor, which is this whole semantic issue. What do you see happening around standardization that is a counter force to that?
Biske: I don’t know that it’s so much standardization that’s the counter force. Certainly it is, but I look at the information integration problem, or master data management, enterprise, logical data model, whatever you want to call it. It's actually a good space to look at and say, “Okay, what do we need to fix to do SOA right?”
Groups trying to do a master data management (MDM) effort or come up with an enterprise canonical data model create this group that goes off on an island. They sit there, lock themselves away, and try to come up with this uber-model, but they miss making the connection back to the people that are working on the projects that actually need to use that information.
It's pointed out that it takes too long to get it done, or it's not meeting my needs. The same thing can happen with SOA if you’re trying to define services in the enterprise context. We don’t want this to happen.
In both of these cases, we need to figure out how to make this information relevant to the projects that need to execute properly and take those incremental steps to get us there.
Clearly, having a consistent semantic model is critical to the success of SOA. If we don’t get the consistency across our services, we wind up creating more work for the consumers. And as the previous presenter from Deutsche Bank said, it’s all about consumption. It’s not about producing. It's about creating services that are easy for our consumers to use.
Gardner: Beth, how do you think that some of the technologies that you mentioned earlier, intelligence and management of applications and services, can be brought to bear on the semantic issue? They seem somewhat divorced at this time.
Gold-Bernstein: Actually a different technology that’s being brought to bear is semantic integration technology. There are standards bodies. I know that the Open Group is working on some of that as well, and some vendors have put forth some semantic integration or enterprise information integration (EII) products.
Software AG came up with one. They have inference engines underneath them. I can remember being at the conference when IBM introduced SNA and told everyone, “You have to build your enterprise data dictionary.”
I worked with organizations that did that. They had the books on their shelves, but it didn’t do anything. They were just books on the shelves. The difference now is that we have the integration technology, and with these semantic integration technologies, we can build these taxonomies incrementally on a project-by-project basis, and then add to them over time, grow this in an organic way, and still be able to move forward that way.
I think that will be the successful way. Semantic integration is not receiving a lot of press. Software AG brought out a great product a year and a half ago or so, and is going nowhere with that so far. That is the future and a very important one, because a dirty little secret about integration, is that 80 percent of the budget is spent on semantic integration.
Gardner: Eric, in addition to the semantic issue internally, as we discussed, being more influenced from outside the organization, we already see popularity of things like mashups, RSS feeds, and content brought to bear on business processes.
Do you think that, as SOA matures and we look to the future, there needs to be a delineation between internal and external content, and who's going to be in role of managing that boundary?
Knorr: I'm not sure who's going to be in that role, but mashups have an immediate role now for SOA adoption. Basically, if you were able to wave the magic wand today and transform your entire application infrastructure to make it SOA-based, you wouldn’t see any difference.
So mashups are a great selling tool. They may not be strictly SOA, but if you got a couple of internal data sources, and maybe Google Maps on the outside, and you throw some Salesforce.com data on there too, you can begin to illustrate for upper management, what agility looks like. That’s one good thing about mashups.
I've been surprised in some of the panels that I have done, with the resistance you get, even to something like Salesforce.com, which is a fairly well-established application right now in the marketplace.
Gardner: It’s a business application right?
Knorr: Right. But, back to the mashups, I think a good way of looking at mashups from a practical perspective and how they fit into all this is that there is a lot of rogue application development going on there under the radar, that nobody in upper management really knows about.
So mashup platforms -- whether they are from BEA or IBM -- they have their first iterations of those types of platforms, or outside the firewall, where you are actually doing development on the outside. These provide a framework for the rogue application development.
Eventually this kind of stuff outside the firewall will be folded into the greater SOA somewhere down the line. In a way, that’s really what's most exciting about SOA, and most difference is the ability to begin to connect to those external services and bring them into the fold.
This goes back to the early days of web services. XML grew out of a desire to open up EDI, and there was this whole thread of web services that were all about B2B connection. Then came Microsoft, saying that one day the Internet will be the operating system, and we will build applications on top of it. Well, that’s interesting to see.
Gardner: I suppose another way of looking at this is that the pendulum keeps swinging across several major variables, one being distributed, and then consolidated. And we go back and forth. And another is complexity that gets solved, but then simply opens up another type of complexity that needs to be solved arrives.
And so if SOA is successful, it seems like we're dealing with a complexity of integration, but then that opens up the complexity of semantic issues, and people and behavioral issues, and then boundary and political and government issues.
So assuming that we can get boundarylessness on this sort of scale, and we have the pendulum going back and forth among these complexities, does the business recognize enough return on investment to say that SOA was worth it? And when will we reach that sort of an economic business rationale? Tony?
Baer: I hate to say this, but I just recalled what David Linthicum was saying this morning about the inherent tension between-project based SOA and the enterprise architects. We all seem to be talking past each other.
To satisfy your client in the business, you are going to show them that it’s deliverable. And the client is saying, "I am not paying for your enterprise architecture. I want a solution that helps me right now that’s a sure fit, an 100 percent fit. I do care if you make it 80 percent, and I have to pay the cost of generalizing it to the rest of the enterprise." So, there is an inherent tension there.
I like what Eric was saying about using mashups as a step forward. That’s what you were saying about semantic integration. We need to work this from the ground up instead of the grand enterprise data model.
We have to take an incremental approach, and don’t try a project to boil the ocean. Then, after you’ve done that, if you can somehow sell it to the business, there might be some internal budgeting mechanism or brownie points, where there can be some sort of internal trading system. And then maybe there is a way to subsidize that extra 20 percent of development.
Gardner: Now, wait. I thought that SOA was going to make agility the number-one requirement that we are going to meet in IT departments. Whether you're Deutsche Bank with a five- to seven-year window on recovering costs -- or a bit more of a common company that has to deal with quarterly returns, and are seemingly always under pressure to cut costs -- there's got to be a better business payback here.
Does anyone, as we discuss the future of SOA, have a sense of where the business people recognize that this is successful and we have made it? What sort of benchmarks would that require?
Knorr: I have a comment on that one. It all has to come back to the business strategy. There are a number of organizations out there that don’t have an enterprise architecture practice, yet, they're trying to do SOA.
As a result you have SOA applied to projects, and you are missing the point that’s on the strategic side of this. We've heard the term "boil the ocean."
We can go to the opposite extreme as well and say, “Okay, I am going to just redo all of IT. We're going to change all of the fundamentals behind this and do it all at once.” That’s not going to work, to completely do the opposite extreme in saying, "everything is ground up."
You need this middle-out approach, but it has to be driven by the business strategy. The way to determine "Have I been successful?" is, "Have I been successful in adopting my business strategy and meeting my business goals?" If I have, then I am doing the right things.
Every enterprise is going to vary in the extent to which IT contributes to those goals. For some organizations, IT may be a strategic asset. There may be lots of specializations that do what I would call a vertical domain that makes sense to them.
For another organization, IT may not be so much of a strategic asset, and they just use it for the standard, horizontal features of HR and business support services.
The best thing for them may be to go down an outsourcing route with a company, saying, “Okay, make sure you are able to keep up with what we need by adopting SOA on your end. But that’s really a part of you being a good business, not necessarily our being a good business.”
It all comes back to what the business is trying to do, and to try to understand how IT can contribute to that solution. If I don’t have any idea on how IT contributes, I am never going to be able to say I was successful or not.
Biske: I think we focused a lot on organizational resistance today, but that’s not to say that there isn’t a lot of success out there in the marketplace right now.
Where you see that success is in telecom and in financial services. Financial services, of course, because they have the money and they are always first, but also because you are talking about a relatively heterogeneous array of services that they can create, recombine and put together into different products. The market demands that they come up with a very broad variety of products that they can deliver to their customers. They are seeing that agility quicker than anybody else.
Gardner: A mega trend that's with us these days, but we didn’t bring up yet is globalization. Companies are competing in ways that they never had to before. So perhaps competition, the ability to compete and win markets and outflank your direct competitors and partners efficiently, and to do mergers and acquisitions well because your IT department can keep up with the business strategies -- perhaps these are the big payoffs from SOA.
Perhaps we'll we see companies effectively embrace what we’ve talked about today, make agility a de facto means through which IT performs at the pace of business. And then they become big companies, get bigger and better, take over more customers, deliver better services, partner across a larger ecology, become masters because they are at the forefront of their markets, and other players become slaves.
Or, am I being a little bit too loose? What do you think, Tony?
Baer: In this thing you were talking about -- globalization, M&A scenarios -- sometimes circumstances have a nice little way of concentrating the mind.
When you all of a sudden are faced with putting two organizations together, which happens pretty often in the business -- M&A is not exactly the exception these days. At some point you have to say, "Look, we need to take an architectural approach."
"Our tried and true methods have been tried and they are true, but they may not be valuable. We keep just going back on our traditional way, our traditional path of execution, and we're just going to develop ourselves into a brick wall."
Gardner: Eric, do you really think that if you are a successful SOA company, you are going to just drop the SOA and just say, “We’re a successful company?”
Knorr: A successful SOA company as what?
Gardner: As someone who executes well on what we have been discussing -- SOA. Does that really make you a successful company?
Knorr: Absolutely. It can contribute to it in terms of agility. There's no question about that. I can’t imagine somebody hanging out their shingle as an SOA company and having trouble getting over that one.
Gold-Bernstein: You asked when a company can say they are successful? In my view, every SOA project needs to contribute to the benefit of the business. It's not down the road five years. Everything you do needs to show benefit, and incrementally. There are different styles of SOA.
We talked about mashups. There is interactive SOA and event-driven SOA. So you are monitoring things and you are giving business intelligence. There is process-driven SOA, with which I'm optimizing my business processes or I'm using a horizontal business process and reusing what I already have.
So I may be building my architecture incrementally, based on different business problems. It's just the approach I'm taking to give more agility. All along the way, to be successful at all, you have to deliver business value for every single project. I don’t think you can wait at all.
Gardner: Okay, before we go to questions from the audience, Todd, for an IT department that can go to their business leadership and say, "You tell us what you want us to do and we will help you do it," versus one that says, "Get in line and wait, give us a list of requirements, and we will get back to you" -- which one is going to be around in five years?
Biske: I don’t know that either of those would be around in five years. We're talking internal IT. I don’t like the notion of a customer-supplier relationship.
It has to be a partnership. So it's not that IT comes to the business, or the business comes to IT. IT is part of the business. And they all sit down at the table together and have equal discussions on what's the right thing from a strategic point of view for that business.
If ever it's the case of just me going and saying, “Give me your requirements, and I will build what you need" ... that’s like saying, "Why don’t you go ahead and outsource me, because someone else can come in and ask you that same question and probably leverage economies of scale better than I can."
If it's the case of "it's pushing too far," you're not creating that shared understanding of the technologies involved to really enable the business to contribute. It has to go in both directions, you have to mutually understand -- IT has to understand the business better, as well as the business side has to understand technology better.
Gardner: Let's take some questions from the audience. This one is, “SOA can’t exist without governance, and there's been a lot of discussion about technical reference architectures. Where are the governance reference models? Are we there yet with governance? Do we have something we can point to and say, 'That’s the way you do it,' and if not, why not?”
Biske: I'll jump in on that one. Working for a consultant firm, obviously we come in and help with governance. It's interesting that we heard "technical reference architecture," and then "governance reference architecture." But I actually view the reference architecture as part of governance. I like to simplify governance to three simple things: people, policies, and process.
First, you have to have some level of authority to put policies in place. If no one recognizes any authority to establish policies, they're not going to be successful. So that should be enterprise architecture or something. Certainly, at an architectural level, you would expect reference architecture material to come from enterprise architecture as the authoritative source.
The reference architecture itself now is an expression of the policies associated with architecture. I have got something concrete. Later, you are using the style, saying, "must," "must not," "should," "should not," but that’s giving exclusive guidance, creating the policies that now the development projects have to follow.
Finally, you have to have process that goes along with that. So, as Beth pointed out, I don’t want to create a bunch of dictionaries that just sit on the shelf and gather dust. I have to have the process to figure out how I actually get people to follow these policies. It’s unlikely that you're going to be able take a strong-arm view, create your police force and just have your key checkpoints where the law is laid down.
Probably it's going to come back with a lot of resistance from your developers, unless your culture is already used to operating in that manner, which, I suspect, most companies are not.
You need to create transparency. You have to have involved with the establishment of the policies people who are also the ones that work with the projects on a day-to-day basis, so there's a feedback loop from the trenches, as well as guidance from above.
You need transparency into the activities that are going on as they occur, versus sitting back, waiting eight weeks to hit the architecture checkpoint, and when it's too costly to make changes, and say "Oh, you're not doing it according to the reference architecture."
You want to have that constant communication, transparency on both sides into the reference architecture development, as well as the development of the specific solution architectures, and everybody really works harmoniously. Its not going to happen overnight, but you always have to consider your consumers.
Gold-Bernstein: Can I add one thing, Todd? I agree. One other thing you need is management, because in SOA it’s dynamically binding. You don’t know whether the policies or the SOA’s are being delivered upon. You need real-time management and visibility, and automated management. So, if something is out of governance, it’s out of policy that is reported on, and you have procedures there to act upon it.
Those are becoming very important, and the thing is that people aren’t adopting them until they’re way down the road. In the beginning of SOA, you have to begin your governance, it has to develop alongside your architecture from the very beginning.
Gardner: So this is clearly another topic for another day: management and governance, do or don’t they come together?
Here is another question: "There are continuing references to semantic data. Two approaches to adding semantic information to data include Resource Description Framework (RDF) and Topic Maps. Any thoughts on RDF and Topic Maps?"
Baer: I think its too early, frankly. I mean, at this point it’s kind of "chicken and the egg." Very often, you’ll have vendor product come out, and then standards will then follow. I don’t mean to make this sound like this is vendor-driven, but in the area of semantic integration of services, there are a handful of products out there and maybe a handful of people actually working with this at this point. So, my sense is it’s too early to call.
Biske: RDF has one advantage right now in that, there are some close ties between that and REST, and certainly REST is continuing to gain in mindshare. I'd like to describe REST as more resource-orientated architecture than service-oriented architecture, but again that’s a debate for another podcast.
If you take this resource-oriented view, clearly RDF fits very nicely into that. But again, those technologies are still just coming out of the incubation stage. How to properly apply that in an enterprise context is still quite a ways off.
Gardner: And, as an indicator of the interest on this topic of semantics, another semantic question, "How far are organizations in their internal enterprise semantic model activities, and what is the extent of adoption?"
Beth, I think, you said it’s something on the order of 80-90 percent, and you had some percentage of activities devoted to semantic issues?
Gold-Bernstein: Oh, no! What I said was that’s the budget of an integration solving semantic issues, but they are doing it manually, re-solving that every time they hook something up. It’s a huge problem, but as far as solving it at a semantic level with semantic metadata and tools automatically doing it, we’re just not there.
Gardner: So your answer to the question of the extent of adoption is "relatively low."
Gold-Bernstein: Yes. It’s not even on the map.
Knorr: And it’s one of the oldest, ugliest problems within IT.
Gold-Bernstein: Absolutely.
Gardner: And we’ve heard that the data issues need to get worked out before many of the other benefits of SOA can be.
Biske: If you're going to have composite applications in the enterprise, and they're tapping into data stores that contradict each other, it’s not going to work.
Gardner: So get your data and semantic acts together.
Biske: Where we do see a little bit of effort happening, and it's wrong to necessarily call it semantics, is in increased adoption of vertical standards, a set of XML tags for a particular vertical domain.
Is that really semantics? Probably in the purest sense. Is it a step in the right direction? Yes.
If you're trying to say, “How do I represent my messages on my services in a way that’s consistent," you certainly can’t go wrong with starting with an industry-specific model. Is that the full way to leverage semantic technologies? No. You are a long way from that, but it's a step in the right direction.
Gardner: It seems to me that if you want to carve out a leadership position in your vertical, that you will take on the heavy lifting of semantics, and then extend it across your ecology.
Gold-Bernstein: But it doesn’t solve the problem for all of those packaged applications, home-grown applications, and everything else you are trying to reuse and integrate with. So, that’s the real semantic problem.
Gardner: Okay, last question, we’re out of time. "We talked about product-based versus project-based cultures. We have a plethora of products and enterprises, they have to continuously evaluate and determine the right fit. So, this is saying, 'We have a lot of product strategy activities, but if we productize services, won’t that result in a similar world of confusion?' And if we can’t even decide on what products to take to market and when, how can we decide which services are appropriate in order to allow us to then fight over which products to take to market?"
Biske: Let me make one quick answer to that, and let everybody else jump on me. By that logic, eBay should never have worked.
Gardner: Explain.
Biske: The whole idea of eBay and this marketplace of things that people are selling, they are selling probably new products. How are you going to make sense of it? How are you going to find it?
Somehow, eBay has put together mechanisms that are partially based on timeliness. There's some very good semantic integration and very good metadata management. Somehow, they put together what should otherwise, by conventionalism, be an impossible task.
In an enterprise, the process isn't easy, but the scope of the problem is going to be a little more containable than eBay.
Gardner: I hear another mega trend here, and that is social networks and the wisdom of crowds applied to services and products with governance, so that we have somewhat of a free-market approach that allows the wisdom to rise to the top?
Biske: I'd actually turn it around a little bit. The original question was part of the reason why we have all this problem in choosing products is that we didn’t have a product-based view to begin with. We didn’t have a capability-based view of, "These are the capabilities I need. Let’s go find the right thing to fit that." Instead we said, "Let’s go pick a product, and now look at what capabilities it gave us."
If you begin with this capability-driven view, now the products just naturally fit, whether they are internally developed, purchased externally. It creates the opportunity for more of a marketplace, leveraging all the conversations about it that the social networks may create for evaluating and selecting things appropriately, whether its internal or external.
I think we should have the same conversations about internal service providers and the quality of service they provide that we have with external providers.
Gardner: Well, we will have to leave it there. Thanks, you’ve been listening to BriefingsDirect SOA Insights Edition.
I want to thank our panel in a live presentation before the Open Group Architecture Practitioners Conference. We’ve had Eric Knorr, executive editor-at-large at InfoWorld; Tony Baer, principal at onStrategies; Todd Biske, principal architect at MomentumSI, and Beth Gold-Bernstein, vice president of ebizQ’s Learning Center.
I'm Dana Gardner, your host and moderator, principal analyst at Interarbor Solutions. Thanks very much for listening.
Listen to the podcast here. Produced as a courtesy of Interarbor Solutions: analysis, consulting and rich new-media content production.To learn more about BriefingsDirect B2B informational podcasts, or to become a sponsor of this or other B2B podcasts, contact Interarbor Solutions at 603-528-2435.
Transcript of Dana Gardner’s BriefingsDirect Podcast on The Future of SOA. Copyright Interarbor Solutions, LLC, 2005-2007. All rights reserved.
Subscribe to:
Posts (Atom)
