Rapidly Evolving IT Trends Make Open, Agile App Integration More Important Than Ever

Transcript of a sponsored BriefingsDirect podcast on the maturing of open source integration software and its role in making enterprises responsive to a rapidly changing IT landscape.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Register for CamelOne. Sponsor: FuseSource.

Dana Gardner: Hi, this is Dana Gardner, Principal Analyst at Interarbor Solutions, and you're listening to BriefingsDirect.

Today, we present a sponsored podcast discussion on how enterprise integration requirements are rapidly shifting to accommodate such trends as cloud computing, mobile devices' explosion, and increased demand for extended enterprise business processes.

Application-to-application integration inside an enterprise's four walls is well understood, but very quickly the demands placed on integration are spanning multiple enterprises, multiple types of applications, and varieties of service providers. Software as a service (SaaS) and cloud computing are joining with legacy systems to form new and varied hybrid models that require whole new sets of integration needs and challenges.

Once these newer breeds of integrations are set up, can the old, brittle management and upkeep of them suffice -- or will agility and rapid upgrades and innovations require new tools to make integration a lifecycle function with ongoing management and more automated governance?

In this discussion, we'll examine how open-source integration projects like Apache Camel and lightweight integration implementations and graphical tools are making developers and architects more agile. At the same time, these open-source approaches are proving less vulnerable to the complexity, fragility, and cost that often plague aging commercial middleware integration products. [Learn more about the CamelOne conference May 24 in Washington, DC.]

Here to examine the new need for open and agile integration capabilities is Rob Davies, Chief Technology Officer at FuseSource. Welcome, Rob.

Rob Davies: Hi, Dana. Good to be here.

Gardner: We're also here with Debbie Moynihan, Vice President of Marketing at FuseSource. Hello, Debbie.

Debbie Moynihan: Hi, Dana.

Gardner: Debbie, what's going on out there? Why are things happening so rapidly? Why do people need to rethink integration?

Many challenges

Moynihan: Dana, there are so many things happening out there -- integration, in particular. There are so many challenges right now. The business models are changing and people are being asked to do more with less. Teams and applications are more distributed than they have ever been.

There are a lot of new technologies coming out that people are struggling to learn about, and figuring out how to incorporate them into their infrastructure: cloud, mobile, the explosion of the huge amounts of data that enterprises are trying to understand and make sense out of. Not to mention the social media technologies that people are being asked about and wondering how to incorporate into their enterprise infrastructure.

There are a lot of different skills that people are looking to have that they've never been asked to have before. More and more people are being asked to perform IT tasks. It isn’t just highly skilled developers, but also business analysts and people who have never done integration before are being asked to do integration activities.

A lot of people are looking for solutions and ideas. They're not sure how to keep up with all of these changes. Costs are a problem because essentially everyone has the same or smaller budget going forward and a lot of people have fewer people to do what they've been doing before.

People are challenged. I think open source is a great solution. At FuseSource, we've seen a lot people looking more and more to open source to solve some of these problems.

The reason why open source is a good solution is that with open source there's a lot of flexibility. When the environment changes and new technologies come out, you need to integrate new things into your environment.

The community people, when they see a problem or new technology, just make it happen. They can add, expand, and modify what's involved in the various open-source integration projects without the overhead and bureaucracy of some of the traditional software development environments.

Gardner: Debbie, in the past, when we had a shift in computing, we'd bring in a new set of applications, we'd update our platforms, and then think about integrating them. It was a sequential process and it could take three to five years to go through something like that.

We don’t really have that luxury anymore. Now things are happening in a simultaneous fashion. So integration really can't be an afterthought, but needs to be part and parcel with how you go about designing and implementing your applications.

Doesn’t open source, in a sense, allow for a compression of the time that we’ve traditionally taken with commercial products?

Moynihan: Absolutely. Open-source is a componentized, lightweight approach. As people develop their applications, they develop them in such a way that they can be broken apart in new and different ways down the road, and it's very transparent. It makes it easier over time to further integrate what you’ve built and to make changes as you need.

Gardner: Rob Davies, let's dive a little deeper into this notion of open and agile integration capabilities. What's wrong with simply going into traditional, commercial integration capabilities and somehow broadening them into this new domain? Is it not something that can be extended?

The pace of change

Davies: I think it is, but I think the real crux of the problem goes back to what Debbie was talking about earlier -- the pace of change. If you’ve got an open-source framework, you can actually have an insight into how the project works.

After we launched Apache Camel at the Apache Software Foundation, we provided a number of default integration components for Camel. But, as soon as they got out there and the community started to use them and saw the benefits of using them, we saw no end of contributions. People contributed adapters to weird and wonderful systems, and contributed them right back into the Apache project. [Learn more about the CamelOne conference May 24 in Washington, DC.]

Then we’ve got other components that people use to automate open-source but not at Apache. A number of components have grown rapidly since the inception of a project. When we started, we had probably 20 components, and now it's well over 100. Those are the ones we know about, the ones that people have open-sourced.

We know from our customers that they’ve got specific needs. They’ve got legacy applications. Because we've gone to the effort of making sure that it's very easy to add a new component into Apache Camel, it's very straightforward for someone to add in extra functionality.

For example, if you want to write a component for legacy mainframe application, you could very easily do in a matter of hours. The old approach would take you weeks, months, maybe even years, especially if you don’t have access to the source code. So, you’ve got that added flexibility.

The fact that it's an open-source project at Apache means that there is a vibrant community of users and developers.

The fact that it's an open-source project at Apache means that there is a vibrant community of users and developers. You can get feedback instantly, if you’ve got issues and problems. Of course, if you want professional help, there’s FuseSource as well. We have our own community at fusesource.com. So, all these things combined means that you have more flexibility and a much more agile way of doing integration.

Gardner: You know it strikes me that when we begin to talk about integration that I’d mentioned service-oriented architecture (SOA), but that was sort of yesterday’s buzzword. We're now into cloud, hybrid, and mobile. But, from an architectural perspective, you can't really scale and leverage these open components without that proper underpinning, typically an enterprise-service-bus (ESB) architecture.

Rob, help me understand why doing this correctly from an architecture (not just an open-source) perspective is really important as well.

Davies: You hit the core things about the SOA and the ESB architectures. We see where people are using, in particular, Apache Camel and some of our other open-source projects. They want flexibility there. So, they want to leverage a service bus, put things on, expose them as service, and expose them over the service bus, which uses different transports to enable that bus, be that messaging, HTTP, or whatever other means you want to use.

Application integration

At the same time, you also want to have the flexibility now to do it in application integration. You want to have that flexibility for some services and you very much need that enterprise service bus in place. But for other cases, you want to be able to do that more locally, where the integration points are.

The approach that we have is that we enable you to do both, because you can embed Apache Camel inside an application server, if you want it inside your application itself. If you want to use it in a more traditional sense, you can deploy it into ServiceMix. You can define your apps easily, deploy them into ServiceMix, and use it to manage the container.

Having that flexibility as well means that you can have the right architecture for your particular solution. If you look at how people would do the integration before, they’d have to get an ESB, and that would force the whole architecture of how they do things. When you’ve got more flexibility, it means that you can make the right architecture choices that you need, and you're not constrained to one particular style of integration.

Gardner: I'm facing a lot of questions more recently about how to cross the domains that we've mentioned -- SaaS, cloud, on-premises, traditional architecture, and private cloud architecture.

Does the service-bus approach and the open-source approach also give us some sort of a path or vision for how to go about this? I think we're just starting to enter into how to integrate my legacy applications with cloud or SaaS applications in a meaningful way? What are your thoughts about that, Rob?

You can only really get that speed of innovation to keep up with the way the environment is changing by choosing open source.

Davies: I completely agree. Having open source enables you to have the insight into how the integration application works. But more importantly, those environments are changing very rapidly.

If you just look back just a couple of years, when people were starting to use the cloud, they weren’t even thinking about having hybrid clouds. Now, we're seeing more and more people, more of our customers, looking to hybrid clouds and have a private cloud for applications.

When they need the capacity, obviously they can get that capacity in a public cloud. But, to have all those PCs working together seamlessly, they need the agility that you get from an integration solution that can be deployed on a public cloud, locally, or a combination of both. That’s something that you can only get from software that has evolved at the same pace as the demands of the environment.

You can only really get that speed of innovation to keep up with the way the environment is changing by choosing open source, because the open-source community itself is driving the projects to keep up with the demands.

So, you have to try to move outside of a traditional release cycle that you would get from a traditional product company. You don’t really have any other alternatives, if you want to keep up, than to look at open-source projects, the Apache ones in particular. [Learn more about the CamelOne conference May 24 in Washington, DC.]

Apache projects certainly hit the right notes in that you've got both very business-friendly license from the Apache license and very active communities, and you’ve got diversity in that community. You know these projects are going to live beyond the lifetime of particular individuals on the projects.

Support and consultancy

You also have the benefit of having companies like FuseSource, which created the projects in the first place, and who are there and able to provide support and consultancy if you need it. You get the best of having a dynamic community, a dynamic project, and you also get the security of having professional company to back it up.

Gardner: I'd like to revisit that thought about the traditional upgrade path in the product cycle. Many organizations have faced two stages of this. One is to wait for the commercial vendor, to come out with the upgrade or often, an association with larger projects that they have across different platforms, brings in various versions and iterations that they've done.

It's a fairly complex undertaking for the vendor, but then there is the complexity of them bringing that into your organization, and there's cost, because you have the upfront licensing cos. Many times, you’ll incur hardware cost and many times you want to have cohabitation of your older deployments, as well as new ones that come online. This is sometimes a three- to five-year process.

Tell me why an open-source approach and, from a cost perspective, that upgrade path is much different.

Davies: Because it’s open source. The projects that we are involved in Apache are Apache licensed. ActiveMQ, which is a message bus, Camel, ServiceMix, and CXF, are Apache licensed. It means that you don't have to pay the license costs upfront.

The problem that organizations are facing now is that the environments that they can deploy into and have to interface with are changing and evolving so quickly.

You're actually right about the time of the release cycle for a traditional product company. The problem that organizations are facing now is that the environments that they can deploy into and have to interface with are changing and evolving so quickly. You just can't have a luxury waiting for a three- to five-year release cycle.

And what often happens is that the software you are trying to integrate with is really out of date and people have moved onto something else. So, up front, you have to look at what you can use to integrate with these systems as they evolve. Things are evolving more quickly over time. There are different sorts of social networks that you have to interface with, and that market has been very dynamic over the last few years.

Twitter has been around for a few years, but we see people using Twitter as asynchronous communication within their organizations to give out real-time information updates. So, that’s important. Who knows what's going to be just around the corner, because things have evolved very quickly.

If you want your organization to keep pace with the changing environment we're in, you have to look for the right integration solutions right now, and choose the ones that will be able to keep pace.

Gardner: How rapidly are the iterations within the Apache project, within Camel in particular, happening? How rapidly is innovation taking place?

Very fast pace

Davies: It’s happening at a very fast pace. When we do release these out of Apache, it's typically every three months, but in that three month period there could be other components that have gone into the Apache Camel Framework. Because it's open source, people can actually look about, release their own components into an open-source environment, or develop them separately without necessarily releasing to Apache, just to get the functionality out.

That pace of change is very fast and it’s near real time. When the need comes up, within a few days or a week, you would probably find someone who has already written that integration component that you need and it’s available.

Gardner: This is, of course, a global community. You have a great number of different inputs and parties involved, different locations that are supported, and different localizations, and languages.

Davies: Absolutely. That’s another benefit of having an open-source, and a well-known open-source, community to drive our innovation and to back it up.

Gardner: Debbie, let's go look at what's happening in the community. I understand you have a conference that’s coming up May 24, a first of its kind. Why is this a good time to be pulling together the Camel Community, and what you’re going to do?

The nice thing about Camel is that it provides a basic foundation and a terminology of well-defined patterns.

Moynihan: We’re really excited. We have an event coming up in May. It’s called CamelOne and the reason why we focused on Camel with the name of the event, is that it’s actually an event for open-source integration and messaging overall. It’s because Camel is a really great way for people to get started, and it’s also a great way for more advanced integration developers as well. It brings together the entire community.

Rob was talking about earlier about how there is always these new technologies coming up and people can add components. The nice thing about Camel is that it provides a basic foundation and a terminology of well-defined patterns. The integration patterns themselves are very well-defined, but what's happening is all the different ways in which you connect and what you are connecting to have been changing and evolving over time.

Camel is a great foundation and CamelOne is an event to bring together users of Camel and other open-source integration and messaging technologies to learn more about Camel, open-source messaging like ActiveMQ, and ESBs like Apache ServiceMix.

You were talking earlier about cost savings. More and more people are being asked to do integration. The nice thing about Camel and about these other technologies is some people maybe just only needed to do lightweight integration. They can just learn how to use Camel and learn the basics.

Other people are going to be doing more in-depth management of many integration patterns and they may need to know all the nuances of an ESB platform. The focus of CamelOne is to bring people together to understand, learn about, and meet each other and to grow this community of open-source integration users.

Gardner: So, this is CamelOne, May 24, in the Washington D.C. area. Why Washington D.C.? Is there a lot of this going on in the public sector?

Central location

Moynihan: Actually, we do have a lot of users in the Washington D.C. area. We also thought that was a central location, where people could come from not only anywhere in the US but also from other regions of the world as well. There are a lot of direct flights to that location. But, we do have a lot of users in the area. For example, the Federal Aviation Administration (FAA) is going to be speaking and they have selected open-source integration for the next generation of their services infrastructure.

Since they connect with a lot of other agencies, there is a lot of interest in learning more specifically about that program and about the technologies that it's built upon, because a lot of other agencies need to connect.

Gardner: One of the other aspects of this that I'm seeing in the market is that more people need to take part in integration. It can't just go through a bottleneck of "beard-and-sneaker guys" in the back room who can do coding. Integration needs to be part and parcel with process innovation. That means we need to elevate it out to a wider group of individuals, maybe as many as possible that are on the front lines of process innovation and analysis.

The addition of tooling is going to help broaden how many people can do integration, and we're real excited.

What's being done about the integration that we've been describing? It’s wonderful that we have the open source and we have the cost benefits, but how about bringing this to a larger class of individuals, Debbie?

Moynihan: On April 11, we announced the general availability of a new graphical tooling for Apache Camel. [Users can download a trial version of the plug-in, which includes some of the functionality of the fully paid version found on the subscription-based Fuse Mediation Router.]

The addition of graphical tooling makes it easier for more people to do integration development. They don't have to write code. They can use a drag-and-drop environment to select the integration patterns that they want to implement, and the software will implement them. They can test them and deploy them into production as well.

The addition of tooling is going to help broaden how many people can do integration, and we're real excited. We've been doing a beta program since the end of January with over 500 participants. Rob mentioned the breadth of all the components and how hot Apache Camel has been. We're not surprised that more and more people want to use it. So, the idea of having tooling on top of it is really attractive to users.

Gardner: So, what's the name and where do you go to find out more about them?

Moynihan: The Fuse IDE for Camel is the name. It plugs into an Eclipse environment and you can get it at fusesource.com.

Gardner: And how about more information on CamelOne? It’s simple, I suppose search on CamelOne will get you there.

Moynihan: Yes, camelone.com is the website as well.

Gardner: Now, you guys have been involved with a series of books and you have something new coming out in that series. Tell me about that.

Camel in Action

Moynihan: There are a couple of books that recently have come out. One is Camel in Action, which is fantastic for people who want to get going with Camel and learn how to use and deploy it. Rob is coauthor of the ActiveMQ in Action book, which has come out in print recently from Manning Publications.

Davies: ActiveMQ in Action is really a scripted book, which goes through all the different use cases of using ActiveMQ, right from getting started and what messaging is about. It walks you through different deployment options, all the way up through using clusters of ActiveMQ brokers, to using ActiveMQ as a wide area network, so you can connect geographically dispersed locations.

It shows you how to tune the performance of ActiveMQ and get the best out of it. So it's very comprehensive book about how to use ActiveMQ. It's somewhat complementary to Camel in Action as well, which goes through all the different patterns you can use.

It doesn't talk about using Camel. It talks about integration patterns as well and then describes how you can use those using Apache Camel, and you can use Apache Camel with ActiveMQ. ActiveMQ also can embed Apache Camel. So, you have routes running inside the broker from Camel. The two of them are very complementary.

Gardner: Let's step back for a wider perspective. I'm seeing that the need for integration is increasing. The things that need to be integrated are increasing, perhaps exponentially. The pace at which that needs to take place is very rapid and dramatic, compared to the history of computing. Open source is well established. We’ve seen many different organizations embracing this. We saw Red Hat come out recently with some very strong growth figures.

It's coming to the point where organizations won’t have a choice other than to use open source as a way to try to keep up with a pace of change.

So, it seems to me that open source is a very mature approach now, not something that’s a new kid on the block, by any stretch. When you put these factors together and when you look at the need for integration as a service within applications from the start -- not something you bolt on or think about after the fact but actually build applications for, of, and by integration capabilities -- this perhaps spells a historic shift.

Maybe we could riff on the future or even look at this from an abstract or even philosophical perspective. Rob, are we at a shift here where the ability to integrate becomes an essential character of businesses?

Davies: We probably are at that shift right now. Sometimes, it's difficult to see things happening like that, if you’re actually right inside in the middle of it. But, if you look at the way the environments change, you’ve got to actually be running your compute resources.

We’ve talked about cloud environment. Also there’s social network, SaaS, and mobile devices, and you need to link all those together. It's coming to the point where organizations won’t have a choice other than to use open source as a way to try to keep up with a pace of change.

We're probably at a point now, where we’re going to see that the traditional model of providing software is going to dwindle over time, probably pretty rapidly as well, as organizations realize that they need the flexibility and the ability to change what they’re doing very quickly.

Future-proofing applications

It's a really good point that you made. You have to start thinking about how you're going to future-proof your applications right from the beginning to adapt to changes in their environments. You have to architect in how you’re going to integrate and future-proof your applications, because it does get more costly if you do it as an afterthought.

Gardner: Many of the SaaS providers are doing multitenancy and providing applications as services on demand at a very attractive and aggressive price point. They're leveraging open source on the back end, I have to imagine. Do you have any insight into what the service providers themselves are building with?

Davies: Most applications now, in particular on the cloud, are using open source at the back end. We can't give you any specific details of vendors that are doing that, but I know they're using open-source projects, and not just the SaaS vendors, but some of the other existing product vendors use open-source as well to enable their products.

We certainly see open-source as definitely mainstream now, and we’ve seen it has been the first choice that people use for building any kind of application or service they’re providing. It's more a case of people asking the questions now of not should we be using open source, but why shouldn’t we use open source? It's starting to become a first choice for people to go to.

Gardner: Let's look at some of the ways in which those people are making that choice. Debbie, you mentioned the FAA. Are there other organizations that you can point to and say, either by name or by use case scenario, that they’ve taken this leap, they’ve made those choices, they’ve embraced some of the new requirements around integration, and they have some positive proof points? Any examples that we can look to?

It’s more dynamic and flexible, and being involved with the community is really exciting.

Moynihan: Sure. Sabre is one of our customers. Sabre Holdings is using the FuseSource open-source software, and they started using open-source software many years ago. Years ago, a lot of people chose it because they were looking at cost and flexibility. Now, they're seeing that you actually were getting more features faster in open source than you were getting in traditional software. It’s more dynamic and flexible, and being involved with the community is really exciting.

One of the things that Sabre has done with open-source software is a travel gateway. They connect to many different airline technologies and travel agencies and they have over one-and-a-half billion transactions running through their infrastructure on any given day. They've been using FuseSource open-source software for over a couple of years with zero downtime.

Being able to use open-source, they have that flexibility, have the interaction with the community, and also have high-performance and reliability.

People are getting all of the traditional benefits of high-quality software, but also that dynamic ability to get new features, to get new technologies, to get bugs fixed, for example, really quickly with the community. With support vendors like FuseSource providing subscription so that they can have access to the engineers directly who are working on these projects, they can quickly get turnaround and get what they need to make those dynamic changes in their business.

Retail industry

Another area that we are seeing a lot of people look at open source is in the retail industry. Earlier on, people were looking for cost savings. If you think about retail, it’s common for retailers to have a lot of locations, whether it’s franchises or stores. Using open-source, you can save a lot of cost on your IT footprint in those locations.

Specsavers is one of our customers. They're deploying open-source to over 1,000 retail stores. We're seeing more and more retailers looking at open-source to be able to do that. They're going to get all the flexibility of being able to incorporate these new technologies as we incorporate them into the open-source projects really quickly. But, right from the get-go, they have reduced costs, flexibility and the involvement within the open-source community and directly with the development teams through working with vendors like FuseSource that support the open-source communities.

Gardner: For folks who are looking to ramp up their adoption of open-source integration, are there some resources that they should be aware of in terms of getting started?

Moynihan: I would encourage people, specifically if they are looking at open-source integration and messaging, Apache Camel is a good place to get started. We have a productized distribution at fusesource.com as well.

I would encourage people, specifically if they are looking at open-source integration and messaging, Apache Camel is a good place to get started.

The reason why I suggest Apache Camel is that it's is based on the enterprise integration patterns book and provides a nice foundation and definition around some of the most commonly used integration patterns. It’s a great way to get started.

People could obviously come to CamelOne, which is going to be really exciting, meet a lot of the people who are experts in the community, and meet other users of open-source integration messaging software.

Also on our website fusesource.com, we have a lot of webinars, which are happening live on a regular basis. We have a lot of archived webinars, which actually walk you through technical tutorials on how to get started with these various open-source projects.

So I’d highly recommend to check that out and to check out the books that we've mentioned and the documentation on our site as well.

Gardner: Very good. We have been talking about how enterprise integration requirements are rapidly shifting in order to accommodate such general and global trends as cloud computing, mobile device use explosion, and the increase demand for extended enterprise business processes.

I want to thank our guests, Rob Davies, Chief Technology Officer at FuseSource. Thanks a lot, Rob.

Davies: Thank you very much, Dana.

Gardner: And Debbie Moynihan, Vice President of Marketing at FuseSource. Thanks, Debbie.

Moynihan: Thank you, Dana.

Gardner: You've been listening to a sponsored BriefingsDirect podcast. This is Dana Gardner, Principal Analyst at Interarbor Solutions. Thanks for listening, and come back next time.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Register for CamelOne. Sponsor: FuseSource.

Transcript of a sponsored BriefingsDirect podcast on the maturing of open source integration software and its role in making enterprises responsive to a rapidly changing IT landscape. Copyright Interarbor Solutions, LLC, 2005-2011. All rights reserved.

You may also be interested in:

• FuseSource Gains New Autonomy to Focus on OSS Infrastructure Model, Apache Community Innovation, Cloud Opportunities
• Apache Camel Addresses Need for Discrete Infrastructure for Services Mediation and Routing
• Apache CXF: Where it's Been and What the Future Holds for Web Services Frameworks
• Workday Integration Cloud debut raises bar on integration as service, deeply ingrains integration as apps function