Security Skills Offer Top Draw Across Still Challenging U.S. IT Jobs Outlook

Edited transcript of BriefingsDirect Analyst Insights Edition podcast, Volume 48, on where to find opportunities in IT during this stage of the recession.

Listen to the podcast. Download the transcript. Find it on iTunes/iPod and Podcast.com. Learn more. Charter Sponsor: Active Endpoints. Also sponsored by TIBCO Software.

Special offer: Download a free, supported 30-day trial of Active Endpoint's ActiveVOS at www.activevos.com/insight.

Gain additional data and analysis from Foote Partners on the IT jobs market.

Dana Gardner: Hello, and welcome to the latest BriefingsDirect Analyst Insights Edition, Volume 48. I'm your host and moderator Dana Gardner, principal analyst at Interarbor Solutions.

This periodic discussion and dissection of IT infrastructure related news and events, with a panel of industry analysts and guests, comes to you with the help of our charter sponsor, Active Endpoints, maker of the ActiveVOS business process management system, and through the support of TIBCO Software.

Our topic this week on Analyst Insight Edition centers on the IT job landscape for 2010. We have with us here today David Foote, CEO and chief research officer, as well as co-founder, at Foote Partners LLC of Vero Beach, Fla.

David closely tracks the hiring and human resources trends across the IT landscape. He'll share his findings of where the recession has taken IT hiring and where the recovery will shape up. We'll also look at what skills are going to be in demand and which ones are not. David will help those in IT, or those seeking to enter IT, identify where the new job opportunities lie.

So, please join me in welcoming David Foote to our show. Hello, David.

David Foote: Great to be here, Dana.

Gardner: First, tell us a little bit about yourself, your background, and then a bit more about the Foote Partners.

Foote: I started my career in technology in Silicon Valley, just about the time the IBM PC came to market. About half of the 10 years I spent there were working with vendors. I started in consumer electronics, in the games business.

Then, I gravitated to consulting, particularly with extremely rapid-growth technology companies, those that were making their first billion within four years. That led to becoming an analyst, and I started at Gartner in the '90s.

I was recruited to META Group to start and run their service for CIOs. While I was there, the discovery I made was that there was so much the analyst community was offering executives in IT and in business around vendors and products, and not much on the execution side.

Once we've made all of our purchases, how do we actually execute and get this stuff done? That's a spending issue, a people issue, a workforce issue, and, these days, very much a skills issue, and there really wasn't much being done.

Business focus

So, I developed a number of products and took them out to my own company in 1997. I co-founded this company with a former senior partner at McKinsey. We not only have that big IT executive and trends focus as analysts, but also very much a business focus.

We've also populated this company with people from the HR industry, because one of the products we are best known for is the tracking of pay and demand for IT salaries and skills. It's been a wild ride over 13 years, but that's pretty much it. We are in the execution business.

The last thing I'll say is that we have a proprietary database -- which I'll be drawing from today -- of about 2,000 companies in the U.S. and Canada. It covers about 95,000 IT workers. We use this base to monitor trends and to collect information about compensation and attitudes and what executives are thinking about as they manage IT departments.

Gardner: Before we get into where we're going in 2010, maybe we should pause and take a look at what's happened for the last three years. We know the bigger story about the recession, but what has happened in IT hiring? What have the last three years and, more specifically, 2009 been like?

Foote: One of the biggest trends has been in defining who an IT person is. For many years, IT people were basically people with deep technical skills in a lot of areas of infrastructure, systems, network, and communications. Then, the Internet happened.

All of a sudden, huge chunks of the budget in IT moved into lines of business. That opened the door for a lot of IT talent that wasn't simply defined as technical, but also customer facing and with knowledge of the business, the industry, and solutions. We've been seeing a maturation of that all along.

What's happened in the last three years is that, when we talk about workforce issues and trends, the currency in IT is much more skills versus jobs, and part of what's inched that along has been outsourcing.

We have labor arbitrage now. If you need to get something done, you can certainly purchase that and hire people full-time or you can rent it by going anywhere in the world, Vietnam, Southeast Asia, India, or many other places. Essentially, you are just purchasing a market basket of skills. Or, these days, you can give it over to somebody, and by that I mean managed services, which is the new form of what has been traditionally called outsourcing.

In execution, you don't have to hire bodies. It's not so much about hiring, but about how we determine what skills we need, how we find those, and how we execute. What's really happened in two or three years is that the speed at which decisions are made and then implemented has gotten to the point where you have to make decisions in a matter of days and weeks, and not months. To move an organization, or shuffle resources around, which in this case are people and skills, is tough to do in these very, very short time frames.

So, agility, flexibility, and speed of execution in a global market have been extremely important. In the recession, the idea of being constrained by costs has really tested the mettle of a lot of managers -- how good they are as leaders and managers. Clearly, we have seen that a lot of them are not, but a lot of them are.

Resisting the temptation

There have been some interesting behaviors during this recession that I haven't seen in prior recessions. That lead me to believe that people have really resisted the temptation to reduce cost at the expense of what the organization will look like in 2011 or 2012, when we are past this recession and are back into business as usual.

Do you have the people and the skills necessary for the long-term or were you stupid enough to let people go, simply because they weren't the right person at the right time, in the right spot, and were vulnerable? People have learned something. That's been a big difference in the last three years.

Gardner: So, clearly this is a complex issue. Shuffling people around and maintaining skills, while also recognizing there is a pressure for cost, but also wanting to be mindful of what the recovery is going to look like, has all got to be done carefully. But, generally, did IT, as a sector, fare better than other sectors in this recession? Was it as bad as it was perhaps back in 2001-2002?

Foote: IT has always done better. You have to realize that unemployment in IT is usually half of what it is in the general job market, if you look at Bureau of Labor Statistics (BLS) numbers. I can tell you right now that jobs, in terms of unemployment in IT, have really stabilized.

We put out a report recently. Every month we look at the BLS numbers, and in this most recent report we saw that if you look at the five bellwether segments in the BLS where you would find IT jobs, in the last three months there has been a net gain of 11,200 jobs in these five categories. If you look at the previous eight months, prior to September, there was a loss of 31,000 jobs.

There has been a real stability, and in the case of the services industry in particular, a BLS segment called Management and Technical Consulting Services has gained 11,600 jobs in the first 11 months of 2009, including gains for the last five months.

Looking for talent

So, let's just say that, going into 2010, the services industry will absolutely be looking for talent. There's going to be probably a greater need for consultants, and companies looking for help in a lot of the execution. That's because there are still a lot of hiring restrictions out there right now. Companies simply cannot go to the market to find bodies, even if they wanted to.

Gardner: So, 2010 is already a recovery period for IT. IT's not as bad as other sectors in this recession, but clearly this is spotty. This isn't across the board that we're seeing these recoveries already. What are the hot areas that you can see so far for hiring picking up in IT?

Foote: Well, number one, overall in 2010, we're not going to see any kind of, what I would call, a meaningful return to employment in IT. It's going to be a very slim-to-no chance of any noticeable hiring, particularly if you look at the way things were in 2008. I don't see that happening in 2010. At the earliest, 2011.

There are certain areas, for example, like security, where there is a tendency to not want to hire talent outside, because this is too important to a company.

Companies are still very nervous about hiring, or to put it this way, investing in full-time talent, when the overhead on a full-time worker is usually 80-100 percent of their salaries. If they can find that talent somewhere else, they are going to hire it.

There are certain areas, for example, like security, where there is a tendency to not want to hire talent outside, because this is too important to a company. There are certain legacy skills that are important, but in terms of things like security, a lot of the managed services that have been purchased in 2009 were small- to medium-sized companies that simply don't have big IT staffs.

If you have 5,000, 6,000, or 7,000 people working in IT, you're probably going to do a lot of your own security, but small and medium size have not, and that's an extremely hot area right now to be working in.

Generally speaking, Dana, you're not going to see much happening in hiring in 2010, except maybe certain industries or niche skills, which I will talk to today. Security is one. Enterprise resource planning (ERP), particularly around SAP is another. We're looking at eCommerce, whole market basket of eCommerce jobs and skills, and web development. We can talk about a few of those as we go on today. There are pockets, but overall, don't expect much.

Gardner: Right. So, as with the general economy, doing more with less will continue for some time. I think people have a sense that productivity can be squeezed out in a number of ways, and just because the recession technically might be over, doesn't mean that they are going to stop looking for that productivity.

I guess the question then moves to, if you are an IT person, and you already have a job, what skills should you be investing in? Where should you be taking your own skills in order to make yourself more valuable, perhaps to your current employer, or perhaps even to ensure your ongoing employment with a particular company? Where are the hot skills?

Security gains value

Foote: We track the value of skills and premium pay for skills, and the only segment of IT that has actually gained value, since the recession started in 2007, is security, and it has been progressive. We haven't seen a downturn in its value in one quarter.

Since 2007, when this recession started, overall the market value of security certs is up 3 percent. But if you look at all 200 certified skills that we track in this survey that we do of 406 skills, overall skills have dropped about 6.5 percent in value, but security certifications are up 2.9.

It is a tremendous place to be right now. We've asked people exactly what skills they're hiring, and they have given us this list: forensics, identity and access management, intrusion detection and prevention systems, disk file-level encryption solutions, including removable media, data leakage prevention, biometrics, web content filters, VoIP security, some application security, particularly in small to medium sized companies (SMBs), and governance, compliance, and audit of course.

Special offer: Download a free, supported 30-day trial of Active Endpoint's ActiveVOS at www.activevos.com/insight.

Gain additional data and analysis from Foote Partners on the IT jobs market.

Most recently, we've seen some nice numbers go up in the certifications related to auditing. You tend to see that towards the end of the year and the beginning of the year when a lot of this auditing happens. There are few more that I could mention, but these are areas in security for which companies have said they are absolutely looking for talent and skills.

Oh yes. The public sector has been on a real tear.

Gardner: Does this cut across both public and private sectors, do you break that out?

Foote: Oh yes. The public sector has been on a real tear. As you do, we get a lot of privileged information. One of the things that we have heard from a number of sources, I can't tell you the reason why, is that a lot of recruiting is happening in the private sector right now with the National Security Agency and Homeland Security -- in-the-trenches people.

I think there was a feeling that there weren't enough real deep technical, in-the-trenches kind of talent, in security. There were a lot of policy people, but not enough actual talent. Because of the Cyber Security Initiative, particularly under the current administration, there has been a lot of hiring.

Gardner: And that of course reduces the pool available to the private sector?

Smart place to be

Foote: Well, you could say that, but I believe that there are more people that have gotten religion and listen to people like us. We've actually gone out on a limb and we have been saying for about a year and a half that, bar none, for short and long range IT job security, the smartest place to be is, ironically, in IT security.

Unlike a lot of other job segments, pay and demand has risen steadily for the last two years. Neither budget nor headcount has diminished during the recession. There is this perfect storm of drivers right now that is driving momentum, and that is the idea of more regulation, constant fear of increasing threats, greater customer expectations and demands aimed at vendors who have responded. If you look at the Oracles and the Ciscos, they have built a lot of security into their products now.

The other thing, which isn't spoken about a lot, has been the splitting of the operational securities activity side from the business and strategic risk side. That has created a lot of job opportunities in security for people who don't come at it from this deep, in-the-trench technology focus. You can now work in a line of business or in a corporate staff without these deep technical skills, and work in governance and risk management side of that. It has attracted people into the field who didn't think they had the chops to really do it.

As more people come into the field, supply starts to match demand, and it has been such that there has been a pretty steady increase in pay, because demand has really exceeded supply in a lot of cases.

So, Dana, what you've seen is a lot of people who were network administrators and systems administrators now choosing security as a career. As more people come into the field, supply starts to match demand, and it has been such that there has been a pretty steady increase in pay, because demand has really exceeded supply in a lot of cases.

Gardner: Now, another big issue that we have heard about in the last year is cloud computing. And I see from your list of where hiring has taken place the strongest and SAP and ERP, but also open-source operating systems, virtualization, IP networks, social media, PHP, and what is called web-oriented architecture (WOA). Is there a building sense that the cloud hysteria is not just vapor, but that there is now starting to be a call for these sorts of jobs that might be driven by cloud-computing activity?

Foote: Let's be careful about what we call cloud. There's a lot of disagreement as to what cloud computing actually means. Let's call it managed services for a second. That market is projected to be a $66 billion market by 2012. Most people estimate the global managed services market at between $22 billion and $31-32 billion right now.

You're seeing compound annual growth rates in various segments of managed services, particularly in networking of between 19 and 39 percent, between now and 2012-2013. There absolutely is this x-factor. If people choose managed services, and many of them are right now, more will companies need to hire many full-time workers.

We're saying that you'll see increase in headcount in IT, because as companies farm out some of the work that they don't want to do and purchase services in those areas, that allows them to reshuffle their workforce and put people into a lot of other areas, which don't get a lot of mention, except from companies like ours.

Business technology work

And, that is the number of IT people you see in product development groups, marketing, finance, accounting. There are so many IT people now that don't work in IT departments under a CIO, but work in the lines of businesses, doing all kinds of business technology work.

They are in product development groups right now, because most companies have gotten smart and said, "Security is a product feature now, because our customers tell us it is. We need to have IT people involved from the very beginning as we are conceiving our products, which we'll be delivering over the web and supporting and maintaining, and all of that stuff."

I don't want to simplify it, but the definition of an IT worker has really changed quite a bit over the years. When I visit companies, I can't really tell if a person is “IT" or not, because they are in the fabric of companies.

Plus, there is all that market intelligence that can be gained. The web has become central to business globally.

That's one of the reasons right now that in 2010 we'll see a lot of activity around eCommerce and web development, in places you wouldn't exactly expect. It's cost effective to deliver products over the World Wide Web, and companies have simply now defined that as a launching point or as a delivering mechanism for a lot of what they do.

Plus, there is all that market intelligence that can be gained. The web has become central to business globally.

Gardner: That’s probably going to become probably more the case over the next several years, right?

Foote: At some point, where we have an IT department, or where we just have a bunch of consultants, analysts, and expeditors, a lot of the work traditionally done will be farmed out -- either rented, as I said before, or given over -- but that doesn’t diminish the number of people working in IT. It just changes what they do on a day-to-day basis.

Gardner: If service management is a hot area, does that affect the regions where hiring would be taking place? You look at the U.S. and Canada. Are there any geographic or regional trends afoot that we should look at vis-à-vis security, the recession, or even this emphasis on managed services providing?

Foote: I don’t know if we have enough time in this forum to go much outside of North America, but I would say, yes, of course, it’s global.

Gardner: And global, but it's not in terms of regions?

Foote: You mean around the world?

Gardner: Let’s just stick with North America for now.

Managed services

Foote: I talked about managed services before. It looks like one of the hottest areas right now in managed services is in networking and communication: Metro Ethernet, VPNs, IP voice, and wireless security. And if you look at the wireless security market right now, it’s a $9 billion market in Europe. It’s a $5.7 billion market in Asia-Pacific. But in North America it’s between $4 and 5 billion.

There's a lot of activity in wireless security. We have to go right down into every one of these segments. I could give you an idea of where the growth is spurting right now. North America is not leading a lot of this. Other parts of the world are leading this, which gives our companies opportunities to play in those markets as well.

For many years, as you know, Dana, it was everybody taking on America, but now America is taking on the rest of the world. They're looking at opportunities abroad, and that’s had a bigger impact on labor as well. If you're building products and forming alliances and partnerships with companies abroad, you're using their talent and you're using your talent in their countries. There is this global labor arbitrage, global workforce, that companies have right now, and not just the North American workforce.

For many years . . . we were reminding companies that the way that you need to run yourself is to be flexible, to have an extremely agile workforce.

Is it appropriate to always think of a home labor force? No, it’s appropriate to think of your labor force all around the world. And it’s cost-effective too. Look at the number of people that IBM has displaced in Europe and North America and moved those jobs over to places like India. That’s our world now.

Gardner: So, just as the idea about cloud computing is that you go for resources when you need them and acquire them the best way that you can under the particular circumstances, you think there is a parallel from cloud computing into hiring. You don’t necessarily hire people. You acquire skills. You pick them up and then you drop them as you need them. Is there a strong parallel there? It’s like a human resources cloud of skills that you can have elasticity with?

Foote: For many years, to anyone who would listen, we were reminding companies that the way that you need to run yourself is to be flexible, to have an extremely agile workforce. Companies had such trouble with that idea, because they were so used to the idea of hiring people, keeping them around, reskilling them, and carrying so much inappropriate skills. The idea is that, at any one moment, six months to six months to six months to six months, the mix of skills you need changes.

The idea they finally learned is that it’s better to mix-and-match your skills. Hire people up front who you know are flexible enough in terms of the market basket of skills, the multidimensionality they bring into the job, when they start, knowing that you are going to be moving that person around in the company. You're going to be tweaking their skills from here to there.

Not cutting it

The idea of a specialist who came in -- this is our Microsoft guy, this is our Lotus guy, this is our Novell guy or girl -- ain't cutting it. Companies are looking for Swiss Army knives quite a bit, when they hire people. If they need deep specialists, often they will go to the market, independent consultants, or contractors to get a lot of that deep skill. Then, when it’s over, bye, bye.

Gardner: To this issue of recruitment, traditional recruitment methods seem to be giving way to more social-media or community-generated word of mouth, but in a digital format sort of activity. Can you help us, David, understand a little better about what’s shifting in terms of how companies are recruiting these skills?

Foote: Right now, about 80 percent of employers are using or tell us that they are using or plan to use social media to fill vacancies. Ninety five percent of employers that we talked to use LinkedIn in one way or another. On the recruiting side, about 60 percent of recruiters that we talked to use Facebook.

Right now, about 80 percent of employers are using or tell us that they are using or plan to use social media to fill vacancies.

Now, that was 36 percent in 2008 and 59 percent in 2009. Forty two percent of recruiters use Twitter. You're looking at a situation now where companies have finally decided to do this, after many years of being unwilling to change the traditional method of recruiting. That was to use body shops and headhunters, collect CVs and resumes, apply the software that searches through them with filters. You had a lot of people who are incredibly good at what they do, but who don’t have certification get knocked out.

They have now said, "If you are an employee, let’s talk to those people in your audience." You have got a brand, and your brand is you. It’s a combination of technical skills, your personality, and other skills that you have. What is it that companies are hiring when they hire you, beyond simply what you see on paper in terms of skills? What have you done? What solutions have you been a part of? If you worked in SAP, how many cycles have you gone through? Have you been through two or three full development cycles?

Using social media sites, you're going right upfront answering the kind of questions that people ask you in a job interview. You're networking in places like LinkedIn, where you can actually get people to recommend you. Recruiters are looking at those recommendations that are given by executives and by colleagues. They are right there. You see it. It’s an online picture of you and your brand.

Why wouldn’t you use social media, if you look at the kind of numbers I just gave you? The recruiters and employers, they are all using it. You should be smart as a worker in IT to connect with them there.

Going back to security for a second, it’s unbelievable to me the number of boutique, niche security consulting firms that I have been talking to over the last two and three years who say that the demand for their services is greater than their ability to meet that demand. They are in a perpetual situation of not having enough consultants on board to do the work that comes their way.

Now that’s unbelievable. Why can't they connect to the people? There are reasons for that. If you go to RSA conferences or SAP TechEd conferences, you meet all these people who say, "I have been looking for a job for months and months and months and I just can’t find one." I say to them, "Well, exactly how are you looking for that job? Are you on LinkedIn? Are you using Facebook? Are you using Twitter? There are people out there that want to find you. They can’t find you. You have to find them and make yourself available to them." Basically, it’s old fashion networking, web-enabled, that's what social networking is.

Strut your stuff

Gardner: Perhaps there's never been a better time, if you have good stuff to strut, to strut it in these forums. And, there's never been a better time, if you're recruiting, to use these tools to find those folks that you need to fit that exact need that you have.

Foote: Yes. And those areas right now seem to be in security, SAP, web development, eCommerce applications and systems, business intelligence (BI). There are number of areas that we have identified in 2010 that will be hot.

You can go to our website. We put out a hot list where we look at all the data we have: pay and demand data and what 2,000 companies are telling us they are looking for in the market. We put all that into a list of 24 certifications and 32 non-certified skills, and we rank them 1, 2, 3, 4, 5, and you can see that list.

If you are anywhere in the top 15 or 20, get out there, pound the pavement, and be smart.

It's updated every three months as to what the hot skills are. These are skills people are looking for. People should get ahold of that hot list and map to it. Where are you on that list? If you are anywhere in the top 15 or 20, get out there, pound the pavement, and be smart.

Gardner: What about the not-so-hot list? Are there certain IT niches or skill sets that just seem to be drying up, where there is really no demand or very little growth?

Foote: Well, always. Yes. Do you want to talk about those?

Gardner: Yeah. What's the new not-hot list?

Foote: I haven't looked at that list lately, but we do get calls, about every two years, from an enterprising editor or a reporter, who wants to publish the dead or dying list. We say to them, "This is going to kill your career as a writer, if you do this, because anybody who is on that list, and that's their specialty, is going to come after you." It has happened where reputations have been badly sullied. You can Google "Foote Partners' dead and dying skills" and you can find us, because we contribute to every one of these lists for good or bad.

Gardner: At the risk of being overly reality-oriented, let's take that plunge anyway and come up with a few areas that just don’t seem to be growing any more.

Foote: There are a number of networking skills that have fallen out of favor over the years. Things like Smalltalk. I would say probably the C language, not C++ and C#, but I would say the C language probably has taken hits. There have been a number of older web development, web authoring tools that are no longer on that list.

I wouldn't put Gigabit Ethernet on that in general, but I think there are some people that will tell you that certain old brands of Gigabit Ethernet, like 1 Gig versus 10, would be falling more out of favor. The OS/2 from IBM, is that still hanging around? It's probably dead.

There are so many dead skills that companies have just abandoned them as platforms and products. There are still people around that might have those skills. So yeah, there are plenty of dead skills.

What's hot

Gardner: Back to the more lively news. What certification areas are hot? You mentioned the idea of being a Swiss Army knife as a person, the walking in with multiple skills. I would think that having multiple certifications would be a very strong thing to have, make you more valuable and more interesting. What sort of certifications should people be looking at?

Foote: Well, the certifications that are on our current hot list that are doing quite well -- in fact, the first and sixth positions -- are Red Hat certification engineers, open systems, Linux. Twelve of 24 on this list of hot certifications are security.

In order, the GIAC Certified Incident Handler is at the number three position. The ISC-squared Systems Security Certified Practitioner is at number four. Check Point Certified Security Administrator is at number eight. The CISA, the gold standard auditing certification from ISACA is number 11. And then at positions 14, 15, 16, Check Point Security Expert, the Forensics Analyst, and Intrusion Analyst from SANS Institute.

One not related to security is Cisco Certified Design Expert, which is the number five position. That is an excellent advanced-level certification.

One not related to security is Cisco Certified Design Expert, which is the number five position. That is an excellent advanced-level certification. Again, it's always on our list, somewhere in the top ten positions most of the time.

What's interesting this time is that the SAS-Certified Advanced Programmer is there. Before, I mentioned BI, and now we see SAS coming in at number seven on that, so good news for SAS. BI and business analytics is on our hot list, as I mentioned before, as strong skills for 2010.

Even the Java Programmer Certification from Sun. Java is also number eight out of 32 on our non-certified skills hot list too. As I look at eCommerce and web development skills, we're looking at things like the Microsoft Commerce Server, Microsoft SharePoint, C++, SOAP, Python, Perl, a number of those. The certifications, I think, are mainly in the areas of open systems, security, and believe it or not, things like WebSphere.

Some of the service-oriented architecture (SOA) skills are back. There has been sort of a resurgence of not these big grand, huge SOA projects, but point solutions. Small-chunk SOA has made a comeback this year. We see that both in the number 12 ranked certification, the Certified Solutions Developer - WebSphere from IBM. And then you see the BEA SOA Enterprise Architecture Certification. In that regard, what I didn't mention before is, architecture continues to be a good skill area for 2010.

What's interesting about architecture is its one of those skills, as we've said to people, is a long run kind of career opportunity for you. There will always be a great need for architects, and most companies will tell us that. They have been flat lining as far as pay in those certifications in 2009 because of cost containment.

You don’t look at architect skills and certifications as short-term investments, but more sort of long-term, core competency skills for organizations. So, the focus has gone off of architecture a little bit.

Need for architects

On the other side of this recession, I think you'll see what we saw in 2007 and 2008, which is a lot of companies looking for architects. They're continually telling us, "We don’t have enough of these architects around here. We need more of them." I mean, enterprise architects, data architects, security architects, SAP architects, and information architects, a whole lot of them.

I'd like to see that come back in 2010. I always put architecture on the list, but it comes down to how much money people have, and how short range a lot of their decisions are.

Gardner: To just touch briefly on the notion of having a cross-pollenization of skills. In the past, someone with an engineering degree and an MBA would have done well in certain sectors. Are there any two areas that we might not think intuitively would be related, but that stack up really well in 2010, if you wanted to combine skills in a way that would give you an advantage in the market?

When you look at IT hiring in the last two or three years, you see that they want IT people working in their marketing departments, their human resource departments, their finance and accounting departments, operations, and logistics.

Foote: Well, I have been saying for a while, and also for 2010, is if you are in college, you want a minor in computer science and a major in something like mathematics, marketing, finance, or accounting. When you look at IT hiring in the last two or three years, you see that they want IT people working in their marketing departments, their human resource departments, their finance and accounting departments, operations, and logistics.

They want people with that kind of background to understand what that part of a business does. Then, they want that person to help them to think through how technology can either have them do it quicker and more efficiently to raise the bar on their ability to deliver services internally in those areas.

I have often said that you can easily have a major in any of the functional areas of business that I just mentioned and have some sort of knowledge that you have gained, either through a degree or by personal experience in IT, and you would be a very strong candidate for a lot of companies.

These are those jobs that don’t fall under the corporate CIO, but under the lines of business, where you are an IT person working in a line of business, usually under a CTO or some managing director level person in IT. But, you're working in a P&L part of the business. Your job is to figure out how to make money, how to increase profitability, and how to improve customer satisfaction. There are so many ways that IT has been able to do that. As I pointed out before, you're also now involved in product development in these areas.

Gardner: That's very interesting, because in the past, let's say, 10 or 20 years, the conventional wisdom was that you wanted to have a general business education background, but then specialized in IT. I think what I am hearing from you is that you want to have a jack-of-all-trades, general sense of IT skills and maybe some certifications, but then specialize in a particular business function. Is that what's happened, a sort of a flip?

Multiple areas

Foote: Yeah. Right now, you can hear this by talking to anybody. We talk to a lot of people who hire and we ask them exactly what they need and what they're looking for. And they will say, "I need somebody who understands our business and our customers, who really understands this industry and the products that we supply in this industry. And, oh, by the way, they should have a good IT background too." It's the third or fourth thing they may mention.

Now, of course, if it's in security or if it's in routers or something like that, they may be looking for a deep certified person in that infrastructure specialty. This is particularly true in areas like SAP and ERP, and in Oracle. Look at what SAP has done in its product strategy. They have really gotten down to industry and function.

You're really looking at being a specialist in finance and accounting systems, sales distribution or logistics, or even global travel management, and you become an expert in that area using SAP products. So, you're an IT person, but you are specializing in a domain that is usually functional or industry.

That's what a specialist is in IT these days. They have a technical specialty and they have a business specialty . . .

That was a smart move for SAP, and a lot of companies have taken to that sort of segmentation of the market. That's what a specialist is in IT these days. They have a technical specialty and they have a business specialty, and it's probably in a function within that business that is supported and enabled by technology.

Gardner: Well, just to recap a little bit, we have said that the recession has certainly hurt the IT sector. There are fewer jobs, not as severe in cuts in terms of some others, perhaps half the general unemployment rate in IT. Things have leveled out. We're seeing some rehiring in specific areas, late in 2009, going into 2010.

You expect 2010 to be fairly flat, not a huge spike in demand, but in specialized areas that we have talked about, security being prominent among them. There's going to be growing demand. There's going to be this opportunity to position yourself, if you have that right balance of technology, skills, and business acumen.

But, you expect things to be picking up later in 2010, perhaps 2011, where we might see a bit more of that significant growth. And, the hot skills and being able to present yourself through a social network makes a lot of sense. Did we miss anything?

Foote: No, but the projection of late 2010 or 2011, that's for hiring. There is still a tremendous amount of investment and money moving around in 2010, but the focus is more on skills than it is on hiring full-time workers. That creates a lot of opportunities for contractors and independent consultants, as I pointed out before, which is supported by the BLS numbers.

The IT services business, the systems integrators (SIs), the small boutique consulting firms, are going to get a lot of play in 2010, because there won’t be a lot of full-time workers added to companies in that period.

They're still in a time where they are going to have a lot of work to do and they've got to get it done. They won’t have nearly the restrictions in renting or giving over some of this activity to managed services, but they still will have certain restrictions in full-time hiring, and that full-time hiring will be in niche areas and in the four, five, or six specializations I have mentioned.

Gardner: So, perhaps it's a good time, if you've got the right skills, to hang your shingle up, go into your own boutique consultancy, maybe partner up with a few other folks that are in complimentary skill sets, and then promote yourself through a social network. This will be a good time to do something like that.

Foote: Yeah. But, as a person who has actually hung a shingle a lot and competes against those big firms, you have to have a lot of intestinal fortitude if you are going to do that. I'd recommend that the best thing you could do is find old pals who you used to work with -- even if it was five, six, seven, eight years ago -- and network with them. You might find that they're interested in bringing you on, because I do think there is going to be much more activity in 2010.

And then, in 2011, we may see a return to some full-time hiring. Then, you can choose whether you want to go back into working for a big company or a small company, or stay as a consultant. Again, Dana, consulting and that kind of activity is not for everybody’s personality.

Gardner: Excellent. We've had a deep penetrating and interesting discussion about IT hiring and trends. I certainly want to thank our guest David Foote, CEO and chief research officer, as well as co-founder at Foote Partners LLC, in Vero Beach, Fla. Thanks so much, David.

Foote: You bet, Dana.

Gardner: I also want to thank our sponsors for this BriefingsDirect Analyst Insights Edition podcast, and that would be Active Endpoints and TIBCO Software.

This is Dana Gardner, principal analyst at Interarbor Solutions. Thanks for listening, and come back next time.

Listen to the podcast. Download the transcript. Find it on iTunes/iPod and Podcast.com. Learn more. Charter Sponsor: Active Endpoints. Also sponsored by TIBCO Software.

Special offer: Download a free, supported 30-day trial of Active Endpoint's ActiveVOS at www.activevos.com/insight.

Gain additional data and analysis from Foote Partners on the IT jobs market.

Edited transcript of BriefingsDirect Analyst Insights Edition podcast, Volume 48, on where to find opportunities in IT during this stage of the recession. Copyright Interarbor Solutions, LLC, 2005-2010. All rights reserved.

You may also like:

• BriefingsDirect analysts unpack the psychology of project management via 'Pragmatic Enterprise 2.0' and SOA
  
  
• BriefingDirect analysts discuss business commerce clouds: wave of the future or old wine in a new bottle?
  
  
• Analysts define growing requirements list for governance in any move to cloud computing

Technical and Economic Incentives Mount Around Seeking Alternatives to Mainframe Applications

Transcript of the third in a series of sponsored BriefingsDirect podcasts on the rationale and strategies for application transformation.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Learn more. Sponsor: Hewlett-Packard.

---

Gain more insights into "Application Transformation: Getting to the Bottom Line" via a series of HP virtual conferences. For more on Application Transformation, and to get real time answers to your questions, register to access the virtual conferences for your region:

Access the Asia Pacific event.
Access the EMEA event.
Access the Americas event.

---

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you’re listening to BriefingsDirect.

Today, we present a sponsored podcast discussion on why it's time to exploit alternatives to mainframe computing applications and systems. As enterprises seek to cut their total IT costs, they need to examine alternatives to hard to change and manage legacy systems. There are a growing number of technical and economic incentives for modernizing and transforming applications and the data center infrastructure that support them.

Today, we'll examine some case studies that demonstrate how costs can be cut significantly, while productivity and agility are boosted by replacing aging systems with newer, more efficient standards-based architectures.

This podcast is the third and final episode in a series that examines "Application Transformation: Getting to the Bottom Line." The podcast, incidentally, runs in conjunction with a series of Hewlett-Packard (HP) webinars and virtual conferences on the same subject.

Access the Asia Pacific event. Access the EMEA event. Access the Americas event.

Here with us now to examine alternatives to mainframe computing, is John Pickett, Worldwide Mainframe Modernization Program manager at HP. Hello, John.

John Pickett: Hey, Dana. How are you?

Gardner: Good, thanks. We're also joined by Les Wilson, America's Mainframe Modernization director at HP. Welcome to the show, Les.

Les Wilson: Thank you very much, Dana. Hello.

Gardner: And, we're also joined by Paul Evans, Worldwide Marketing Lead on Applications Transformation at HP. Welcome back, Paul.

Paul Evans: Hello, Dana.

Gardner: Paul, let me start with you if you don't mind. We hear an awful lot about legacy modernization. We usually look at it from a technical perspective. But it appears to me that in many of the discussions I have with organizations is that they are looking for more strategic levels of benefit, to finding business agility and flexibility benefits. The technical and economic considerations, while important in the short-term, pale in comparison to some of the longer term and more strategic benefits.

Pushed to the top

Evans: Where we find ourselves now -- and it has been brought on by the economic situation -- is that it has just pushed to the top an issue that's been out there for a long time. We have seen organizations doing a lot with their infrastructure, consolidating it, virtualizing it, all the right things. At the same time, we know, and a lot of CIOs or IT directors listening to this broadcast will know, that the legacy applications environment has somewhat been ignored.

Now, with the pressure on cost, people are saying, We've got to do something, but what can come out of that and what is coming out of that?" People are looking at this and saying, "We need to accomplish two things. We need a longer term strategy. We need an operational plan that fits into that, supported by our annual budget."

Foremost is this desire to get away from this ridiculous backlog of application changes, to get more agility into the system, and to get these core applications, which are the ones that provide the differentiation and the innovation for organizations, able to communicate with a far more mobile workforce.

At an event last week in America, a customer came up to me and said, "Seventy percent of our applications are batch running on a mainframe. How do I go to a line-of-business manager and say that I can connect that to a guy out there with a smartphone? How do I do that?" Today, it looks like an impossible gap to get across.

What people have to look at is where we're going strategically with our technology and our business alignment. At the same time, how can we have a short-term plan that starts delivering on some of the real benefits that people can get out there?

Gardner: In the first two parts of our series, we looked at several case studies that showed some remarkable return on investment (ROI). So, this is not just a nice to have strategic maturity process, but really pays dividends financially, and then has that longer term strategic roll-out.

Evans: Absolutely. These things have got to pay for themselves. An analyst last week, looked me in the face and said, "People want to get off the mainframe. They understand now that the costs associated with it are just not supportable and are not necessary."

One of the sessions you will hear in the virtual conference will be from Geoffrey Moore, where he talks about this whole difference between core applications and context -- context being applications that are there for productivity reasons, not for innovation or differentiation.

Lowest-cost platform

With a productivity application you want to get delivery on the lowest-cost platform you possibly can. The problem is that 20 or 30 years ago, people put everything on the mainframe. They wrote it all in code. Therefore, the challenge now is, what do you not need in code that can be in a package? What do you not need on the mainframe that could be running on a much more lower cost infrastructure or a completely different means of delivery, such as software as a service (SaaS).

The point is that there are demonstrably much less expensive ways of delivering these things. People have to just lift their heads up and look around, come and talk to us, and listen to the series and they will begin to see people who have done this before, and who have demonstrated that it works, as well as some of the amazing financial rewards that can be generated from this sort of work.

Gardner: John Pickett, let's go to you. We've talked about this, but I think showing it is always more impressive. The case studies that demonstrate the real-world returns tend to be the real education points. Could you share with us some of the case studies that you will be looking at during the upcoming virtual conference and walk us through how the alternative to mainframe process works?

Pickett: Sure, Dana. As Paul indicated, it's not really just about the overall cost, but it's about agility and being able to leverage the existing skills as well.

One of the case studies that I will go over is from the National Agricultural Cooperative Federation (NACF). It's a mouthful, but take a look at the number of banks that the NACF has. It has 5,500 branches and regional offices, so essentially it's one of the largest banks in Korea.

One of the items that they were struggling with was how to overcome some of the technology and performance limitations of the platform that they had. Certainly, in the banking environment, high availability and making sure that the applications and the services are running were absolutely key.

At the same time, they also knew that the path to the future was going to be through the IT systems that they had and they were managing. What they ended up doing was modernizing their overall environment, essentially moving their core banking structure from their current mainframe environment to a system running HP-UX. It included the customer and account information. They were able to integrate that with the sales and support piece, so they had more of a 360 degree view of the customer.

We talk about reducing costs. In this particular example, they were able to save $40 million on an annual basis. That's nice, and certainly saving that much money is significant, but, at the same time, they were able to improve their system response time two- to three-fold. So, it was a better response for the users.

But, from a business perspective, they were able to reduce their time to market. For developing a new product or service, that they were able to decrease that time from one month to five days.

Makes you more agile

If you are a bank and now you can produce a service much faster than your competition, that certainly makes it a lot easier and makes you a lot more agile. So, the agility is not just for the data center, it's for the business as well.

To take this story just a little bit further, they saw that in addition to the savings I just mentioned, they were able to triple the capacity of the systems in their environment. So, it's not only running faster and being able to have more capacity so you are set for the future, but you are also able to roll out business services a whole lot quicker than you were previously.

Gardner: I imagine that with many of these mainframe systems, particularly in a banking environment, they could be 15 or 20 years old. The requirements back then were dramatically different. If the world had not changed in 20 years, these systems might be up to snuff, but the world has changed dramatically. Look at the change we have seen in just the last nine months. Is that what we are facing here? We have a general set of different requirements around these types of applications.

Pickett: There are a couple of things, Dana. It's not only different requirements, but it's also being driven by a couple of different factors. Paul mentioned the cost and being able to be more efficient in today's economy. Any data center manager or CIO is challenged by that today. Given the high cost of legacy and mainframe environment, there's a significant amount of money to be saved.

It's not a one-size-fits-all. It's identifying the right choice for the application, and the right platform for the application as well.

Another example of what we were just talking about is that, if we shift to Europe, Middle East, and Africa region, there is very large insurance company in Spain. It ended up modernizing 14,000 million instructions per second (MIPS). Even though the applications had been developed over a number of years and decades, they were able to make the transition in a relatively short length of time. In a three- to six-month time frame they were able to move that forward.

With that, they saw a 2x increase in their batch performance. It's recognized as one of the largest batch re-hosts that are out there. It's just not an HP thing. They worked with Oracle on that as well to be able to drive Oracle 11g within the environment.

So, it's taking the old, but also integrating with the new. It's not a one-size-fits-all. It's identifying the right choice for the application, and the right platform for the application as well.

Gardner: So, this isn't a matter of swapping out hardware and getting a cheaper fit that way. This is looking at the entire process, the context of the applications, the extended process and architectural requirements in the future, and then looking at how to make the transition, the all important migration aspect.

Pickett: Yes. As we heard last week at a conference that both Paul and I were at, if all you're looking to do is to take your application and put it on to a newer, shinier box, then you are missing something.

Gardner: Let's go now to Les Wilson. Les, tell us a little bit about some studies that have been done and some of the newer insights into the incentives as to why the timing now for moving off of mainframes is so right.

Customer cases

Wilson: Thanks, Dana. I spend virtually every day talking directly to customers and to HP account teams on the subject of modernizing mainframes, and I'll be talking in detail about two particular customer case studies during the webinar.

Before I get into those details though, I want to preface my remarks by giving you some higher level views of what I see happening in the Americas. First of all, the team here is enjoying an unprecedented demand for our services from the customer base. It's up by a factor of 2 over 2008, and I think that some of the concepts that John and Paul have discussed around the reasons for that are very clear.

There's another point about HP's capabilities, as well, that makes us a very attractive partner for mainframe modernization solutions. Following the acquisition of EDS, we are really able to provide a one-stop shop for all of the services that any mainframe customer could require.

That includes anything from optimization of code, refactoring of code on the mainframe itself, all the way through re-hosting, migration, and transformation services. We've positioned ourselves as definitely the alternative to IBM mainframe customers.

In terms of customer situations, we've always had a very active business working with organizations in manufacturing, retail, and communications. One thing that I've perceived in the last year specifically -- it will come as no surprise to you -- is that financial institutions, and some of the largest ones in the world, are now approaching HP with questions about the commitment they have to their mainframe environments.

We're seeing a tremendous amount of interest from some of the largest banks in the United States, insurance companies, and benefits management organizations, in particular.

Second, maybe benefiting from some of the stimulus funds, a large number of government departments are approaching us as well. We've been very excited by customer interest in financial services and public sector. I just wanted to give you that by way of context.

In terms of the detailed case studies, when John Pickett first asked me to participate in the webinar, as well as in this particular recording, I was kind of struck with a plethora of choices. I thought, "Which case study should I choose that best represents some of the business that we are doing today?" So, I've picked two.

The first is a project we recently completed at a wood and paper products company. This is a worldwide concern. In this particular instance we worked with their Americas division on a re-hosting project of applications that are written in the Software AG environment. I hope that many of the listeners will be familiar with the database ADABAS and the language, Natural. These applications were written some years ago, utilizing those Software AG tools.

Demand was lowered

They had divested one of the major divisions within the company, and that meant that the demand for mainframe services was dramatically lowered. So, they chose to take the residual applications, the Software AG applications, representing about 300-350 MIPS, and migrate those in their current state, away from the mainframe, to an HP platform.

Many folks listening to this will understand that the Software AG environment can either be transformed and rewritten to run, say, in an Oracle or a Java environment, or we can maintain the customer's investment in the applications and simply migrate the ADABAS and Natural, almost as they are, from the mainframe to an alternative HP infrastructure. The latter is what we did.

By not needing to touch the mainframe code or the business rules, we were able to complete this project in a period of six months, from beginning to end. They are saving over $1 million today in avoiding the large costs associated with mainframe software, as well as maintenance and depreciation on the mainframe environment.

They're very, very pleased with the work that's being done. Indeed, we're now looking at an additional two applications in other parts of their business with the aim of re-hosting those applications as well.

They are saving over $1 million today in avoiding the large costs associated with mainframe software, as well as maintenance and depreciation on the mainframe environment.

The more monolithic approach to applications development and maintenance on the mainframe is a model that was probably appropriate in the days of the large conglomerates, where we saw a lot of companies trying to centralize all of that processing in large data centers. This consolidation made a lot of sense, when folks were looking for economies of scale in the mainframe world.

Today, we're seeing customers driving for that degree of agility you have just mentioned. In fact, my second case study represents that concept in spades. This is a large multinational manufacturing concern. They never allow their name to be used in these webcasts, so we will just refer to them as "a manufacturing company." They have a large number of businesses in their portfolio.

Our particular customer in this case study is the manufacturer of electronic appliances. One of the driving factors for their mainframe migration was precisely what you just said, Dana, that the ability to divest themselves from the large mainframe corporate environment, where most of the processing had been done for the last 20 years.

They wanted control of their own destiny to a certain extent, and they also wanted to prepare themselves for potential investment, divestment, and acquisition, just to make sure that they were masters of their own future.

Gardner: You mentioned earlier, John, about a two-times increase in the demand since 2008. I wonder if this demand increase is a blip. Is this something that is just temporary, or has the economy -- and some people call it the reset economy, actually changed the game -- and therefore IT needs to respond to that?

In a nutshell the question is whether this is going to be a two-year process, or are we changing the dynamic of IT and how business and IT need to come together in general?

Not a blip

Pickett: First, Dana, it's not a blip at all. We're seeing increased movement from mainframe over to HP systems, whether it's on an HP-UX platform or a Windows Server or SQL platform. Certainly, it's not a blip at all.

As a matter of fact, just within the past week, there was a survey by AFCOM, a group that represents data-center workers. It indicated that, over the next two years, 46 percent of the mainframe users said that they're considering replacing one or more of their mainframes.

Now, let that sink in -- 46 percent say they are going to be replacing high-end systems over the next two years. That's an absurdly high number. So, it certainly points to a trend that we are seeing in that particular environment -- not a blip at all.

Dana, that also points to the skills piece. A lot of times when we talk to people in a mainframe environment the question is, "I've got a mainframe, but what about the mainframe people that I have? They're good people, they know the process, and they have been around for a while." We found that HP, and moving to an HP centralized environment is really a soft landing for these people.

They can use the process skills that they have developed over time. They're absolutely the best at what they do in the process environment, but it doesn’t have to be tied to the legacy platform that they have been working on for the last 10 or 20 years.

We've found that there is a very strong migration for those skills and landing in a place where they can use and develop them for years to come.

We've found that you can take those same processes and apply them to a large HP Integrity Superdome environment, or NonStop environment. We've found that there is a very strong migration for those skills and landing in a place where they can use and develop them for years to come.

Gardner: Les, why do you see this as a longer term trend, and what are the technological changes that we can expect that will make this even more enticing, that is to say, lower cost, more efficient, and higher throughput systems that allow for the agility to take place as well?

Wilson: Good question, Dana, and you have two parts to it. Let me address the first one about the trend. I've been involved in this kind of business on and off since 1992. We have numbers going back to the late 1980s as to the fact that at that time there were over 50,000 mainframes installed worldwide.

When I next got into this business in 2004, the analyst firms confirmed that the number was now around 15,000-16,000. Just this week, we have had information, confirmed by another analyst, that the number of installed mainframes is now at about 10,400. We've seen a 15-20 year trend away from the mainframe, and that will continue, given this unprecedented level of interest we are seeing right now.

You talked about technology trends. Absolutely. Five years ago, it would have been fair to say that there were still mainframe environments and applications that could not be replaced by their open-system equivalents. Today, I don't think that that's true at all.

Airline reservation system

To give you an example, HP, in cooperation with American Airlines, has just announced that we're going to be embarking on a three-year transition of all of the TPF-based airline reservation systems that we HP has been providing as services to customers for 20 years.

That TPF environment will be re-engineered in its entirety over the course of the next three years to provide those same and enhanced airline reservation systems to customers on a Microsoft-HP bladed environment.

That's an unprecedented change in what was always seen as a mainframe centric application, airlines reservations, with the number of throughputs and the amount of transactions that need to be done every second. When these kinds of applications can be transformed to open systems' platforms, it's open season on any mainframe application.

Furthermore, the trend in terms of open-systems price performance improvement continues at 30-35 percent per annum. You just need to look at the latest Intel processors, whether they be x86 or Itanium-based, to see that. That price performance trend is huge in the open systems market.

I've been tracking what's been going on in the IBM System Z arena, and since there are no other competitors in this market, we see nothing more than 15 percent, maybe 18 percent, per annum price performance improvement. As time goes on, HP and industry standard platforms continue, and will continue, to outpace the mainframe technology. So, this trend is bound to happen.

People have to take considered opinions. Investments here are huge. The importance of legacy systems is second to none.

Gardner: Paul, we've heard quite a bit of compelling information. Tell us about the upcoming conference, and perhaps steps that folks can take to get more information or even get started as they consider their alternatives to mainframes?

Evans: Based on what you've heard from John and Les, there is clearly an interest out there in terms of understanding. I don't think this is, as they say in America, a slam dunk. The usual statement is, "How do you eat an elephant? and the answer is, "One bite at a time."

The point here is that this is not going to happen overnight. People have to take considered opinions. Investments here are huge. The importance of legacy systems is second to none. All that means that the things that John and Les are talking about are going to happen strategically over a long time. But, we have people coming to us every day saying, "Please, can you help me understand how do I start, where do I go, where do I go now, or where do I go next week, next year, or next month?

The reason behind the conference was to take a sort of multi-sided view of this. One side is the business requirement, which people like Geoffrey Moore will be talking about -- where the business is going and what does it need.

We'll be looking at a customer case study from the Italian Ministry of Education, looking at how they used multiple modernization strategies to fix their needs. We'll be looking at tools we developed, so that people can understand what the code is doing. We'll be hearing from Les, John, and customers -- Barclays Bank in London -- about what they have been doing and the results they have been getting.

Then, at the very end, we'll be hearing from Dale Vecchio, vice president of Gartner research, about what he believes is really going on.

Efficiency engine

The thing that underpins this is that the business requirement several decades ago drove the introduction of the mainframe. People needed an efficiency engine for doing payroll, human resources, whatever it may be, moving data around. The mainframe was invented and was built perfectly. It was an efficiency engine.

As time has gone on, people look at technology now to become an effectiveness engine. We've seen the blending of technologies between mainframes, PCs, and midrange systems. People now take this whole efficiency thing for granted. No one runs their own payroll, even to the point that people now look to BPOs or those sorts of things.

As we go forward, with people being highly mobile, with mobile devices dramatically exploding all over the place in terms of smartphones, Net PCs, or whatever, people are looking to blend technologies that will deliver both the efficiency and the effectiveness, but also the innovation. Technology is now the strategic asset that people will use going forward. There needs to be a technological response to that.

Over the last year or two, either John or Les referred to the enormous amounts of raw power we can now get from, say, an Intel microprocessor. What we want to do is harness that power and give people the ability to innovate and differentiate, but, at the same time, run those context applications that keep their companies alive.

That's really what we're doing with the conference -- demonstrating, in real terms, how we can get this technology to the bottom-line and how we can exploit it going forward.

Gardner: Well, great. We've been hearing about some case studies that demonstrate how costs can be cut significantly, while productivity and agility are boosted.

I want to thank our guests in today’s discussion. We've been joined by John Pickett, Worldwide Mainframe Modernization Program manager. Thank you, John.

Pickett: Thank you, Dana.

Gardner: We've also been joined by Les Wilson, America’s Mainframe Modernization director. Thank you, Les.

Wilson: Thank you for the opportunity, Dana.

Gardner: And also Paul Evans, worldwide marketing lead on Applications Transformation at HP. Thanks again Paul.

Evans: It's a pleasure.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions. You have been listening to a sponsored BriefingsDirect podcast. Thanks for listening, and come back next time.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Learn more. Sponsor: Hewlett-Packard.

---

Gain more insights into "Application Transformation: Getting to the Bottom Line" via a series of HP virtual conferences. For more on Application Transformation, and to get real time answers to your questions, register to access the virtual conferences for your region:

Access the Asia Pacific event.
Access the EMEA event.
Access the Americas event.

---

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Learn more. Sponsor: Hewlett-Packard.


Transcript of the third in a series of sponsored BriefingsDirect podcasts on the rationale and strategies for application transformation. Copyright Interarbor Solutions, LLC, 2005-2009. All rights reserved.

Game-Changing Architectural Advances Take Data Analytics to New Performance Heights

Transcript of a BriefingsDirect podcast on how new advances in collocating applications with data architecturally provides analytics performance breakthroughs.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Learn more. Sponsor: Aster Data Systems.

Dana Gardner: Hi, this is Dana Gardner, principal analyst at Interarbor Solutions, and you're listening to BriefingsDirect.

Today, we present a sponsored podcast discussion on how new architectures for data and logic processing are ushering in a game-changing era of advanced analytics.

These new approaches support massive data sets to produce powerful insights and analysis, yet with unprecedented price-performance. As we enter 2010, enterprises are including more forms of diverse data into their business intelligence (BI) activities. They're also diversifying the types of analysis that they expect from these investments.

We're also seeing more kinds and sizes of companies and government agencies seeking to deliver ever more data-driven analysis for their employees, partners, users, and citizens. It boils down to giving more communities of participants what they need to excel at whatever they're doing. By putting analytics into the hands of more decision makers, huge productivity wins across entire economies become far more likely.

But such improvements won’t happen if the data can't effectively reach the application's logic, if the systems can't handle the massive processing scale involved, or the total costs and complexity are too high.

In this discussion we examine how convergence of data and logic, of parallelism and MapReduce -- and of a hunger for precise analysis with a flood of raw new data -- all are setting the stage for powerful advanced analytics outcomes.

Here to help us learn how to attain advanced analytics and to uncover the benefits from these new architectural activities for ubiquitous BI, are Jim Kobielus, senior analyst at Forrester Research. Welcome, Jim.

Jim Kobielus: Hi, Dana. Hi, everybody.

Gardner: We're also joined by Sharmila Mulligan, executive vice president of marketing at Aster Data. Welcome, Sharmila.

Sharmila Mulligan: Thank you. Hello, everyone.

Gardner: Jim, let me start with you. We're looking at a shift now, as I have mentioned, in response to oceans of data and the need for analysis across different types of applications and activities. What needs to change? The demands are there, but what needs to change in terms of how we provide the solution around these advanced analytical undertakings?

Rethinking platforms

Kobielus: First, Dana, we need to rethink the platforms with which we're doing analytical processing. Data mining is traditionally thought of as being the core of advanced analytics. Generally, you pull data from various sources into an analytical data mart.

That analytical data mart is usually on a database that's specific to a given predictive modeling project, let's say a customer analytics project. It may be a very fast server with a lot of compute power for a single server, but quite often what we call the analytical data mart is not the highest performance database you have in your company. Usually, that high performance database is your data warehouse.

As you build larger and more complex predictive models -- and you have a broad range of models and a broad range of statisticians and others building, scoring, and preparing data for these models -- you quickly run into resource constraints on your existing data-mining platform, really. So, you have to look for where you can find the CPU power, the data storage, and the I/O bandwidth to scale up your predictive modeling efforts. That's the number one thing. The data warehouse is the likely suspect.

Also, you need to think about the fact that these oceans of data need to be prepared, transformed, cleansed, meshed, merged, and so forth before they can be brought into your analytical data mart for data mining and the like.

Quite frankly, the people who do predictive modeling are not specialists at data preparation.

Quite frankly, the people who do predictive modeling are not specialists at data preparation. They have to learn it and they sometimes get very good at it, but they have to spend a lot of time on data mining projects, involved in the grunt work of getting data in the right format just to begin to develop the models.

As you start to rethink your whole advanced analytics environment, you have to think through how you can automate to a greater degree all these data preparation, data loading chores, so that the advanced analytics specialists can do what they're supposed to do, which is build and tune models of various problem spaces. Those are key challenges that we face.

But, there is one third challenge, which is advanced analytics producing predictive models. Those predictive models increasingly are deployed in-line to transactional applications, like your call center, to provide some basic logic and rules that will drive such important functions as "next best offer" being made to customers based on a broad variety of historical and current information.

How do you inject predictive logic into your transactional applications in a fairly seamless way? You have to think through that, because, right now, quite often analytical data models, predictive models, in many ways are not built for optimal embedding within your transactional applications. You have to think through how to converge all these analytical models with the transactional logic that drives your business.

Gardner: Okay. Sharmila, are your users or the people that you talk to in the market aware that this shift is under way? Do they recognize that the same old way of doing things is not going to sustain them going forward?

New data platform

Mulligan: What we see with customers is that the advanced analytics needs and the new generation of analytics that they are trying to do is driving the need for a new data platform.

Previously, the choice of a data management platform was based primarily on price-performance, being able to effectively store lots of data, and get very good performance out of those systems. What we're seeing right now is that, although price performance continues to be a critical factor, it's not necessarily the only factor or the primary thing driving their need for a new platform.

What's driving the need now, and one of the most important criteria in the selection process, is the ability of this new platform to be able to support very advanced analytics.

Customers are very precise in terms of the type of analytics that they want to do. So, it's not that a vendor needs to tell them what they are missing. They are very clear on the type of data analysis they want to do, the granularity of data analysis, the volume of data that they want to be able to analyze, and the speed that they expect when they analyze that data.

There is a big shift in the market, where customers have realized that their preexisting platforms are not necessarily suitable for the new generation of analytics that they're trying to do.

They are very clear on what their requirements are, and those requirements are coming from the top. Those new requirements, as it relates to data analysis and advanced analytics, are driving the selection process for a new data management platform.

There is a big shift in the market, where customers have realized that their preexisting platforms are not necessarily suitable for the new generation of analytics that they're trying to do.

Gardner: Let's take a pause and see if we can't define these advanced analytics a little better. Jim, what do we mean nowadays when we say "advanced analytics?"

Kobielus: Different people have their definitions, but I'll give you Forrester's definition, because I'm with Forrester. And, it makes sense to break it down into basic analytics versus advanced analytics.

What is basic analytics? Well, that's BI. It's the core of BI that you build your decision support environment on. That's reporting, query, online analytical processing, dashboarding, and so forth. It's fairly clear what's in the core scope of BI.

Traditional basic analytics is all about analytics against deep historical datasets and being able to answer questions about the past, including the past up to the last five seconds. It's the past that's the core focus of basic analytics.

What's likely to happen

Advanced analytics is focused on how to answer questions about the future. It's what's likely to happen -- forecast, trend, what-if analysis -- as well as what I like to call the deep present, really current streams for complex event processing. What's streaming in now? And how can you analyze the great gushing streams of information that are emanating from all your applications, your workflows, and from social networks?

Advanced analytics is all about answering future-oriented, proactive, or predictive questions, as well as current streaming, real-time questions about what's going on now. Advanced analytics leverages the same core features that you find in basic analytics -- all the reports, visualizations, and dashboarding -- but then takes it several steps further.

First and foremost, it's all about amassing a data warehouse or a data mart full of structured and unstructured information and being able to do both data mining against the structured information, and text analytics or content analytics against the unstructured content.

Then, in the unstructured content, it's being able to do some important things, like natural language processing to look for entities and relationships and sentiments and the voice of the customer, so you can then extrapolate or predict what might happen in the future. What might happen if you make a given offer to a given customer at a given time? How are they likely to respond? Are they likely to jump to the competition? Are they likely to purchase whatever you're offering? All those kinds of questions.

The query and reporting aspect continues to be very important, but the difference now is that the size of the data set is far larger than what the customer has been running with before.

Gardner: Sharmila, do you have anything to offer further on defining advanced analytics in this market?

Mulligan: Before I go into advanced analytics, I'd like to add to what Jim just talked about on basic analytics. The query and reporting aspect continues to be very important, but the difference now is that the size of the data set is far larger than what the customer has been running with before.

What you've got is a situation where they want to be able to do more scalable reporting on massive data sets with very, very fast response times. On the reporting side, in terms of the end result to the customer, it is similar to the type of report they are trying to achieve, but the difference is that the quantity of data that they're trying to get at, and the amount of data that these reports are filling up is far greater than what they had before.

That's what's driving a need for a new platform underneath some of the preexisting BI tools that are, in themselves, good at reporting, but what the BI tools need is a data platform beneath them that allows them to do more scalable reporting than you could do before.

Kobielus: I just want to underline that, Sharmila. What Forrester is seeing is that, although the average data warehouse today is in the 1-10 terabyte range for most companies, we foresee the average warehouse size going, in the middle of the coming decade, into the hundreds of terabytes.

In 10 years or so, we think it's possible, and increasingly likely, that petabyte-scale data warehouses or content warehouses will become common. It's all about unstructured information, deep history, and historical information. A lot of trends are pushing enterprises in the direction of big data.

Managing big data

Mulligan: Absolutely. That is obviously the big topic here, which is, how do you manage big data? And, big data could be structured or it could be unstructured. How do you assimilate all this in one platform and then be able to run advanced analytics on this very big data set?

Going back to what Jim discussed on advanced analytics, we see two big themes. One is
the real-time nature of what our customers want to do. There are particular use cases, where what they need is to be able to analyze this data in near real-time, because that's critical to being able to get the insights that they're looking for.

Fraud analytics is a good example of that. Customers have been able to do fraud analytics, but they're running fraud checks after the fact and discovering where fraud took place after the event has happened. Then, they have to go back and recover from that situation. Now, what customers want, is to be able to run fraud analytics in near real-time, so they can catch fraud while it's happening.

What you see is everything from cases in financial services companies related to product fraud, as well as, for example, online gaming sites, where users of the system are collaborating on the site and trying to commit fraud. Those type of scenarios demand a system that can return the fraud analysis data near real-time, so it can block these users from conducting fraud while it's happening.

The other big thing we see is the predictive nature of what customers are trying to do. Jim talked about predictive analytics and modeling analytics. Again, that's a big area that we see massive new opportunity and a lot of new demand. What customers are trying to do there is look at their own customer base to be able to analyze data, so that they can predict trends in the future.

. . . The other big theme we see is the push toward analysis that's really more near real time than what they were able to do before.

For example, what are the buying trends going to be, let's say at Christmas, for consumers who live in a certain area? There is a lot around behavior analysis. In the telco space, we see a lot of deep analysis around trying to model behavior of customers on voice usage of their mobile devices versus data usage.

By understanding some of these patterns and the behavior of the users in more depth, these organizations are now able to better service their customers and offer them new product offerings, new packages, and a higher level or personalization, by understanding the behavior of their customers in more depth.

Predictive analytics is a term that's existed for a while, and is something that customers have been doing, but it's really reaching new levels in terms of the amount of data that they're trying to analyze for predictive analytics, and in the granularity of the analytics itself in being able to deliver deeper predictive insight and models.

As I said, the other big theme we see is the push toward analysis that's really more near real time than what they were able to do before. This is not a trivial thing to do when, it comes to very large data sets, because what you are asking for is the ability to get very, very quick response times and incredibly high performance on terabytes and terabytes of data to be able to get these kind of results in real-time.

Gardner: Jim, these examples that Sharmila has shared aren't just rounding errors. This isn't a movement toward higher efficiency. These are game changers. These are going to make or break your business. This is going to allow you to adjust to a changing economy and to shifting preferences by your customers. We're talking about business fundamentals here.

Social network analysis

Kobielus: We certainly are. Sharmila was discussing behavioral analysis, for example, and talking about carrier services. Let's look at what's going to be a true game changer, not just for business, but for the global society. It's a thing called social network analysis.

It's predictive models, fundamentally, but it's predictive models that are applied to analyzing the behaviors of networks of people on the web, on the Internet, Facebook, and Twitter, in your company, and in various social network groupings, to determine classification and clustering of people around common affinities, buying patterns, interests, and so forth.

As social networks weave their way into not just our consumer lives, but our work lives, our life lives, social network analysis -- leveraging all the core advanced analytics of data mining and text analytics -- will take the place of the focus group. In an online world, everything is virtual. As a company, you're not going to be able, in any meaningful way, to bring together your users into a single room and ask them what they want you to do or provide for them.

What you're going to do, though, is listen to them. You're going to listen to all their tweets and their Facebook updates and you're going to look at their interactions online through your portal and your call center. Then, you're going to take all that huge stream of event information -- we're talking about complex event processing (CEP) -- you're going to bring it into your data warehousing grid or cloud.

You're also going to bring historical information on those customers and their needs. You're going to apply various social network behavioral analytics models to it to cluster people into the categories that make us all kind of squirm when we hear them, things like yuppie and Generation X and so forth. Professionals in the behavioral or marketing world are very good at creating segmentation of customers, based on a broad range of patterns.

They can get a sense of how a product or service is being perceived in real-time, so that the the provider of that product or service can then turn around and tweak that marketing campaign . . .

Social network analysis becomes more powerful as you bring more history into it -- last year, two years, five years, 10 years worth of interactions -- to get a sense for how people will likely respond likely to new offers, bundles, packages, campaigns, and programs that are thrown at them through social networks.

It comes down to things like Sharmila was getting at, simple things in marketing and sales, such as a Hollywood studio determining how a movie is being perceived by the marketplace, by people who go out to the theater and then come out and start tweeting, or even tweeting while they are in the theater -- "Oh, this movie is terrible" or "This movie rocks."

They can get a sense of how a product or service is being perceived in real-time, so that the the provider of that product or service can then turn around and tweak that marketing campaign, the pricing, and incentives in real-time to maximize the yield, the revenue, or profit of that event or product. That is seriously powerful and that's what big data architectures allow you to do.

If you can push not just the analytic models, but to some degree bring transactional applications, such as workflow, into this environment to be triggered by all of the data being developed or being sifted by these models, that is very powerful.

Gardner: We know that things are shifting and changing. We know that we want to get access to the data and analytics. And, we know what powerful things those analytics can do for us. Now, we need to look at how we get there and what's in place that prevents us.

Let's look at this architecture. I'm looking into MapReduce more and more. I am even hearing that people are starting to write MapReduce into their requests for proposals (RFPs), as they're looking to expand and improve their situation. Sharmila, what's wrong with the current environment and why do we need to move into something a bit different?

Moving the data

Mulligan: One of the biggest issues that the preexisting data pipeline faces is that the data lives in a repository that's removed from where the analytics take place. Today, with the existing solutions, you need to move terabytes and terabytes of data through the data pipeline to the analytics application, before you can do your analysis.

There's a fundamental issue here. You can't move boulders and boulders of data to an application. It's too slow, it's too cumbersome, and you're not factoring in all your fresh data in your analysis, because of the latency involved.

One of the biggest shifts is that we need to bring the analytics logic close to the data itself. Having it live in a completely different tier, separate from where the data lives, is problematic. This is not a price-performance issue in itself. It is a massive architectural shift that requires bringing analytics logic to the data itself, so that data is collocated with the analytics itself.

MapReduce, which you brought up earlier, plays a critical role in this. It is a very powerful technology for advanced analytics and it brings capabilities like parallelization to an application, which then allows for very high-performance scalability.

What we see in the market these days are terms like "in-database analytics," "applications inside data," and all this is really talking about the same thing. It's the notion of bringing analytics logic to the data itself.

One of the biggest shifts is that we need to bring the analytics logic close to the data itself.

I'll let Jim add a lot more to that since he has developed a lot of expertise in this area.

Gardner: Jim, are we in a perfect world here, where we can take the existing BI applications and apply them to this new architecture of joining logic and data in proximity, or do we have to come up with whole new applications in order to enjoy this architectural benefit?

Kobielus: Let me articulate in a little bit more detail what MapReduce is and is not. MapReduce is, among other things, a set of extensions to SQL -- SQL/MapReduce (SQL/MR). So, you can build advanced analytic logic using SQL/MR that can essentially do the data prep, the data transformations, the regression analyses, the scoring, and so forth, against both structured data in your relational databases and unstructured data, such as content that you may source from RSS feeds and the like.

To the extent that we always, or for a very long time, have been programming database applications and accessing the data through standard SQL, SQL/MR isn't radically different from how BI applications have traditionally been written.

Maximum parallelization

But, these are extensions and they are extensions that are geared towards enabling maximum parallelization of these analytic processes, so that these processes can then be pushed out and be executed, not just in-databases, but in file systems, such as the Hadoop Distributed File System, or in cloud data warehouses.

MapReduce, as a programming model and as a language, in many ways, is agnostic as to the underlying analytic database, file system, or cloud environment where the information, as a whole lives, and how it's processed.

But no, you can't take your existing BI applications, in terms of the reporting, query, dashboarding, and the like, transparently move them, and use MapReduce without a whole lot of rewriting of these applications.

You can't just port your existing BI applications to MapReduce and database analytics. You're going to have to do some conversions, and you're going to have to rewrite your applications to take advantage of the parallelism that SQL/MR enables.

MapReduce, in many ways, is geared not so much for basic analytics. It's geared for advanced analytics. It's data mining and text mining. In many ways, MapReduce is the first open framework that the industry has ever had for programming the logic for both data mining and text mining in a seamless way, so that those two types of advanced analytic applications can live and breathe and access a common pool of complex data.

In the marriage of SQL with MapReduce, the real intent is to bring the power of MapReduce to the enterprise, so that SQL programmers can now use that technology.

MapReduce is an open standard that Aster clearly supports, as do a number of other database and data warehousing vendors. In the coming year and the coming decade, MapReduce and Hadoop -- and I won't go to town on what Hadoop is -- will become fairly ubiquitous within the analytics arena. And, that’s a good thing.

So, any advanced analytic logic that you build in one tool, in theory, you can deploy and have it optimized for execution in any MapReduce-enabled platform. That’s the promise. It’s not there yet. There are a lot of glitches, but that’s the strong promise.

Mulligan: I'd like to add a little bit to that Dana. In the marriage of SQL with MapReduce, the real intent is to bring the power of MapReduce to the enterprise, so that SQL programmers can now use that technology. MapReduce alone does require some sophistication in terms of programming skills to be able to utilize it. You may typically find that skill set in Web 2.0 companies, but often you don’t find developers who can work with that in the enterprise.

What you do find in enterprise organizations is that there are people who are very proficient at SQL. By bringing SQL together with MapReduce what enterprise organizations have is the familiarity of SQL and the ease of using SQL, but with the power of MapReduce analytics underneath that. So, it’s really letting SQL programmers leverage skills they already have, but to be able to use MapReduce for analytics.

Important marriage

Over time, of course, it’s possible that there will be more expertise developed within enterprise organizations to use MapReduce natively, but at this time and, we think, in the next couple of years, the SQL/MapReduce marriage is going to be very important to help bring MapReduce across the enterprise.

Hadoop, itself, obviously is an interesting platform too in being able to store lots of data cost effectively. However, often customers will also want some of the other characteristics of a data warehouse, like workload management, failover, backup recovery, etc., that the technology may not necessarily provide.

MapReduce right now, available with massive parallel processing (MPP), the new generation of MPP data warehouse is such a vast data solution, does bring kind of the best of both worlds. It brings what companies need in terms of the enterprise data warehouse capabilities. It lets you put application logic near data, as we talked about earlier. And, it brings MapReduce, but through the SQL/MapReduce framework, which really primarily is designed to ease adoption and use of MapReduce within the enterprise.

Gardner: Jim, we are on a journey. It’s going to be several years before we are getting to where we want to go, but there is more maturity in some areas than others. And, there is an opportunity to take technologies that are available now and do some real strong business outcomes and produce those outcomes.

Give me a sense of where you see the maturity of the architecture, of the SQL, and the tools and making these technologies converge? Who is mature? How is this shaking out a little bit?

Kobielus: Maturity is a best practice, in this case in-database analytics. As I said, it’s widely supported through proprietary approaches by many vendors.

In terms of the maturity, it's judged by adoption of an open industry framework with cross-vendor interoperability.

In terms of the maturity, it's judged by adoption of an open industry framework with cross-vendor interoperability. it's not mature yet, in terms of MapReduce and Hadoop. There are pioneering vendors like Aster, but there are a significant number of established big data warehousing vendors that have varying degrees of support now or in the near future for these frameworks. We're seeing strong indications. In fact, Teradata already is rolling out MapReduce and Hadoop support in their data warehousing offerings.

We're not yet seeing a big push from Oracle, or from Microsoft for that matter, in the direction of support for MapReduce or Hadoop, but we at Forrester believe that both of those vendors, in particular, will come around in 2010 with greater support.

IBM has made significant progress with its support for Hadoop and MapReduce, but it hasn’t yet been fully integrated into that particular vendor's platform.

Looking to 2010, 2011

If we look at a broad range of other data warehousing vendors like Sybase, Greenplum, and others, most vendors have it on their roadmap. To some degree, various vendors have these frameworks in in development right now. I think 2010 and 2011 are the years when most of the data warehousing and also data mining vendors will begin to provide mature, interoperable implementations of these standards.

There is a growing realization in the industry that advanced analytics is more than just being able to mine information at rest, which is what MapReduce and Hadoop are geared to doing. You also need to be able to mine and do predictive analytics against data in motion. That’s CEP. MapReduce and Hadoop are not really geared to CEP applications of predictive modeling.

There needs to be, and there will be over the next five years or so, a push in the industry to embed MapReduce and Hadoop. There are few vendors that are showing some progress toward CEP predictive modeling, but it’s not widely supported yet, and it’s in proprietary approaches.

In this coming decade, we're going to see predictive logic deployed into all application environments, be they databases, clouds, distributed file systems, CEP environments, business process management (BPM) systems, and the like. Open frameworks will be used and developed under more of a service-oriented architecture (SOA) umbrella, to enable predictive logic that’s built in any tool to be deployed eventually into any production, transaction, or analytic environment.

It will take at least 3 to 10 years for a really mature interoperability framework to be developed, for industry to adopt it, and for the interoperability issues to be worked out.

It will take at least 3 to 10 years for a really mature interoperability framework to be developed, for industry to adopt it, and for the interoperability issues to be worked out. It’s critically important that everybody recognizes that big data, at rest and in motion, needs to be processed by powerful predictive models that can be deployed into the full range of transactional applications, which is where the convergence of big data, analytics, and transactions come in.

Data warehouses, as the core of your analytics environment, need to evolve to become in their own right application servers that can handle both the analytic applications or traditional data warehousing in BI and data mining, as well as the transactional logic, and really handle it all with full security and workload isolation, failover, and so forth in a way that’s seamless.

I'm really excited, for example, by what Aster has rolled out with their latest generation, 4.0 of the Data-Application Server. I see a little bit of progress by Oracle on the Exadata V2. I'm looking forward to seeing if other vendors follow suit and provide a cloud-based platform for a broad range of transactional analytics.

Gardner: Sharmila, Jim has painted a very nice picture of where he expects things to go. He mentioned Aster Data 4.0. Tell us a little bit about that, and where you see the stepping stones lining up.

Mulligan: As I mentioned earlier, one of the biggest requirements in order to be able to do very advanced analytics on terabyte- and petabyte-level data sets, is to bring the application logic to the data itself. Earlier, I described why you need to do this. You want to eliminate as much data movement as possible, and you want to be able to do this analysis in as near real-time as possible.

What we did in Aster Data 4.0 is just that. We're allowing companies to push their analytics applications inside of Aster’s MPP database, where now you can run your application logic next to the data itself, so they are both collocated in the same system. By doing so, you've eliminated all the data movement. What that gives you is very, very quick and efficient access to data, which is what's required in some of these advanced analytics application examples we talked about.

Pushing the code

What kind of applications can you push down into the system? It can be any app written in Java, C, C++, Perl, Python, .NET. It could be an existing custom application that an organization has written and that they need to be able to scale to work on much larger data sets. That code can be pushed down into the apps database.

It could be a new application that a customer is looking to write to do a level of analysis that they could not do before, like real-time fraud analytics, or very deep customer behavior analysis. If you're trying to deliver these new generations of advanced analytics apps, you would write that application in the programming language of your choice.

You would push that application down into the Aster system, all your data would live inside of the Aster MPP database, and the application would run inside of the same system collocated with the data.

In addition to that, it could be a packaged application. So, it could be an application like software as a service (SaaS) that you want to scale to be able to analyze very large data sets. So, you could push a packaged application inside the system as well.

One of the fundamental things that we leverage to allow you to do more powerful analytics with these applications is MapReduce. You don’t have to MapReduce enable an application when you push it down into the apps system, but you could choose to and, by doing so, you automatically parallelize the application, which gives you very high performance and scalability when it comes to accessing large datasets. You also then leverage some of the analytics capabilities of MapReduce that are not necessary inherent in something like SQL.

That's a very attractive feature, because fundamentally the data warehousing cloud is an analytic application server.

The key components of 4.0 drive to where it's providing you a platform that can efficiently and cost effectively store massive amounts of data, plus give you a platform that allows you to do very advanced and sophisticated analytics. To run through those key things that we've done in 4.0, is first, the ability to push applications inside the system, so apps are collocated with the data.

We also offer SQL/MapReduce as the interface. Business analysts who are working with this application on a regular basis don’t have to learn MapReduce. They can use SQL/MR and leverage their existing SQL skills to work with that app. So, it makes it very easy for any number of business analysts in the organization to leverage their preexisting SQL skills and work with this app that's pushed down into the system.

Finally, in order to support the ability to run application inside a data, which as I said earlier is nontrivial, we added fundamental new capabilities like Dynamic Mix Workload Management. Workload Management in the Aster system works not just on data queries, but on the application processes as well, so you can balance workloads when you have a system that's managing data and applications.

Kobielus: Sharmila, I think the greatest feature of the 4.0 is simply the ability to run predictive models developed in SaaS or other tools in their native code without converting them necessarily to SQL/MR. That means that your customers can then leverage that huge installed piece of intellectual property or pool of intellectual property, all those models, bring it in, and execute it natively within your distributed grid or cloud, as a way of avoiding having to do that rewrite. Or, if they wish, they can migrate them or convert them over to SQL/MR. It's up to them.

That's a very attractive feature, because fundamentally the data warehousing cloud is an analytic application server. Essentially, you want that ability to be able to run disparate legacy models in parallel. That's just a feature that needs to be adopted by the industry as a whole.

The customer decides

Mulligan: Absolutely. I do want to clarify that the Aster 4.0 solution can be deployed in the cloud, or it can be installed in a standard implementation on-premise, or it could be adopted in an appliance mode. We support all three. It's up to the customer which of those deployment models they need or prefer.

To talk in a little bit more detail about what Jim is referring to, the ability to take an existing app, have to do absolutely no rewrite, and push that application down is, of course, very powerful to customers. It means that they can immediately take an analytics app they already have and have it operate on much larger data sets by simply taking that code and pushing it down.

That can be done literally within a day or two. You get the Aster system, you install it, and then, by the second day, you could be pushing your application down.

If you choose to leverage the MapReduce analytics capabilities, then as I said earlier, you would MapReduce enable an app. This simply means you take your existing application and, again, you don’t have to do any rewrite of that logic. You just add MapReduce functions to it and, by doing so, you have now MapReduce-enabled it. Then, you push it down and you have SQL/MR as an interface to that app.

The process of MapReduce enabling an app also is very simple. It's a couple of days process. This is not something that takes weeks and weeks to do. It literally can be done in a couple of days.

It means that they can immediately take an analytics app they already have and have it operate on much larger data sets by simply taking that code and pushing it down.

We had a retailer recently who took an existing app that they had already written, a new type of analytics application that they wanted to deploy. They simply added MapReduce capabilities to it and pushed it down into the Aster system, and it's now operating on very, very large data sets, and performing analytics that they weren't able to originally do.

The ease of application push down and the ease of MapReduce enabling is definitely key to what we have done in 4.0, and it allows companies to realize the value of this new type of platform right away.

Gardner: I know it's fairly early in the roll out. Do you have any sense of metrics, from some of these users? What do they get back? We talked earlier in the examples about what could be done and what should be done nowadays with analysis. Do you have any sense of what they have able to do with 4.0?

Reducing processing times

Mulligan: For example, we have talked about customers like comScore who are processing 1.6 billion rows of data on a regular basis, and their data volumes continue to grow. They have many business analysts who operate the system and run reports on a daily basis, and they are able to get results very quickly on a large data set.

We have customers who have gone from 5-10 minute processing times on their data set, to 5 seconds, as a result of putting the application inside of the system.

We have had fraud applications that would take 60-90 minutes to run in the traditional approach, where the app was running outside the database, and now those applications run in 60-90 seconds.

Literally, by collocating your application logic next to the data itself, you can see that you are immediately able to go from many minutes of processing time, down to seconds, because you have eliminated all the data movement altogether. You don’t have to move terabytes of data.

Add to that the fact that you can now access terabyte-sized data sets, versus what customers have traditionally been left with, which is only the ability to process data sets in the order of several tens of gigabytes or hundreds of gigabytes. Now, we have telcos, for example, processing four- or five-terabyte data sets with very fast response time.

We're talking about a collision of two cultures, or more than two cultures. Data warehousing professionals and data mining professionals live in different worlds, as it were.

It's the volume of data, the speed, the acceleration, and response time that really provide the fundamental value here. MapReduce, over and above that, allows you to bring in more analytics power.

Gardner: A final word to you, Jim Kobielus. This really is a good example of how convergence is taking place at a number of different levels. Maybe you could give us an insight into where you see convergence happening, and then we'll have to leave it there.

Kobielus: First of all, with convergence the flip side is collision. I just want to point out a few issues that enterprises and users will have to deal with, as they move toward this best practice called in-database analytics and convergence of the transactions and analytics.

We're talking about a collision of two cultures, or more than two cultures. Data warehousing professionals and data mining professionals live in different worlds, as it were. They quite often have an arm's length relationship to each other. The data warehouse traditionally is a source of data for advanced analytics.

This new approach will require a convergence, rapprochement, or a dialog to be developed between these two groups, because ultimately the data warehouse is where the data mining must live. That's going to have to take place, that coming together of the tribes. That's one of the best emerging practices that we're recommending to Forrester clients in that area.

Common framework

Also, transaction systems -- enterprise resource planning (ERP) and customer relationship management (CRM) -- and analytic systems -- BI and data warehousing -- are again two separate tribes within your company. You need to bring together these groups to work out a common framework for convergence to be able to take advantage of this powerful new architecture that Sharmila has sketched out here.

Much of your transactional logic will continue to live on source systems, the ERP, CRM, supply chain management, and the like. But, it will behoove you, as an organization, as a user to move some transactional logic, such as workflow, in particular, into the data warehousing cloud to be driven by real-time analytics and KPIs, metrics, and messages that are generated by inline models built with MapReduce, and so forth, and pushed down into the warehousing grid or cloud.

Workflow, and especially rules engines, increasingly we will find to be tightly integrated or brought into a warehousing or analytics cloud that's got inline logic.

Another key trend for convergence is that data mining and text mining are coming together as a single discipline. When you have structured and unstructured sources of information or you have unstructured information from new sources like social networks and Twitter, Facebook, and blogs, it's critically important to bring it together into your data mining environment. A key convergence also is that data at rest and data in motion are converging, and so a lot of this will be real-time event processing.

Those are the key convergence and collision avenues that we are looking at going forward.

Gardner: Very good. We've been discussing how new architectures for data and logic processing are ushering in this game-changing era of advanced analytics. We've been joined by Jim Kobielus, senior analyst at Forrester Research. Thanks so much, Jim.

Kobielus: No problem. I enjoyed it.

Gardner: Also, we have been talking with Sharmila Mulligan, executive vice president of marketing at Aster Data. Thank you Sharmila.

Mulligan: Thanks so much, Dana.

Gardner: This is Dana Gardner, principal analyst at Interarbor Solutions. You've been listening to a sponsored BriefingsDirect podcast. Thanks for listening, and come back next time.

Listen to the podcast. Find it on iTunes/iPod and Podcast.com. Download the transcript. Learn more. Sponsor: Aster Data Systems.

Transcript of a BriefingsDirect podcast on how new advances in collocating applications with data architecturally provides analytics performance breakthroughs. Copyright Interarbor Solutions, LLC, 2005-2010. All rights reserved.

You may also be interested in:

• Aster Data archietcts application logic with data for speeded-up analytics processing en masse
  
  
• Aster targets mid-market with budget-conscious, massively parallel data warehousing appliance
  
  
• A technical look at how parallel processing brings vast new capabilities to large-scale data analysis