Friday, September 21, 2012

Big Data? NonStop? It’s common knowledge …

What has characterized NonStop for so many years – its underlying highly-scalable Massively Parallel Processing (MPP) “engine” - is the envy of all who pursue Big Data. Could we see it being leveraged in the future?

It is not uncommon for Margo and me to head up into the mountains at this time of year to catch the colors of fall. Furthermore it’s not out of ordinary for us to do so in an open roadster to better see it all and to take in the pleasant scents that accompany the change. And on the occasions when we elect to look at the fall colors to our south, we usually end up stopping for lunch in Aspen – perhaps the epicenter when it comes to checking out the aspens. The picture above was taken during our most recent trip and it shows just how colorful the landscape can become as the leaves change color.

When it comes to technology and to finding common ground among reporters, analysts, and all those prepared to offer opinions there’s been more in common of late than I can recall having seen in a long time. Yes, Apple continues to dominate (surely, their financial performance isn’t truly influencing America’s GDP, but it seems that it will improve it by a couple of decimal points), and with Apples ascendency, there’s a consensus that mobility and interfacing with tablets and smartphones will dominate the agendas of many CIOs. Always on theses CIOs agendas is security and audibility – abiding by the governance mandates overarching all that business does today seems inescapable.

However, there’s even more common ground when it comes to the other two big-ticket items that nearly all who are prepared to discuss IT rarely stray too far from – Clouds and Big Data. Even as I suspect there’s a strong umbilical cord developing between the two, as surely IT cost-effectively addressing Big Data will turn to Clouds. It’s common knowledge within the NonStop community that I volunteered to become the leader of the HP Business Critical Systems (BCS) Cloud Special Interest Group (SIG), that I have set up a LinkedIn group simply called “Cloud SIG – A Connect Community”, and that it complements the LinkedIn group I had previously created called “Clouds, powered by NonStop”. This, together with the LinkedIn group “Fools for NonStop” pretty much sum up my enthusiasm for NonStop and for its potential to play an important role in Clouds for many within the NonStop community.

Perhaps what is now quite as well known is my interest in Big Clouds, a byproduct essentially of earlier work I had done for vendors like GoldenGate (now Oracle GoldenGate) and for the SQL/MX team within HP NonStop. As I was preparing for this post I was looking for the right words to introduce Big Data when I came upon another vendor’s introduction. “Imagine a world where everyone gets precisely the right information at the right time—to make important business decisions faster; to expose what’s relevant in a critical litigation matter; to know what’s no longer needed or just clouding the issue and raising costs; to be able to respond confidently to any regulatory or customer inquiry; to provide a better product or service than the competition”, was how they positioned themselves before adding the obvious “that’s the world we believe in.”

It is with this in mind, making important business decisions faster and providing a better product or service than the competition, I wrote two opinion papers for vendors well known to the NonStop community – Attunity and Integrated Research (IR). For the immediate future, moving data from NonStop to Big Data platforms seems an obvious direction to take, just as determining that with even more to monitor, the data accumulating may be best pushed out to a Big Data store. The paper for Attunity, “HP NonStop Transactional Data: An important ingredient for Big Data” will be published shortly whereas the paper for IR, “Big Changes coming as monitoring turns to Big Data!” has just been published and can be downloaded from

In writing these different opinion papers only twice was their material in common and this had to do with observations made by HP executives. In the earliest references to HP, in both papers, I quoted what has already appeared in the August 29, 2012 post to this blog, “NonStop? Radical transformation!”, as I repeated what HP’s Senior VP and General Manager, Business Critical Systems (BCS), Martin Fink, had observed. “Another complicated vector thrown into all of this … and where NonStop plays – how we leverage NonStop expertise – is Big Data,” Fink said before adding “we have been running MPP engines that are some of the biggest the world has ever seen. There’s no better engine than NonStop!”

In later references to HP, I pulled from a HP News announcement about the HP and Cloudera partnership, where I had quoted HP’s VP, Converged Application Systems, Enterprise Group, Paul Miller. “Companies are increasingly turning to Apache Hadoop to help manage petabytes of information, but are finding they spend more time on operations and maintenance than on strategic data analysis,” Miller had said. “HP Converged Infrastructure provides a strong portfolio of comprehensive Big Data management tools that when combined with Cloudera’s solutions bring the benefits of Hadoop to clients faster without the administrative burden."

In the weeks that followed the post to this blog, as well as the writing of the opinion papers, I had the opportunity to talk to Miller and expand on the comments above. What was interesting to me, of course, was the potential role of NonStop – already in the paper for Attunity I had looked at how there would be some in the NonStop community replicating data from NonStop to Big Data platforms like Hadoop – as it seems unlikely that NonStop would provide an economical platform on which to run Big Data. Just how “converged” would be the Converged Application Systems when it came to NonStop?

“Today we have a mix of ‘systems of record’ (including ERP) as well as ‘systems of engagement’ (including web access apps), in addition we have customers who are looking at popular offerings in support of Big Data initiatives (including Apache’s Hadoop) as part of systems of engagement,” Miller observed.  He then added that “by marrying NonStop systems with Big Data systems, customers can leverage their existing data with new sources of data to provides advanced analytics to the lines of business.”

Converging? Marrying? This echoed similar sentiments as had been previously expressed by HP NED’s product manager, SQL, Ajaya Gummadi, who during our last conversation has said, “Big Data? It’s not going to be as much about storing and moving data as it is a matter of how we connect the dots between transactional data and social media feeds. There will be occasions where organizations will launch a new product or service, check the transaction volumes of the last five minutes and then want to correlate this activity with what was being said about the product in social media channels including blogs, tweets, ‘Facebook likes’ etc. both positive and negative in order to make rapid adjustments to the product or service as well as the marketing messages being used. This type of capability needs a platform that can handle extreme volumes of workloads and be always available to do business.”

“There’s no better engine than NonStop”, according to Fink. “Marrying NonStop systems with Big Data systems”, added Miller.  “Connect the dots between transactional data and social media feeds”, proposes Gummadi. When it comes to Big Data then, for those who may not have considered the matter too deeply, there appears lots of potential opportunities for NonStop as it harbors the transactional data, critical to making the types of business decisions projected as necessary in the future, and all on a rock-solid platform without the frailty of what today serves meeting the needs of Big Data.

“Furthermore, while there are many applications where organizations require high orders of governance, back-up, archival access, etc. there are also customers who do not require this level of rigidity and this is where the system of record continues to be important for organizations”, Miller then advised me. “However, there is great value to customers who co-mingle data between the two types of systems but retain the system of record which is rock-solid, reliable, whereas the system of engagement is typically less so and the rules are looser.”

“When it comes to a role for NonStop within Converged Application Systems, it is immersed in it – not tangential at all,” Miller then summed it up for me. “HP Integrity NonStop is built on the same core Converged Infrastructure building blocks of blades, fabrics and infrastructure management as our other Integrity and ProLiant systems. But then each is purpose-built to meet specific use-cases from mission critical to web surfing. Organizations benefit from a common architectural approach that can scale across all their needs and operating environments from Windows, Linux, UNIX to the most important applications in the world on NonStop.”

There’s a common thread through all of this that is probably not lost on many within the NonStop community. In its early phases, few organizations will even be considering the NonStop, but as the time when information is needed   shortens and heads to being real time, what is held on the NonStop transactional systems will become more important. Moving data will serve the purpose for quite some time, however in subsequent phases as the technologists and architects look further out, moving data may need to be complemented with a broader Big Data framework that embraces the NonStop as it stands – new protocols and new services coming to the fore as part of such a broadening of the Big Data framework. Finally, in an even later phase the intellectual property of NonStop, as we see it today, easily capable of handling enormous MPP configurations, will most likely be leveraged in new and highly innovative ways.

We may not immediately appreciate what we see, but even as discussions within the NonStop community center on how today NonStop has become a software solution,  perhaps sometime soon we will find it even less surprising to find HP leveraging NonStop in such a way that ‘NonStop Inside’ may become widespread. And that could pave the way for commonality way beyond what we may have thought possible in the past. Indeed, what NonStop mastered nearly four decades ago, may turn out to be what’s truly Big!


Justin said...

Richard I do believe NonStop is a player in Big Data just as Paul, Martin and Ajaya have indicated. As you know "Big Data" is the latest buzzword and vendors are focusing on how they fit, however structured data, which has provided us with insights for over 2 decades will not be scraficed to Big Data but will compliment it. Good EDW's and ODS's will continue to provide structured analytics around transactional data, of which most is useful. Hadoop and others will attempt to provide insights around social media data, most of which is not useful. Blending the two will yield better insights provided we get the proper filtering on the unstructured/social/sentiment pieces. TC Janes and I recently spoke at the XLDB (eXtremely Large DataBase conference) on exactly this topic. Here was our abstract for the talk "Although most discussion today revolve around Big Data, Social Media and unstructured data most of the world is still using and will continue to use structured analytics - Enterprise Data Warehouse and Operational Data Stores. The Analytic Cloud is a discussion on incorporating all these components by having an intelligent query router that would provide several functions. First to determine where queries should be routed (that is which platform is best suited to receive and process the query). Second to provide real-time datastream analysis for urgent analytics. Finally to provide an in-memory database for redundant queries (for speed). Queries would enter the analytic cloud and either be responded to by the intelligent query router or direct the query to the EDW, ODS, Big Data analytic engine or unstructured analysis (Hadoop/Autonomy). Submitters would not know or care which backend system responds to the query." TC and I can provide more info on the "Analytic Cloud" concept if there is interest.

Richard Buckle said...

Sure am interested Justin as I assume others will be as well ... its something we discuss among ourselves within Infrasoft a lot more these days ... will catch up when I return from GTUG / Dresden, Germany