Big News on Big Data?

What has kept NonStop entrenched in our data centers for many years – its unique attributes when it comes to running mission-critical transaction processing – ensures that it will not be overlooked when it comes to building out Big Data frameworks!

Having the RV as our “command center” when we travel is giving us new freedoms when it comes to being able to be flexible with our travel plans. Any time we need to head back to the west coast, there’s now few issues preventing us from stopping by good friends at HP and to catch up with former colleagues from our times at Tandem Computers. With the recent move to new facilities, within the HP campus at Palo Alto, we were able to enjoy an unexpected pleasure – visiting for the first time the well-maintained original offices of both Mr. Hewlett and Mr. Packard, and the photo above was taken of us alongside the desk of Mr. Hewlett.

Big occasion and a big highlight! The street and city were one of the first US addresses that I came across – anyone who worked with IBM mainframes back in the 1970s should recall that manuals came out of an IBM distribution center with the address of Page Mill Rd, Palo Alto, and when the opportunity came for a drive down from Edmonton, Alberta, to California I went out of my way to find this street and to drive past the IBM building which, if I recall correctly, was on the opposite side of the road to where HP now hosts its current campus.

The more things change the more they stay the same – yes, all those decades ago I didn’t flinch at an 8,000+ mile road trip – my route from Edmonton to the Bay area was via San Diego and Dallas / Fort Worth. Big occasions, big trips, and of course a visit to the Big D itself! For someone from the Big Country, what else would you have expected? Whenever we label anything as being big, it goes without further explanation that it is out of the ordinary, sometimes even beyond superlatives, and possibly so immense that its significance just cannot be overlooked. It’s important.

All of this is a lead to the topic that has occupied much of my time over the past couple of weeks. Big Data. In the time I have been looking at what it means for the NonStop community – and yes, there are still those that question the relevance and likely impact on NonStop – I can’t escape the feeling that this holds great potential for even greater NonStop visibility and I cannot dismiss the intentions of those advocating for the inclusion of NonStop in any plans for Big Data. It’s all about the need to include transactional data, of course, and the developing acceptance that simply capturing unstructured data without any connection to what was actually going on at the time, leaves a lot on the table in terms of understanding customers. Knowing they purchased a jacket, based on a discount coupon delivered on a smartphone, holds a lot more significance if you know that the purchase transaction occurred within half an hour of the coupon appearing on your smartphone than knowing only the date.

In my last three “Musings on NonStop!” features, published in the Tandemworld.net eNewsletter, I have focused on Big Data starting with the observation in September of how there is a lot going on for the NonStop community when it comes to Clouds, but there’s also a lot happening when it comes to Big Data – it’s how companies will be able to better compete in the marketplace and to provide business unit managers with even greater insight into all that is happening within specific market segments. I followed this up in the October musings with the observation of how industry experts seem only too keen to write-off the contribution being made by NonStop, even when they reference transaction processing. And yet, when you look at the data, it’s hard to overlook the presence of NonStop. Yes, I wrote rather tongue-in-cheek, critics of NonStop seem to be fond of promoting other platforms directly interfacing with customers even when their tolerance of failure seems pretty weak.

In the November musings I again referenced the work I had been doing for Attunity, in addition to that for IR. And they aren’t my only clients with an interest in Big Data – of late I have become associated with a start-up developing new ways to integrate data from many platforms to provide a better mix of relevant data for analytics programs and they are starting with the transactional data coming from NonStop. Surprised? I was, but as I talked to them, it made a lot of sense. The intersection of data generated by mission-critical transactional applications with the data coming out of emails, social media networking and other sources of unstructured data just makes sense. And if you think the Tandemworld.net eNewsletter is just a collage of vender promotions, you may want to take another look as there are interesting commentaries scattered throughout this publication well worth checking out.


But it is the work Attunity has been doing that has kept me focused on Big Data. Even as my previous opinion paper on Big Data became available as a download at http://resources.attunity.com/HP-NonStop-transactional-data-important-ingredient-for-big-data-success I have complete a follow-on opinion paper that hopefully you will be able to download before year end. Whereas the central theme of the now-available paper looks at the challenges facing companies, as the landscape of servers that typically populate a data center continues to include a diverse mix of heterogeneous systems, the theme of the just-completed paper looks at the impact of an increasingly mobile client population, initiating transactions at any time, will have no plans to ensure transactional data is a part of Big Data. Moving from “what do you know” to “you know too much” to “yes, you can take it with you” provided an interesting backdrop for the material provided in these two opinion papers.

“We’re seeing Big Data being managed and processed in three types of Big Data stores,” Attunity’s Ankorion explained during the recent interview and covered in the previous opinion paper – again, follow the link above. “These include large-scale Data Warehouses that have been available for some time, cloud storage as we have today from vendors such as Amazon Web Services, and Apache Software Foundation’s Hadoop Distributed File System (HDFS), the primary storage system used by Hadoop applications. “For the NonStop community, the good news is that we have been actively engaged with HP’s Advanced Technology Center (ATC),” added Ankorion, “and we have already successfully demonstrated Attunity pulling data stored on NonStop including SQL and loading it to large DW such as HP’s Vertica - perhaps the very first time NonStop’s structured transactional data has been integrated with the essentially batch mix of structured and unstructured data that is in Vertica’s Big Data store.”

Equally as important was the comment posted by Justin Simonds following my initial post on Big Data of September 21, 2012, “Big Data? NonStop? It’s common knowledge …” In that comment, Justin wrote “good EDW's and ODS's will continue to provide structured analytics around transactional data, of which most is useful. Hadoop and others will attempt to provide insights around social media data, most of which is not useful. Blending the two will yield better insights provided we get the proper filtering on the unstructured/social/sentiment pieces,” said Simonds. And on this point he is absolutely correct as all is said and done, there’s still no substitute for the smarts we can bring when the two worlds intersect as they are bound to do. Justin then makes the case for the Analytic Cloud where “Queries would enter the analytic cloud and either be responded to by the intelligent query router or direct the query to the EDW, ODS, Big Data analytic engine or unstructured analysis (Hadoop/Autonomy). Submitters would not know or care which backend system responds to the query." All pointing to a potential role for NonStop in the future that I am not prepared to dismiss out of sight, given the nature of recent conversations.

This then leads me to the Big Conclusion – there will be a lot of data moving around in the near term. Yes, there will come a time when we can better integrate systems, perhaps within a virtualized realm, but until then, we will be moving data. And with that, the investments being made by NonStop partners, like Attunity, hold my interest as I know they do the interest of others (including HP I suspect), and that shouldn’t be Big News for anyone who has been following the posts to this blog.