Skip to main content

Is 30 minutes too long?

I am slowly adjusting to a new place of abode. It's a long story and not something relevant to this posting - perhaps later. However, as anyone who has moved to a new location, getting set-up as fast as you can becomes a necessity, and minimizing disruptions to daily routines, a priority. For me, the urgent needs all centered around getting our audio / video equipment all sorted out and this meant going to the mall. Pretty routine, I thought - nothing much could go wrong with a task this trivial!

Well, I experienced a crashed system first hand this weekend. My wife and I were at Best Buy purchasing DVDs when, after a wait in the check out line, the cash register simply crashed. I knew immediately it wasn't good when the cash register screen went black and I could follow the flow of NETBios data wrapping around on the screen. Eventually, the ubiquitous Windows icon appeared. But too late - the check-out clerk left his post for guidance and then took us to another line to stand and wait.

Our transaction was pretty routine, but did have a few twists. We had a discount coupon and we were paying by credit card. The coupon had been accepted, and the transaction amount approved. But at the time the cash register crashed - the transaction had completed its last step - printing the receipt. Where were we, process wise?

After standing in another line - this one moving a lot slower, of course, as the activity being performed wasn't routine for any of the participants - we were finally in front of another service assistant. We went through the story, physically pointed to the cash register and the check-out clerk where the crash had happened, and began the process again. We had to get the coupon reinstated and check their server logs (which they did provide us with) to make sure there was no duplicate credit card entries, before we were comfortable enough to proceed. By this time, 30 minutes had passed.

While the service assistant took an additional amount off the transaction - at no time did management approach us to say sorry for the inconvenience. Best Buy is a very successful operation these days - putting a lot of pressure on other operators like Circuit City - but to have customer-facing systems that are not reliable and where their operators simply dragged you off to another line to start over, isn't good for business.

Later that same day, we stepped into a branch of the Wells Fargo bank to update our profile information, add online access, and get a local ATM card so I wouldn't have to keep paying other parties ATM fees. All pretty straight forward. But as soon as we walked through the doors, we could see it was busy and that all the service staff were tied up. This time, however, the local manager saw us and came over to greet us. She offered us coffee and made the other staff aware of our presence. Yes, we had to wait - but we were nowhere near as agitated as we had just moments before.

When we sat down with a bank representative we learnt that all hadn't been well for the bank earlier in the week. They had taken a huge outage - with some elements of the network out of action for more than a day. I was stunned. Yes, I had been traveling during that time, and had missed most of the business news. He informed me that the mainframe failed and that it had taken a while to sort out. He then continued to walk me through all the steps I had to take, and was open and informative throughout the process. As we walked out of the bank - I checked and yes, the whole process took 30 minutes. But what a different 30 minutes.

In this day and age, I have little patience for any retailer or financial institution that skips on their infrastructure investments. And into my broad definition of infrastructure I include all the staff and management working with it - and all the folks involved with their education. Systems are failing a lot more than I had previously considered. Availability, and the need to be available through planned and unplanned outages, wasn't just the mantra of a select group of architects. The impact from a failure becomes obvious very quickly - whether it's a simple PC-based cash register or a whole mainframe-supported network.

But what a visible difference in professionalism between two corporations, separated by only a few feet. It boils down to the old fashioned customer service. Be 24 X 7 or invest more heavily in your human infrastructure.

A short while back I met with Tom Moylan - Tom manages the America's sales organization within HP that is responsible for selling NonStop servers. Right now Tom is as excited to be selling NonStop servers as he has ever been - and for good reason. A number of retailers and card processors had been considering alternatives to the NonStop server product line - but over the past few months, a number of the very biggest have come down decidely in support of NonStop.

There have been projects to move off of NonStop that were well under way; when the end users were polled, companies reversed course and reinvested in NonStop. There were internal competitions and "cook offs" between different cluster-based systems and when the finished product was presented, the winner was NonStop. Even when management had been cautiously supportive of NonStop but had let projects drag, there is now renewed enthusiasm for the platform.

If I was working up in one of the border states that loves ice hockey, then I would know who the big retailer was. If I knew much about card processors - credit or debit, it doesn't matter - then I wouldn't have to think too long or hard to figure it out. They have gone big-time with NonStop.

In the coming months I am going to revisit the topic of services, and of the impact they are having on the way we interface with our applications. In particular, I am very interested in the topic of Services-Oriented Architecture (SOA) and the impact that is having on the roll-out of new applications. The deployment of this architecture will play a very important role in the continued viability of NonStop, and I have already begun to see inroads being made into the NonStop customer base. This is an extremely encouraging development and bodes well for the continuing relevance of NonStop!

The number of corporations now very strongly recommitting to NonStop is just great to see, and those that are not had better invest in the customer service. Waiting 30 minutes is not that unpleasant if you are treated right - but being down and having to treat customers with lots of human interaction is a costly gamble on the part of any corporation.

Next time I see Tom Moylan I am pretty sure it will be my turn to buy lunch.

Comments

Anonymous said…
You do't have to wait to buy Tom lunch, You can always buy lunch for me :-)

Interesting experiences, and unfortunately I think that the first is becoming more common as shortcuts are being taken to get new systems into the market quickly.

These are stories that need to get put out into the marketplace. However corporations for obvious reasons don't want to publish these incidents. Though they could be benefit many corporations in knowing what to avoid and how to provide good ervice.
Sami Akbay said…
Richard, if you make your calendar publicly available, I'll make sure to avoid the places that you shop at -- I think you have bad infrastructure karma :)

I think making systems continuously available is one side of the coin but making systems available with some intelligence behind is equally important. Home Depot that's a mile from my home has self checkout (sometimes the only option). The other day, the sku for the item I was purchasing wasn't in the system and I spent 30 minutes trying to buy replacement parts for my weber grill (I could cook you and Tom steaks next time you're in Palo Alto area) and I was really close to vandalizing the .... self-checkout equipment.

Real-time infrastructure requires real-time availability (and continuous access) but without meaningful data and intelligence behind it, it is real-time frustration.
Kelvin Anderson said…
30 Minutes is far too long, In my opinion. I would not have waited, and I have not waited anything more than 10 minutes coupled with a failed front-end system. I would have departed the store minus the merchandise and considered other options for purchase. This only serves outlets like Amazon.com in terms of reinforcing a customer base. Why endure the hassle when you can do it online and most likely entrust a portion of your transaction to a Tandem? Yes yes yes I said Tandem. Better to wait at home than stand in line with other grumpy people only getting redder in the face. -K
Richard Buckle said…
This comment has been removed by the author.
Richard Buckle said…
RT Writer said...
I have had an email exchange with Bill Highleyman and I thought it would be good to post here one of his comments:

Richard, I like your comments on "Is 30 minutes enough?" My series of Never Again articles focuses on just this issue - massive downtime due to stupid errors.
You might be interested in my August article in which I described eight major outages experienced by large enterprises in just the first six months of this year (http://www.availabilitydigest.com/public_articles/0208/you_think_your_system_is_robust.pdf).
Also, the September article talks about a major hosting service that went down for days, taking with it hundreds of online stores (http://www.availabilitydigest.com/public_articles/0209/hostway.pdf).

Bill has now been publishing a newsletter "The Availability Digest" for some time - and it is well worth reading as he tracks some of these outages in more detail than I can cover in this blog. Bill can be reached at - Bill Highleyman [billh@sombers.com]

September 17, 2007 3:58 PM

Popular posts from this blog

If it’s June then it’s time for HPE Discover 2021.

  For the NonStop community there has always been an annual event that proved hard to resist; with changing times these events are virtual – but can we anticipate change down the road? Just recently Margo and I chose to return home via US Highway 129. It may not ring any bells, but for those who prefer to call it the Tail of the Dragon – 318 curves in 11 miles – it represents the epitome of mountain excitement. For Margo and me, having now driven the tail in both directions, driving hard through all these turns never gets old. Business took us to Florida for an extended week of meetings that were mostly conversations. Not everything went to plan and we didn’t get to see some folks, but just to have an opportunity to hit the road and meet in person certainly made the 4,500 miles excursion worthwhile. The mere fact that we made touring in a roadster work for us and we were comfortable in doing so, well, that was a real trick with a car better suited to day trips. This is all just a p

Three more wishes coming soon – the path ahead for NonStop.

So, another three years have passed by and I find myself writing a preview of what I will likely focus on in eighteen months’ time – my next three wishes for NonStop! It wouldn’t be fair on my family if I said 2019 had been a routine year for Pyalla Technologies. It started with the return flight from Sydney, Australia, and continued with three separate trips to Europe plus a lengthy road trip to Las Vegas for HPE Discover 2019 combined with stops in southern California and participation in N2TUG back in Texas. The miles have added up but all the while even as the adventurous life continued to unfold, there was so much news coming out of HPE that scarcely a day passed without a discussion or two over what it all means. Margo and I have our roots firmly anchored in NonStop, dating back to Tandem Computers where Margo had risen through the development organization all the way to the COO role under the stewardship of Bill Heil when Bill headed the NonStop Software BU. As for me

ACI Strategy - it's all about choice!

I have just returned from spending a few days in Omaha attending the annual ACE Focus meeting. These two day meetings provide more in-depth technical coverage than is usually found at the regular ACI user events, and ACI customers have been coming for more than a decade to hear the messages directly from company executives. The picture I have included here is of the venue of the Wednesday night social event – a reception held at a local sports bar called the ICEHOUSE. And I found this extremely ironic as my own involvement with ACI came through my association with the ICE product. For most of the ‘90s, ACI had been the global distributor for ICE and then, as we began the new millennium, ACI purchased Insession, creating a separate business unit that it named Insession Technologies. For nearly six years, as part of ACI it enjoyed a successful partnership with the NonStop community and had provided a number of solutions in communications, web services, and security. But the decision in l