Critical infrastructure has always been a priority for society but when it comes to technology critical infrastructure, October 1987 and Black Friday remains a lesson on just how well NonStop can scale even in the darkest of hours.
The completion of new infrastructure projects is always
an exciting time. When it comes to linking two estranged parts of Sydney, it
meant life could blossom on its North Shore. History will recall that Jack
Lang, Premier of the state of New South Wales, missed his big opportunity to
cut the tape and to be officially part of the legacy of Sydney’s most iconic
landmark as he was upstaged by a sword-wielding protestor on horseback, Captain
de Groot who charged the tape ahead of Lang.
Given the one-off occasion this happened to be, there was no chance at all that history would ever be repeated. With this in mind, I had an occasion this past week to
recall the music of New Zealand’s well-recognized rock group, Split Enz and, in
particular, the lyrics from one of the better known songs -
History never repeats
I tell myself before I go to sleep
Followed by -
Better to jump than hesitate
Indeed, as we all know, history is often repeated but
even so, when it comes to technology, how often are we left to think more on
whether it is better to jump than hesitate. As we get deeper into 2021 there is
so much that is changing that we are more often than not left to wonder whether
history has any relevance at all and whether what worked in the past can
contribute to the future.
We often write about legacy and about how the moment
any application goes live, it becomes legacy. Likewise, much the same can be
said about modernization projects; as soon as they enter production, they are
no longer modern as something more modern has intervened in the time it takes
to incorporate the latest trend. When it comes to history it isn’t hard to
imagine that what transpired a second ago becomes, in that instant, history and
part of a much bigger story.
For NonStop
there is much that has taken place over the four plus decades NonStop systems
have been around. However, amongst the most often referenced is how early
Tandem Computers kept on running, processing transactions, and of course the
day US stock exchanges experienced its infamous October 1987 “Black Monday.”
Even as the
ticker tapes fell behind and processing orders submitted on what can only be
considered as punch cards, the exchanges kept their NonStop systems running by
going to extraordinary lengths to do so. Perhaps at no other time in the
history of IT did scalability have such a meaningful impact on the operations
of such a key piece of financial infrastructure. It may have taken place thirty
four years ago but it’s a real world occurrence that we shouldn’t forget.
If you missed
reading the July – August 2020 issue of The Connection magazine, you would have
missed reading HPE Master Technologist, Justin Simonds’ column, NonStop Trends & Wins where he wrote about Black Monday -
“It
was Monday October 19, 1987, the DJIA (Dow Jones Industrial Average) fell 508
points (22.6%), accompanied by crashes in the futures
exchanges and options markets. This was one of the largest
one-day percentage drops in the history of the DJIA.
“In
1987 Tandem had systems in the stock exchanges and the volume that day was so
great that it appeared the systems couldn’t handle it. The New York
Tandem office shutdown and moved their office systems to the NYSE and added
capacity online during the trading day circumventing an outage. It was
one of the legendary stories if you worked for Tandem.”
Over the past
week both Simonds and I have turned to social media groups to see if there were
those present at the exchanges during Black Monday who could throw additional
light on what took place. Turns out that Tandem engineers and support personnel
went to extraordinary lengths to ensure exchanges kept on processing. Heroic certainly
and definitely legendary as far as the NonStop community is concerned!
Some of the
names you will see referenced here were long-time Tandem Computers and later, HPE
NonStop, employees. However, first a little background as Black Monday happened
while the Tandem community was meeting in New Orleans for their annual ITUG
Summit.
As posted on
Facebook, Frank Sheeman recalls –
“I
was at ITUG in a general meeting when a lot of pagers went off and about half
the room left for the phones. Story was that NYSE was very close to running out
of disk space for log files when 4pm came.
“Major
upgrades (happened) overnight; done by stripping the local office hardware. We
were one of few to survive the 600 million share day the next morning we were
ready to handle a billion. Fortunately we did not have to find out as trading
was lighter.
“But
the NYSE was Very pleased and stayed a fiercely loyal customer for a long time.
Tandem surviving the 600 million share day was impressive but the overnight
upgrade was more impressive. Nobody cared that it wasn’t needed. They cared
that it happened.”
In an update provided to TandemAlumni.groups, Bill Karpowic
wrote that –
“Actually
I do remember that day and night.
“I was on site at NASDAQ not in New York but I remember the NY folks
reconfiguring the systems so they were no longer fault tolerant and hardware
was being flown in from Reston, I believe. Folks from the Connecticut office
drove down to help.
“Both
exchange support teams work through the night to be ready for Tuesday. Systems
at NASDAQ had processors added from other systems that were not directly
connected to trading.”
According to Bill, these were two of the longest days he
can remember. It was from another Bill, this time Bill Honaker, who also posted
to the TandemAlumni.groups, of how he heard the news -
“I
was also at the ITUG conference at the Superdome in New Orleans. The ‘exhibit
hall’ was a cramped room with a low ceiling, and as I heard about it I was
standing next to some SIAC employees who were attending.
“But
I do remember they were planning how to install the surplus hardware that was
being acquired. This was probably around lunchtime in NYC.”
One year after the event,
Computerworld ran an article in its October 17, 1988 issue, under the general
heading of Wall Street Rebuilds -
“Indeed, most
exchange computers did have trouble handling the volume, The NYSE’s SIAC system
comprising 200 Tandem CPUs, lost two processors under the strain – but the
fault tolerant system kept going.”
It was only a short time later that having come through
Black Monday, SIAC went ahead and migrated from TXP processors to VLX boxes
with increased disk storage for its Limit System and upgraded the memory and
disk capacity of Common Message Switch.
Henry Ford set about modernizing transportation and as
he did, Ford didn’t shy away from being controversial. The most quoted
observation by Ford falls into this categorization even as it defies
contemporary logic -
“History is
more or less bunk. It is tradition. We don’t want tradition. We want to live in
the present and the only history that is worth a tinker’s dam is the history we
make today.
“We want to get away from that and take care of today. We’ve done too much
looking back. What we want to do and do it quick is to make just history right
now.”
Have we done too much looking back? It can only be said
that those extraordinary efforts to keep major applications running during
Black Monday were greatly appreciated by SIAC and that the demonstrable success
in Tandem’s ability to scale-out as dramatically as it did, warranted the
gratitude of SIAC for many years to follow. Yes, they were fiercely loyal for a
long time. More importantly, it certainly helped cement the tradition within
the NonStop community to always put the customer first.
Urban legend? Not even close! The ability to scale out
processors and memory as well as storage saved the day for SIAC. It should come
as no surprise then to read of how seriously HPE values scalability even today
and how NonStop still exhibits industry-leading scale-out properties.
I am sure there are more stories that can be told about
Black Monday, the team at Tandem and SIAC. However, even as we look back at how the
history books recall those times, it’s perhaps even more encouraging to know
that yes, history does repeat and it does so almost daily for enterprises that
continue to rely on NonStop.
Comments