Sunday, May 18, 2008

Virtualization? A Testing Time!

I have just departed Monte Carlo after a longer then planned stay. Yes, I managed to get done the things I had come to do, but anyone who has been to Monte Carlo in May can attest to, just getting around the city can be like walking through an ancient maze. In a little over a weeks time, there will be the running of the Monte Carlo Grand Prix, and already, the stands are being erected, streets closed, and ten foot high fencing is already in place blocking many of the footpaths. The picture I have included here is of me sitting trackside as the work went on around me.

The Monte Carlo Grand Prix is one of the few Formula One (F1) races on a pure street track where the cars race through the streets of Monte Carlo following a circuit laid out more than fifty years ago. There have been only a small number of changes – the chicane after the tunnel exit being one – and while it makes little sense these days to torture the cars the way this course does, it’s tradition!

And part of the tradition is the creation of a city within a city. From the dockside quays on up through to Casino Square, the area surrounding the streets making up the track undergo a radical transformation as the “F1 circus” comes to town. For all other events, this is hidden from the spectators and other city residents, but in Monte Carlo, its citizens witness first hand the hosting of a completely foreign world. And the incongruity of it all - with the newest of F1 cars racing alongside some of the Mediterranean’s oldest structures – continues to amaze me.

I came to Monte Carlo by sailing ship. I had the option to fly or to catch the train, but in the end elected to sail. I was fortunate enough to catch a commercial yacht on a “repositioning voyage” as it was being moved from the Caribbean to Istanbul for a season of sailing through the Turkish and Greek Islands. I had sailed on this vessel a few times and was pretty familiar with its operations, and one reason I liked it was that it maintained an open bridge where passengers were welcomed at all times of the day.

Being this familiar with the yacht, I was somewhat surprised to see the tug come to help us out of the port in Barcelona. The vessel’s bow-thruster, a separate propeller system that allows the yacht to move sideways, wasn’t being used. The yacht had been in dry dock immediately following the Atlantic crossing and for the week it was there, dock engineers had overhauled the hydraulics used to drive the propeller. All the testing they had done but one week later, the whole system had failed. For the next eighteen months, the yacht was going to have to pay additional port fees for the use of a tug.

It wasn’t the only surprise in stall for us – as we sailed through islands just south of Marseilles, the yacht began to gyrate severely between headings. One minute the vessel was heading out to sea while only minutes later it was headed back to land. The changes in direction were being executed more rapidly than I could recall having ever witnessed before. With the open bridge policy, I walked onto the bridge to watch the captain giving orders to the helmsman to swing from 20 degrees to port back to mid-ships, and then across to 20 degrees to starboard. For a vessel a little over 400 ft, this was giving everyone a pretty interesting ride.

The captain explained to me that during the Atlantic crossing, he had noticed that one of the two propeller shafts wasn’t “working as hard” when close to maximum throttle settings, particularly when there was a heavy sea and the swells were sizeable. So, as he had time to spare and it was a rather sheltered sea lane, he was simulating the swells to gather data that could help him understand what was happening to the one propeller with the theory being that perhaps the propeller folding mechanism was allowing the propeller to partially and temporarily fold and not working as well.After a few hours the yacht came back on course and proceeded as planned to Monte Carlo.

On the afternoon of our departure, with the tug alongside and the pilot onboard, the yacht had just backed out of the harbor at Monte Carlo – and yes, to those familiar with the harbor, maneuvering a 400 ft yacht within the inner harbor was a lot of fun – all the yachts generators failed as one. Nothing! Not a light, not a refrigerator, not a toilet (marine toilets require power).

Test after test failed – and we had come to an end only a few boat-lengths from the harbor entry. While the crew was frantically trying to localize the problem, the authorities towed us a short distance from the entry, but wouldn’t allow a “disabled vessel” to move out of their control. Essentially, we remained under the supervision of the Monaco port authority. The problem? As all generators failed as a unit – it had to be the fuel delivery system and yes, they had just taken on a new fuel load in port. The fuel was the same octane and to the same specs, but as the old and new fuel worked their way into the fuel lines, it turned to a jellified mess and blocked the filters.

Coming as this did after the failure of the bow thruster, the unusual behavior of one of the propeller-shafts, to some of us it was rather amusing. We would spend another night in Monte Carlo – onboard, unfortunately, but enjoying an outside BBQ with the skyline of Monte Carlo behind us, wasn’t all that bad.

From my experience working with Container ships, I know that you don’t mix new fuel with old, with separate tanks for each, and that there’s a highly complex method of delivery to ensure the stability of the vessel is not put at risk. You don’t simply empty one tank and move on to the next and I assumed the same would apply for the yacht I was now on While they had tested the fuel on delivery, and had run additional tests of old versus new, they were quite unprepared for the fall out from mixing the two. They had not been able to simulate in their onboard lab what they encountered as they left the harbor.

While all of this was happening, I took a call from one of the sales folks at GoldenGate concerned over our ability to support an environment she wasn’t familiar with. The prospect was very anxious to make sure we had been certified in a cluster environment that turned about to be another virtualization deployment. The concern was whether we knew for sure we would operate with the host operating system and data base management system. We knew that we supported this combination at other customer sites, but could we be absolutely sure we would not be impacted by the virtualization software?

Just as Monte Carlo was watching a virtual city unwrap on its streets, and just as the yachts captain was looking at the results of lad simulations, it really did occur to me that we are going to see a lot of changes coming with the increased popularity of virtualization software. And I don’t think NonStop users will be immune to this, or be left completely unaffected. To some extent, as we move to a bladed architecture where the number of processors will depend on the number of sockets on the blade and the number of cores in the Intel chip used, some form of virtualization will be encountered immediately – the blade will appear to NonStop as just a single CPU, or processor. The level of granularity for fail-over will be at the blade level.

But there’s much more than this coming, I have to believe. As blades are “pooled” in support of a hypervisor, there’s every potential that we will see NonStop hosted in a virtual machine – and how will we test that environment? How will we simulate fail over in a completely virtual world. And will it be worth it? Will we want to see a return to simpler configurations just for reliability?

All of us that have been around technology for any length of time know, we can never turn back the clock. We continue to adapt. I can see virtualization affecting us all and I can see NonStop being caught up in it – operating systems that cannot be hosted within a virtual machine will fall by the wayside and become no more than historical reference points.

But are we truly prepared to go to sea with these new configurations? Have we begun to think about the ramifications that will come from an additional software layer between NonStop and the hardware? I have been reminded, in a very practical way, that when we begin to mix the new with the old then we need to be prepared for just about anything!

After all the data had been collected on the faulty propeller, it turned out to have nothing to do with the mechanism supporting the folding propeller. The “in port” software was being used by the auto pilot and not the “at sea” software, a situation where much finer tolerances were calibrated. By having this software engaged at sea, the propeller was trying to adjust to every swell it encountered! And yes, the yacht is now under full power once again, the fuel delivery problem resolved, and we are all headed to Civitavecchia, the port of Rome.

No comments: