Best of…: Classic WTF: A Systematic Approach

It’s a holiday in the US today, where we celebrate labor and laborers. Enjoy a story of working smarter, not harder, to meet unrealistic deadlines and create a lot more work for someone in the future. Original —Remy

It was the early 1990s and Frank was living the dream – unshaven, in pajama bottoms and his favorite hockey jersey, having just woken up at 12:18 PM, was now working in the dim light of his basement on one of his freelance projects. Just as he was sipping a cup of coffee, the phone rang.

Frank tried fruitlessly to fight an unexpected open-mouthed yawn when he picked up the receiver. “OOOOAAAaaahhhh… hello?”

“Hi, Frank. We have a very exciting opportunity for a qualified individual like you. Interested?” Odd that this caller hadn’t even identified what company he was with, Frank thought. After sleepily getting some of the details, yeah, Frank was mildly interested, and he agreed to an interview in 30 minutes… at a bagel shop that was 25 minutes away.

Ten minutes late and having not changed at all aside from putting on non-pajama pants and shoes, he was immediately flagged over by a man in a nice suit. “You have the look like a computer genius, I already know you’re perfect for this,” he said without a hint of sarcasm. He asked lots of vague questions about computers; stuff like “So you know computers? What about, like, servers?” Frank had all the traits of a young man in IT – beyond his physical appearance, he was brash, overconfident, and narcissistic. Throughout the brief interview, he didn’t restrain any of these traits, but still, he was only vaguely interested in the position, so he winged all the questions. Ten minutes later, the man pulled out a marker, wrote something on a napkin, and slid it across the table. “That’s what your hourly rate would be.” The man arched his eyebrow. “Interested?”

Frank was already pulling in some respectable bank, but this was more than twice what he was currently making. Gears started turning in his brain while he mulled it over. Well, I guess if I was to take this jo-“YES.” On his way home from the interview, Frank picked up a razor, stopped for a haircut, and generally got himself kempt.

The Job

The day after the interview, he was sitting opposite the Executive Vice President of Data Processing, who was giving a second interview. “Let’s say I asked you to consolidate twelve server facilities into one. How would you do it?” Ahh, a riddle question, like “how would you move a mountain to the other side of a village?”

“Well,” Frank started, winging his second interview in as many days, “I’d take a systematic approach. Figure out where everything was, how the different systems were integrated, how they communicate, and gradually migrate servers over.”

The Bank Boss was clearly impressed since Frank was given a contract on the spot; no references or background checks. And not just that, Frank was given keycard access to everything – all of the different datacenters and various facilities. He didn’t encounter anything his keycard didn’t give him access to (though he never tried the bank’s vault). He was also given passwords to every sensitive system, including the money transfer system, responsible for transferring money in the range of around one trillion dollars every day.

It quickly became clear that the interview question wasn’t a hypothetical, and Frank was dealing with the impossible. And maybe that’s why they hired him – he was an idealistic hard worker, a badass who could do anything. What he lacked in experience then he’d make up for in tenacity. At least that’s what Frank was having a harder and harder time reminding himself.

And for better or for worse, he’d effectively be on his own. As it turned out, his team and the man that hired him were electricians. Back in those days, the bank (and, more importantly, their union contract) considered moving servers to be “electrical work” and, therefore, only union electrician companies were allowed to do it. Or those that the electricians hired.

The Problem

In short, the bank had a dozen VAX datacenters that they wanted as a single cluster in a new building. Everything communicated over DECnet, which was a network protocol not too dissimilar from TCP/IP. One of the nice things about DECnet was the ability to route datagrams (packets) around the network in different ways to avoid failures, a fact which later proved to be useful.

The problem with the banks infrastructure was that either no one knew everything that was running in each datacenter, how it all worked, or how they’d work together in a single facility. Even worse, the people at the datacenters didn’t want to talk to Frank, as the sooner Frank finished his job, the sooner they’d lose theirs.

The visits to the various datacenters always left Frank in a worse mood. All of them were in various states of disrepair – dirty, perhaps a flickering light, disorganized, but one stood above the others. Its floor was almost completely torn up due to a major renovation project following an asbestos contamination. All of the systems in this room were caked with dust, so much so that the labels were completely illegible. Only one system had lights blinking on its network port; so Frank brushed the dust off its label. “VDL-23 – Maintenance Test Box.” Why does that sound so familiar?

Earlier that morning, Frank had done some network analysis, and found that all redundant links between the Money Transfer System and the other datacenters (aside from VDL-23) had been down for quite a while. Since DECnet routed around the failures and no one had thought to set-up network monitoring, no one had noticed that all of the other links had gone down. And before Frank discovered it, no one knew (or no one would tell him) where the box was physically. Immediately, Frank got cracking on a plan to bring the redundancy back online while keeping VDL-23 up, thanking the stars that the architecture of DECnet prevented the network from going down with all of the other servers. Still, something told him that the twisted, crushed, mistreated cables strewn across the floor might not remain reliable for transfers of $1T/day for long…

His watch alarm went off before he finished, it was a reminder that he had a mandatory meeting for all staff involved in the migration – about two hundred people in total. Frank rushed back to the main office, taking shortcuts using his everything-access keycard, and arriving just as the meeting was starting.

“Thank you all for coming,” the executive vice president said, voice quivering, with a sweaty brow. “We’re required to migrate the network to TCP/IP by the communications standard director.” Frank winced.

The VP of communications chimed in, equally nervous. “Well… it’s DECnet… it uses the non-routable LAT protocol. I… *ahem* we may need to push the deadline out…” The executive VP frowned, and asked for an update from the apps team.

The VP of Applications looked equally nervous, stammering out “we’re still gathering information on the applications… and… we’re not totally sure if we can actually run this all in one system…”

In a hushed tone, the executive VP took the floor again. “You’re telling me we’re going to miss our deadline. The board is very firm about our deadline. I’m not happy.”

Frank craned his neck and looked around the room, everyone was staring at the floor. This was his moment. Once again feeling like a badass he rose from his seat dramatically, and in his most heroic voice, he boomed “There is a way.“

Success!

Three months later, a beautifully appointed datacenter hummed along, processing transfers and perfectly replicating all of the functionality from the former twelve separate datacenters. Pristine rows of computers and neatly arranged drive controllers worked dutifully, everything in perfect order. The board came through the datacenter to inspect the work, and left with no complaints. It had been completed on schedule and was looking great, everyone was happy.

Under the raised floor, the tangle of cables told the real story. What was really in that room was the original twelve datacenters, all of the systems sitting pretty on the raised floor, underneath which were piles of cables connected exactly the same way they’d always been connected, and with back-hauled communication lines running between the systems. As they say, out with the old, in with the old.

And off in one corner, on upgraded hardware, was VDL-23; still responsible for routing all of the money transfer traffic, now safely sitting in a datacenter where it would remain unperturbed.

Source: Read More