Once upon a time, there was a bank whose business relied on a mainframe. As the decades passed and the 21st century dawned, the bank’s bigwigs realized they had to upgrade their frontline systems to applications built in Java and .NET, but—for myriad reasons that boiled down to cost, fear, and stubbornness—they didn’t want to migrate away from the mainframe entirely. They also didn’t want the new frontline systems to talk directly to the mainframe or vice-versa. So they tasked old-timer Edgar with writing some middleware. Edgar’s brainchild was a Windows service that took care of receiving frontline requests, passing them to the mainframe, and sending the responses back.
Edgar’s middleware worked well, so well that it was largely forgotten about. It outlasted Edgar himself, who, after another solid decade of service, moved on to another company.
A few years later, our submitter John F. joined the bank’s C# team. By this point, the poor middleware seemed to be showing its age. A strange problem had arisen: between 8:00AM and 5:00PM, every 45 minutes or so, it would lock up and have to be restarted. Outside of those hours, there was no issue. The problem was mitigated by automatic restarts, but it continued to inflict pain and aggravation upon internal users and external customers. A true solution had to be found.
Unfortunately, Edgar was long gone. The new “owner” of the middleware was an infrastructure team containing zero developers. Had Edgar left them any documentation? No. Source code? Sort of. Edgar had given a copy of the code to his friend Bob prior to leaving. Unfortunately, Bob’s copy was a few point releases behind the version of middleware running in production. It was also in C, and there were no C developers to be found anywhere in the company.
And so, the bank’s bigwigs cobbled together a diverse team of experts. There were operating system people, network people, and software people … including the new guy, John. Poor John had the unenviable task of sifting through Edgar’s source code. Just as the C# key sits right next to the C key on a piano, reasoned the bigwigs, C# couldn’t be that different from C.
John toiled in an unfamiliar language with no build server or test environment to aid him. It should be no great surprise that he got nowhere. A senior coworker suggested that he check what Windows’ Process Monitor registered when the middleware was running. John allowed a full day to pass, then looked at the results: it was now clear that the middleware was constantly creating and destroying threads. John wrote a Python script to analyze the threads, and found that most of them lived for only seconds. However, every 5 minutes, a thread was created but never destroyed.
This only happened during the hours of 8:00AM to 5:00PM.
At the next cross-functional team meeting behind closed doors, John finally had something of substance to report to the large group seated around the conference room table. There was still a huge mystery to solve: where were these middleware-killing threads coming from?
“Wait a minute! Wasn’t Frank doing something like that?” one of the other team members piped up.
“Frank!” A department manager with no technical expertise, who insisted on attending every meeting regardless, darted up straight in his chair. For once, he wasn’t haranguing them for their lack of progress. He resembled a wolf who’d sniffed blood in the air. “You mean Frank from Accounting?!”
This was the corporate equivalent of an arrest warrant. Frank from Accounting was duly called forth.
“That’s my program.” Frank stood before the table, laid back and blithe despite the obvious frayed nerves of several individuals within the room. “It queries the middleware every 5 minutes.”
They were finally getting somewhere. Galvanized, John’s heart pounded. “How?” he asked.
“Well, it could be that the middleware is down, so first, my program opens a connection just to make sure it’s working,” Frank explained. “If that works, it opens another connection and sends the query.”
John’s confusion mirrored the multiple frowns that filled the room. He forced himself to carefully parse what he’d just heard. “What happens to the first connection?”
“What do you mean?” Frank asked.
“You said your program opens two connections. What do you do with the first one?”
“Oh! I just use that one to test whether the middleware is up.”
“You don’t need to do that!” one of the networking experts snarled. “For Pete’s sake, take that out of your code! Don’t you realize you’re tanking this thing for everyone else?”
Frank’s expression made clear that he was entirely oblivious to the chaos wrought by his program. Somehow, he survived the collective venting of frustration that followed within that conference room. After one small update to Frank’s program, the middleware stabilized—for the time being. And while Frank became a scapegoat and villain to some, he was a hero to many, many more. After all, he single-handedly convinced the bank’s bigwigs that the status quo was too precarious. They began to plan out a full migration away from mainframe, a move that would free them from their dependence upon aging, orphaned middleware.
Now that the mystery had been solved, John knew where to look in Edgar’s source code. The thread pool had a limit of 10, and every thread began by waiting for input. The middleware could handle bad input well enough, but it hadn’t been written to handle the case of no input at all.
[Advertisement]Utilize BuildMaster to release your software with confidence, at the pace your business demands. Download today!
Source: Read MoreÂ