Authors: Mark Russinovich
That’s where he and Frank had pried a space open yesterday.
10
TRADING PLATFORMS IT SECURITY
WALL STREET
NEW YORK CITY
3:03
P.M.
Bill Stenton placed the telephone in its cradle and leaned back in his chair. This is happening too often, he thought. It was the third call in the last two weeks, each from a senior director on the Exchange, each with the same complaint. He’d been receiving similar calls for months.
His right hand had developed a tremor, and he placed his left on it. He closed his eyes for a moment and forced himself to breathe deeply, to slow down. Always tightly wound, he worked hard to present himself in the assured manner expected of someone in his position. He checked the clock on his computer screen. In three hours, he’d have his first double Scotch. He pulled his mind away from the thought.
Earlier, he’d received a call from a financial institution. The caller was a college fraternity brother who reported having experienced unanticipated and significant losses in a major trade. On such calls, Stenton had observed no pattern in the type of company or in the nature of the securities involved. In some cases they’d been hedge funds, in others private investment groups, in another a retirement fund—but in every case, the complaint was the same: Something out of the ordinary had taken place during a major transaction, which resulted in an unexplained reduction in the anticipated return, losses well outside the anticipated parameters.
The most common complaint was that HFT algorithms that had been reliable in the past, previously providing a profit within the margins established, were suddenly showing dramatic failures. As HFT systems were all located at the hub, they should have had the least latency. Instead, trades that should have netted a modest profit or at least been neutral ended up implausibly losing tens or hundreds of thousands. In more than one case, the loss had exceeded a million dollars.
Until now, Stenton had viewed the complaints, each taken in isolation, as so much griping by traders who were not keeping up with the game. Dealing with complaints like these came with the territory, but over the past few months, their rate and the magnitude of the individual issues had caused him to suspect there might be more to this than the usual Wall Street whining.
Still, Stenton had explained to the callers that it was not unusual for brokers to blame the system when they made a bad judgment call or when the market suddenly moved against a position they’d taken. In these days of high-speed transactions and high-frequency trades, that was to be expected. Yes, he understood that men and women had been fired, careers likely ruined over unforeseen moves, and that the losses had been significant enough to place the survival of some of the smaller financial institutions at risk. But the system wasn’t at fault, Stenton was sure of that and had said as much. He told them he knew they wouldn’t like it, but that was the reality.
But in the last few weeks, he’d also received two calls from other men he knew personally, reasonable brokers with serious questions about what was taking place. They’d been puzzled at unanticipated losses, not suspicious, and Stenton assured them that all was well with the Exchange.
But taken together, the string of calls caused him to rethink his position. He’d been searching for common links in the complaints. He thought perhaps there was a shared broker somewhere in the mix, or common stock. It might have been a time-of-day issue or the location of their programs in the hub. He had some of his top data analysts working to mine the available data, searching for any correlating factors.
Now Stenton held last week’s report from the Chicago office. An IT operations manager there, Vince Piscopia, had forwarded a report to his superior, which then landed on Stenton’s desk. As director of the Trading Platform IT Security for NYSE Euronext, he was in charge of this issue, but so far he wasn’t certain how to respond. The day after receiving it, he’d copied the report to all his senior staff and his key analysts, requesting input.
What he didn’t tell anyone, what he scarcely allowed
himself
to think, was that perhaps all these issues were connected.
What the IT manager in Chicago had reported was a file concealed within the core of their system, software outside the directory listing command. He’d been unable to access the file or determine what it did. What he’d been confident of was that it was not part of the legitimate function of the Exchange.
The IT manager, Piscopia, had speculated that it might be a bit of legacy code left over from one of the periodic updates of the system. Unnecessary code was left behind from time to time, but never before had it been hidden, and there was no way to know if it was harmless or somehow interfering with operations. In the same report, he stated that he’d also uncovered trades that were not properly registering and speculated that they were related to the code. This raised the same possibility in the mind of the Chicago IT manager as it had in Stenton’s.
Impossible as he found it to accept, just maybe they’d been hacked.
Stenton shivered at the thought. He was at the helm. If the Exchange had been hacked and clients were experiencing losses as a consequence, his career was finished. In the worst-case scenario, one that in his fear he deemed possible, he might go to prison.
This possibility had come to his attention since Jeff Aiken had been hired and started his penetration test. Stenton had considered alerting the consultant but decided instead to see what Aiken and his man came up with on their own. Also, alerting them would leave a record of his suspicions, which so far existed only in his thoughts. It was his hope that Red Zoya not only stumbled onto what Chicago had described but also figured out what it was in the process. That was his safest course of action.
Just then, a tech poked his head into Stenton’s doorway. “I’m Marc,” he said. “You asked to see me?”
Stenton recalled that Marc Campos worked on the core trading platform team on one of the trading modules at the heart of the matching engines. There was no more sensitive operation in NYSE Euronext. He was one of the techs who attempted to trace a suspect trade a financial institution had reported to Stenton.
“Yes, come in. What do you have?”
Campos was over six feet tall, thirty-three years old, with dark skin and average looks, though his eyes bulged slightly. Originally from Portugal, he’d worked for the NYSE for the past five years, and his performance had been outstanding. He spoke near colloquial English with just the trace of his native accent. He was one of the handful of techs with unfettered access to the core of the Exchange’s trading system. This was the first time Stenton had met with him alone though he’d seen Campos from time to time in staff meetings.
For several minutes, Campos described the steps he’d taken in tracing a reported $8.7 million loss in a transaction by one of the smaller financial institutions that had lodged a complaint. The referral had come directly to Stenton from a broker he’d known for years, a very reasonable man who was more perplexed than angry at what had taken place. When Campos finished, he smiled and made a dismissive gesture. “I worked on it until the trade just vanished. There was nothing I could do.”
“Have you seen this before?” Stenton asked.
“Sure, but not often. Some of these offshore funds like to remain off the radar, you know? They don’t like anyone knowing what they’re doing. They go to great lengths to conceal their tracks beyond the minimum they need to trade with the Exchange. I’ve attempted to trace back trades with them, usually as a result of an SEC subpoena, and not always been successful, though you understand it’s not really my area.”
“Well, thank you anyway. I’d hoped for better news.”
Campos hesitated, then said, “I’ve also been working on this Chicago report you sent out a few days ago.”
“Any luck?”
Campos shook his head. “I don’t see any sign of it. I think the guy in Chicago was confused somehow. He was likely misreading what he was seeing. Frankly, I don’t see how anything could get into our system undetected. We’re as locked down as you can get. Do we have his data? Maybe I’m missing something.”
“No. I requested it, but he didn’t come to work today,” Stenton answered. Campos nodded in return and Stenton asked, “So for now, you don’t think this stealth file exists?”
“I can’t see it—” He paused and smiled. “—but then, it’s supposed to be hidden.”
Stenton thought a second, then took the plunge. “I know that you’ve been with this department for some time now, Marc. Do you think there might be a connection between the trade you traced and this hidden file Chicago reported?”
Campos looked surprised at the question. “That’s an interesting idea, but there’s nothing that connects them in theory. And there’s no way a secret file could get into the engines. If by some magic it did, we’d be all over it in an instant. Like I said, I don’t think this file even exists—and if it does, from what I read, there is no indication of what it does, if it does anything. Not only that, but we run an incredibly complex system. If getting an unauthorized file into the system is hard, manipulating a trade is simply impossible.”
Stenton sighed. “You’re right. I guess I’m just getting paranoid.”
“Is this what those two are working on? Red Zoya?”
“Why do you ask?”
“The timing. I thought maybe you’d put a team on this even before the guy in Chicago made his report.”
“No, that’s something else.” Stenton eyed Campos, then said, “Thanks for dropping by. I’ll let your manager know I requested this meeting, so no concerns there. Have a good day.”
“When you get the data from Chicago, you’ll pass it along?” Campos asked as he stood up.
“Yes. Of course. We need to figure this out. And as for Red Zoya, just let them be. They aren’t connected to this at all.”
11
TRADING PLATFORMS IT SECURITY
WALL STREET
NEW YORK CITY
3:44
P.M.
In practice, a trusted Exchange employee accessed the secure zone by first logging in to the jump server through an account specific to that zone. Since gaining their toehold in the system, Jeff and Frank were next tasked to compromise someone with privileged access.
Working from their office at IT Security that afternoon, Jeff and Frank had been consumed with analyzing the log-in records on the breached Payment Dynamo server. They soon identified a systems administrator who routinely connected to it from the other systems. When a user connected, the encrypted version of his password was cached by the server, allowing the user to connect to other servers without having to reenter the password. Being able to connect to different systems using a single entry of credentials is known as “single sign on” (SSO) and penetration testers, just like hackers, took advantage of SSO’s caching behavior to execute what was known as a “pass the hash” (PTH) attack on other systems. This attack used the cached cryptographic hash of the password, a form of shorthand, to impersonate the user and connect to remote servers. Servers verified only the hash of passwords, not the passwords themselves. Because of the considerable security risk, systems administrators were never to use their administrator accounts when logging on to other servers remotely. Jeff knew it was a common practice, however, either because of ignorance or sheer laziness.
Within minutes, they’d successfully infected the administrator’s computer. For now, they had administrative rights on the insecure network only, not the jump servers and therefore not the secure network they sought. But so far, they could view all the users in the network, identify their computers, even change their passwords and create new accounts, giving themselves administrative permission.
They were confident, however, operations of this kind were audited to prevent the kind of tampering they were doing. Automated software trolls checked logs and flagged unusual reports to detect illegitimate or unauthorized activity.
Next, Red Zoya targeted the team members from the UTP list Frank had identified. Working remotely from the administrator’s own workstation, they determined the computers that corresponded to those users. Some users were inactive, but most were not. It was necessary to employ different users for different functions to prevent any security tool monitoring user activity from spotting the same user executing different operations at the same time. Frank and Jeff gave one account administrative permission to the computer of another user whom they believed had jump server access. Next they logged in to that user’s computer, connected to the system of the programmer they were targeting, dropped their software backdoor, logged out, and then removed the administrative access to conceal their tracks. Even if part of their trail was spotted, it would be difficult for anyone to connect the dots.
During the next phase of their penetration attack, Jeff and Frank performed an enhanced reconnaissance on the UTP programmer’s system. They were careful to keep their presence at a low profile, operating only when their user was logged in, so the activity blended in. They read his e-mail, the documents on his system, and observed the software environment, all undetected.
They finally ascertained the jump server system to which the user connected. The issue they now faced was that the jump server required a two-factor authentication, which meant that a password alone wasn’t enough to get through it. When gaining access, the user read a pass code shown on a USB key fob, issued with a small LCD display on it. They then entered this number, along with a personally chosen four-digit PIN as the password. This scheme ensured that access required both possession of the USB key fob and knowledge of the PIN. And because the pass code changed every sixty seconds, it could not be saved and reused later. This meant that Frank and Jeff had to wait for the moment when the user logged in to the jump server, at which point they’d piggyback onto the connection.