Hunting the hackers: How Ukraine became Russia's test lab for cyberwar

We’re sorry, this feature is currently unavailable. We’re working to restore it. Please try again later.

Advertisement

This was published 6 years ago

Hunting the hackers: How Ukraine became Russia's test lab for cyberwar

By Andy Greenberg

The clocks read zero when the lights went out.

It was a Saturday night last December, and Oleksii Yasinsky was sitting on the couch with his wife and teenage son in the living room of their Kiev apartment. The 40-year-old Ukrainian cybersecurity researcher and his family were an hour into Oliver Stone's film Snowden when their building abruptly lost power.

"The hackers don't want us to finish the movie," Yasinsky's wife joked. She was referring to an event that had occurred a year earlier, a cyberattack that had cut electricity to nearly 250,000 Ukrainians two days before Christmas in 2015.

Yasinsky didn't laugh. He looked over at a portable clock on his desk: The time was 00.00. Precisely midnight.

A cyberattack that cut electricity to nearly 250,000 Ukrainians two days before Christmas in 2015 underlined the new threat modern societies face.

A cyberattack that cut electricity to nearly 250,000 Ukrainians two days before Christmas in 2015 underlined the new threat modern societies face.

He went to the kitchen, pulled out a handful of candles and lit them. Then he stepped to the kitchen window. The entire skyline around his apartment building was dark.

Noting the precise time and the date, almost exactly a year since the December 2015 grid attack, Yasinsky felt sure that this was no normal blackout. He thought of the cold outside, the slowly sinking temperatures in thousands of homes, and the countdown until dead water pumps led to frozen pipes.

For the past 14 months, Yasinsky had found himself at the centre of an enveloping crisis. A growing roster of Ukrainian companies and government agencies had come to him to analyse a plague of cyberattacks that were hitting them in rapid succession. A single group of hackers seemed to be behind all of it. Now he couldn't suppress the sense that those same phantoms had reached into his home.

The Cyber-Cassandras said this would happen. For decades they warned that hackers would soon make the leap beyond purely digital mayhem and start to cause real, physical damage to the world. In 2009, when the US National Security Agency's Stuxnet malware silently accelerated a few hundred Iranian nuclear centrifuges until they destroyed themselves, it seemed to offer a preview of this new era.

Advertisement

"This has a whiff of August 1945," Michael Hayden, former director of the NSA and the CIA, said in a speech. "Somebody just used a new weapon, and this weapon will not be put back in the box."

Now, in Ukraine, the quintessential cyberwar scenario has come to life. Twice.

Ukrainian President Petro Poroshenko in December, when he told the public that there had been 6500 cyberattacks on 36 Ukrainian targets in just the previous two months and blamed Russian intelligence.

Ukrainian President Petro Poroshenko in December, when he told the public that there had been 6500 cyberattacks on 36 Ukrainian targets in just the previous two months and blamed Russian intelligence.Credit: AP

On separate occasions, invisible saboteurs have turned off the electricity to hundreds of thousands of people. And the blackouts weren't just isolated attacks. They were part of a digital blitzkrieg that has pummeled Ukraine for the past three years. A hacker army has systematically undermined practically every sector of Ukraine: media, finance, transportation, military, politics, energy. "You can't really find a space in Ukraine where there hasn't been an attack," says Kenneth Geers, a NATO ambassador who focuses on cybersecurity.

In a public statement in December, Ukrainian President Petro Poroshenko reported that there had been 6500 cyberattacks on 36 Ukrainian targets in just the previous two months.

Many global cybersecurity analysts believe Russia is using the country as a laboratory for perfecting new forms of global online combat. And the digital explosives that Russia has repeatedly set off in Ukraine are ones it has planted at least once before in the civil infrastructure of the United States.

One Sunday morning in October 2015, Yasinsky's phone rang with a call from work. He was then director of information security at StarLightMedia, Ukraine's largest TV broadcasting conglomerate. During the night, two of StarLight's servers had inexplicably gone offline.

"One server going down, it happens," Yasinsky says. "But two servers at the same time? That's suspicious."

June 2017: Screens inside a retail store in Kiev show a demand for ransom on computers infected by the 'Petya' virus.

June 2017: Screens inside a retail store in Kiev show a demand for ransom on computers infected by the 'Petya' virus. Credit: Bloomberg

Yasinsky quickly discovered the attack was indeed far worse than it had seemed: The two corrupted servers had planted malware on the laptops of 13 StarLight employees.

Yasinsky was struck by the layers of cunning obfuscation - the malware had evaded all antivirus scans and even impersonated an antivirus scanner itself, Microsoft's Windows Defender. After his family had gone to sleep, Yasinsky printed the code and laid the papers across his kitchen table and floor, crossing out lines of camouflaging characters and highlighting commands to see its true form.

Beneath all the cloaking and misdirection, Yasinsky figured out, was a piece of malware known as KillDisk. By tracing signs of the hackers' fingerprints, Yasinsky and two colleagues came to the stomach-turning realisation that the intruders had been inside their system for more than six months. Eventually, Yasinsky identified the piece of malware that had served as the hackers' initial foothold: an all-purpose Trojan known as BlackEnergy.

Soon Yasinsky began to hear from colleagues at other companies and in the government that they too had been hacked, and in almost exactly the same way - BlackEnergy for access and reconnaissance, KillDisk for destruction.

"With every step forward, it became clearer that our Titanic had found its iceberg," says Yasinsky. "The deeper we looked, the bigger it was."

An alarm goes off in America

At first, Robert Lee blamed the squirrels.

It was Christmas Eve 2015 - the day before Lee was set to be married in his hometown of Cullman, Alabama. A barrel-chested and bearded redhead, Lee had recently left a high-level job at a three-letter US intelligence agency, where he'd focused on the cybersecurity of critical infrastructure. Now he was settling down to launch his own security startup and marry the Dutch girlfriend he'd met while stationed abroad.

As Lee busied himself with wedding preparations, he saw news headlines claiming that hackers had just taken down a power grid in western Ukraine. Lee blew off the story - he had other things on his mind, and he'd heard spurious claims of hacked grids plenty of times before. The cause was usually a rodent or a bird.

The next day, however, just before the wedding itself, Lee got a text from Mike Assante, a security researcher at the SANS Institute, an elite cybersecurity training centre. When it comes to digital threats to power grids, Assante is one of the most respected experts in the world. And he was telling Lee that the Ukraine blackout hack looked like the real thing.

Just after Lee had said his vows and kissed his bride, a contact in Ukraine messaged him as well: The blackout hack was real, the man said, and he needed Lee's help. The moment Lee had anticipated for years had finally arrived. So he ditched his own reception and began to text with Assante in a quiet spot, still in his wedding suit.

Lee eventually retreated to his mother's desktop computer in his parents' house nearby. Working in tandem with Assante, who was at a friend's Christmas party in rural Idaho, they pulled up maps of Ukraine and a chart of its power grid. The three power companies' substations that had been hit were hundreds of kilometres from one another and unconnected. "This was not a squirrel," Lee concluded.

Lee was struck by similarities between the blackout hackers' tactics and those of a group that had recently gained some notoriety in the cybersecurity world, known as Sandworm. The group's name came from references to the science fiction novel Dune found buried in its code.

All signs indicated that the hackers were Russian. Most disturbing of all for US analysts, Sandworm's targets extended across the Atlantic. Earlier in 2014, the US government reported that hackers had planted BlackEnergy on the networks of American power and water utilities.

Assante thought it was too early to start blaming the attack on any particular hacker group, not to mention a government. But in Lee's mind, alarms went off. "An adversary that had already targeted American energy utilities had crossed the line and taken down a power grid," Lee says. "It was an imminent threat to the United States."

Jeanette Manfra of the Department of Homeland Security speaks while Bill Priestap, of the FBI, right, listens during a congressional hearing on sanctions against Russia.

Jeanette Manfra of the Department of Homeland Security speaks while Bill Priestap, of the FBI, right, listens during a congressional hearing on sanctions against Russia.Credit: Bloomberg

On a cold, bright day a few weeks later, a team of Americans arrived in Kiev. Among them were staff from the FBI, the Department of Energy, the Department of Homeland Security, and the North American Electric Reliability Corporation (NERC), the body responsible for the stability of the US grid, all part of a delegation assigned to get to the bottom of the Ukrainian blackout.

The Feds had also flown Assante in from Wyoming. Lee, a hotter head than his friend, had fought with the US agencies over their penchant for secrecy, insisting that the details of the attack needed to be publicised immediately. He hadn't been invited.

On that first day, the suits gathered in a hotel conference room with the staff of Kyivoblenergo, the city's regional power distribution company and one of the three victims of the power grid attacks. Over several hours, the Ukrainian company's stoic execs and engineers laid out a blow-by-blow account of the raid on their network.

That night, the team boarded a flight to the western Ukrainian city of Ivano-Frankivsk, arriving at its tiny Soviet-era airport in a snowstorm. The next morning they visited the headquarters of Prykarpattyaoblenergo, the company that had taken the brunt of the pre-Christmas attack.

The attack they described was almost identical to the one that hit Kyivoblenergo. But in this operation, the attackers had taken another step, bombarding the company's call centres with fake phone calls - possibly to delay any warnings of the power outage from customers or simply to add another layer of chaos.

There was another difference too. When the Americans asked whether, as in Kiev, cloned control software had sent the commands that shut off the power, the Prykarpattyaoblenergo engineers said no. That's when the company's technical director, a tall, serious man with black hair and ice-blue eyes, cut in. Rather than try to explain the hackers' methods through a translator, he clicked play on a video he'd recorded on his battered iPhone 5s.

The 56-second clip showed a cursor moving around the screen of one of the computers in the company's control room. The video pans from the computer's Samsung monitor to its mouse, which hasn't budged. Then it shows the cursor moving again, seemingly of its own accord, as the engineers in the room ask one another who's controlling it.

The intruders had exploited the company's IT helpdesk tool to take direct control of the mouse movements of the stations' operators. Before their eyes, phantom hands had clicked through dozens of breakers - each serving power to a different swath of the region - and one by one, turned them cold.

'The dark side is united'

In August 2016, eight months after the first Christmas blackout, Yasinsky left his job at StarLightMedia. It wasn't enough, he decided, to defend a single company from an onslaught that was hitting every level of Ukrainian society. "The light side remains divided," he says of the balkanised reaction to the hackers among their victims. "The dark side is united."

So Yasinsky took a position as the head of research and forensics for a Kiev firm called Information Systems Security Partners. Yasinsky turned it into a de facto first responder for victims of Ukraine's digital siege.

Not long after Yasinsky switched jobs, the country came under another, even broader wave of attacks. He ticks off the list of casualties: Ukraine's pension fund, the country's treasury, its seaport authority, its ministries of infrastructure, defence and finance. The hackers hit Ukraine's railway company, knocking out its online booking system for days, right in the midst of the holiday travel season. In the case of the finance ministry, the logic bomb deleted terabytes of data, just as the ministry was preparing its budget for the next year. All told, the hackers' new winter onslaught matched and exceeded the previous year's - right up to its grand finale.

June 2017: employees at Boryspil airport in Kiev struggle to counter data-scrambling software that caused disruption across Europe but hit Ukraine especially hard.

June 2017: employees at Boryspil airport in Kiev struggle to counter data-scrambling software that caused disruption across Europe but hit Ukraine especially hard.Credit: AP

On December 16, 2016, as Yasinsky and his family sat watching Snowden, a young engineer named Oleg Zaychenko was four hours into his 12-hour night shift at Ukrenergo's transmission station just north of Kiev.

He was filling out a paper-and-pencil log, documenting another uneventful Saturday evening, when the station's alarm suddenly sounded, a deafening continuous ringing. To his right Zaychenko saw that two of the lights indicating the state of the transmission system's circuits had switched from red to green - off.

The technician called an operator at Ukrenergo's headquarters to alert him to the routine mishap. As he did, another light turned green. Then another.

The operator ordered Zaychenko to run outside and check the equipment for physical damage. At that moment, the 20th and final circuit switched off and the lights in the control room went out, along with the computer and TV.

That single Kiev transmission station carried 200 megawatts, about a fifth of the capital's electrical capacity, more total electric load than the 50-plus distribution stations knocked out in the 2015 attack combined. Luckily, the system was down for just an hour before Ukrenergo's engineers began bringing everything back online.

But the brevity of the outage was virtually the only thing that was less menacing about the 2016 blackout. Cybersecurity firms that have since analysed the attack say that it was far more evolved than the one in 2015: It was executed by a highly sophisticated, adaptable piece of malware now known as "CrashOverride", an automated, grid-killing weapon.

Marina Krotofil, an industrial control systems security researcher for Honeywell who also analysed the Ukrenergo attack, describes the hackers' methods as simpler and far more efficient than the ones used in the previous year's attack. "In 2015 they were like a group of brutal street fighters," Krotofil says. "In 2016, they were ninjas." But the hackers themselves may be one and the same; researchers at Lee's security startup, Dragos, have identified the architects of CrashOverride as part of Sandworm, based on evidence that Dragos is not yet ready to reveal.

'It can happen here'

I meet Lee in the bare-bones Baltimore offices of Dragos. Outside his office window loom pylons holding up transmission lines. Lee tells me that they carry power 30 kilometres south - to the heart of Washington, DC.

"The people who understand the US power grid know that it can happen here," Lee says.

No one knows how, or where, Sandworm's next attacks will materialise. A future breach might target not a distribution or transmission station but an actual power plant. Or it could be designed not simply to turn off equipment but to destroy it. In 2007 a team of researchers at Idaho National Lab, including Mike Assante, used nothing but digital commands to permanently wreck a 2.25-megawatt diesel generator. In a video of the experiment, a machine the size of a living room coughs and belches black and white smoke in its death throes.

"Washington, DC? A nation-state could take it out for two months without much issue," Lee says.

An isolated incident of physical destruction may not even be the worst that hackers can do. The American cybersecurity community often talks about "advanced persistent threats" - sophisticated intruders who don't simply infiltrate a system for the sake of one attack but stay there, silently keeping their hold on a target.

"Washington, DC? A nation-state could take it out for two months without much issue," says cybersecurity expert Robert Lee.

"Washington, DC? A nation-state could take it out for two months without much issue," says cybersecurity expert Robert Lee.Credit: AAP

"If they did that in multiple places, you could have up to a month of outages across an entire region," he says. "Tell me what doesn't change dramatically when key cities across half of the US don't have power for a month."

A grid attack on American utilities would almost certainly result in immediate, serious retaliation by the US. Some cybersecurity analysts argue that Russia's goal is simply to show that it's capable of penetrating the American grid - a message warning the US not to try a Stuxnet-style attack on Russia or its allies. In that view, it's all a game of deterrence.

But Lee, who was involved in war-game scenarios during his time in intelligence, believes Russia might actually strike American utilities if it ever saw itself as backed into a corner - say, if the US threatened to interfere with Moscow's military interests in Ukraine or Syria. "When you deny a state's ability to project power, it has to lash out," Lee says.

People like Lee have, of course, been war-gaming these nightmares for well over a decade. And for all the sophistication of the Ukraine grid hacks, even they didn't really constitute a catastrophe; the lights did, after all, come back on.

US power companies have already learned from Ukraine's victimisation, says Marcus Sachs, chief security officer of NERC. After the 2015 attack, Sachs says, NERC went on a road show, meeting with power firms to hammer into them that they need to shore up their basic cybersecurity practices and turn off remote access to their critical systems more often.

"It would be hard to say we're not vulnerable. Anything connected to something else is vulnerable," Sachs says. "To make the leap and suggest that the grid is milliseconds away from collapse is irresponsible."

March 2015: A map of the United States shows cyber attacks in real time at the headquarters of Bitdefender, a leading Romanian cyber security company.

March 2015: A map of the United States shows cyber attacks in real time at the headquarters of Bitdefender, a leading Romanian cyber security company.Credit: Mediafax via AP

But for those who have been paying attention to Sandworm for almost three years, raising an alarm about the potential for an attack on the US grid is no longer crying wolf.

For John Hultquist, head of the team of researchers at FireEye that first spotted and named the Sandworm group, the wolves have arrived.

Three weeks after the 2016 Kiev attack, he wrote a prediction on Twitter and pinned it to his profile for posterity: "I swear, when Sandworm Team finally nails Western critical infrastructure, and folks react like this was a huge surprise, I'm gonna lose it."

'Your own private space is just an illusion'

Yasinsky says he has tried to maintain a dispassionate perspective on the intruders who are ransacking his country. But when the blackout extended to his own home four months ago, it was "like being robbed", he tells me. "It was a kind of violation, a moment when you realise your own private space is just an illusion."

When we meet in his company's offices, the next wave of the digital invasion is already under way. The attacks, Yasinsky has noticed, have settled into a seasonal cycle: During the first months of the year, the hackers lay their groundwork, silently penetrating targets and spreading their foothold. At the end of the year, they unleash their payload.

He sums up the attackers' intentions in a single Russian word: poligon. A training ground. Even in their most damaging attacks, Yasinsky observes, the hackers could have gone further - a restraint that Assante and Lee have also noted. "They're still playing with us," Yasinsky says.

Many global cybersecurity analysts have come to the same conclusion. "This is a place where you can do your worst without retaliation or prosecution," says Geers, the NATO ambassador. "A lot of Americans can't find it on a map, so you can practice there." (At a meeting of diplomats in April, US Secretary of State Rex Tillerson went so far as to ask "why should US taxpayers be interested in Ukraine?".)

Russia isn't only pushing the limits of its technical abilities, says Thomas Rid, a professor in the War Studies department at King's College, London. The Kremlin meddled in the Ukrainian election and faced no real repercussions; then it tried similar tactics in Germany, France, and the United States. "They're testing out red lines, what they can get away with," Rid says. "You push and see if you're pushed back. If not, you try the next step."

What will that next step look like? In the dim back room at his lab in Kiev, Yasinsky admits he doesn't know.

"Cyberspace is not a target in itself," Yasinsky says. "It's a medium." And that medium connects, in every direction, to the machinery of civilisation itself.

Wired Magazine - A Conde Nast publication

Most Viewed in World

Loading