The Inside Story of How Facebook Responded to Tunisian Hacks

 

It was on Christmas Day that Facebook's Chief Security Officer Joe Sullivan first noticed strange things going on in Tunisia. Reports started to trickle in that political-protest pages were being hacked. "We were getting anecdotal reports saying, 'It looks like someone logged into my account and deleted it,'" Sullivan said.

For Tunisians, it was another run-in with Ammar, the nickname they've given to the authorities that censor the country's Internet. They'd come to expect it.

In the days after the holiday, Sullivan's security team started to take a closer look at the data, but it wasn't entirely clear what was happening. In the US, they could look to see if different IP addresses, which identify particular nodes on the network, were accessing the same account. But in Tunisia, the addresses are commonly reassigned. The evidence that accounts were being hacked remained anecdotal. Facebook's security team couldn't prove something was wrong in the data.  It wasn't until after the new year that the shocking truth emerged:

Ammar was in the process of stealing an entire country's worth of passwords.

* * *

Here's what's at stake. December of 2010 saw the most substantial civil unrest in Tunisia in the reign of Zine El Abidine Ben Ali, which began with a bloodless coup in November 1987. Beginning with street protests in the country's poor interior region of Sidi Bouzid, the calls for change were soon echoed by more powerful civil society organizations, notably the country's only labor union, the UGTT. But despite the turmoil, it wasn't clear what exactly might happen.

"It is too early to know if these protests signal the beginning of the end for Ben Ali," wrote Christopher Alexander in Foreign Policy on January 3. "However, Tunisia's current political scene looks a bit like it did in 1975 and 1976, the beginning of the long slide for Ben Ali's predecessor, Habib Bourguiba."

That is to say, even expert analysts of the country couldn't tell if Ben Ali would remain in power for a few more weeks or a decade. It did not feel inevitable that Ben Ali would be deposed. People had protested in the streets before. Revolution had been in the air. It wasn't clear that this time would be different.

There has been a lot of debate about whether Twitter helped unleash the massive changes that led Ben Ali to leave office on January 14, but Facebook appears to have played a more important role in spreading dissent. 

"I think Facebook played a bigger role in this case," said Jillian York of the Berkman Center for the Internet and Society, who has been tracking the Tunisian situation closely. "There are a lot more Facebook users than Twitter users. Facebook allows for strong ties in a way that Twitter doesn't. You're not just conversing."

One early sign that Tunisians felt Facebook could be useful: Back in July, bloggers Photoshopped a picture of Mark Zuckerberg to show him holding up a sign that read, "Sayeb Sala7, ya 3ammar," the slogan for a freedom of expression campaign late in 2010. Later, Zuckerberg popped up on a sign outside the Saudi Arabian embassy carried by Tunisian protesters demanding the arrest of Ben Ali.

York said that Tunisian bloggers and activists had told her that the ability to upload video to Facebook drove its usage because many other video-sharing sites had been blocked by the government.

The videos -- shot shakily with cameraphones -- created a link between what was happening on the streets in the poor areas of the country and the broader Tunisian population. Many are graphic. In one video -- since taken down, apparently -- a young man is lying on a gurney with his skull cracked open. Brain oozes out. Cries are heard all around. The video focuses in on the man's face and as the camera pulls back, we see that there are two other people with cameraphones recording the injury. Video after video of the revolutionary events captures other people videoing the same event. Those videos, and the actions they recorded, became the raw material for a much greater online apparatus that could amplify each injury, death, and protest.

But it wasn't just videos that people were sharing. All kinds of information passed between Tunisians. For activists as well as everyday people, Facebook became an indispensable resource for tracking the minute-by-minute development of the situation. By January 8, Facebook says that it had several hundred thousand more users than it had ever had before in Tunisia, a country with a few more people than Michigan. Scaled up to the size to the U.S., the burst of activity was like adding 10 million users in a week. And the average time spent on the site more than doubled what it had been before.

Rim Abida, a Tunisian-born, Harvard-educated development consultant now living in Rio de Janeiro, said that over the course of the events, her "relationship to Facebook changed entirely."

"It basically went from being a waste of time or procrastination tool, to my go-to source on up-to-date information," Abida wrote in a Facebook message to me. "My mom is back in Tunisia on her own, and my Tunisian network on Facebook was posting the most up-to-date info on what was happening on the ground. It was stuff the major media channels weren't reporting, such as numbers to call to reach the military and what was happening when in what specific neighborhood."

In between the scenes of local unrest and people like Abida, there was a whole stratum of bloggers, writers, and social media sharers who watched and shared important videos.

While clashes with security forces took place in the streets, Rim, who asked we not use her last name, was in her bed in her apartment in Tunis. Like the blogger cliché, Rim sat in her pajamas sharing videos. In her hands, small protests that reached 50 people could suddenly reach another 50, who would share it with another 50. The idea that it might be time for the regime to change spread from city to city faster than street protests and even middle class places got involved.

Rim doesn't think the Tunisian revolution was a "Facebook revolution," but it was sufficiently important that when rumors started to fly on the 13th about what kind of retaliation the government was prepared to take, it took this form:

"There were rumors that Facebook or electricity was going to be shut down," Rim IM'd me from Tunis. "Or both." 

After more than ten days of intensive investigation and study, Facebook's security team realized something very, very bad was going on. The country's Internet service providers were running a malicious piece of code that was recording users' login information when they went to sites like Facebook.

By January 5, it was clear that an entire country's worth of passwords were in the process of being stolen right in the midst of the greatest political upheaval in two decades. Sullivan and his team decided they needed a country-level solution -- and fast.

Though Sullivan said Facebook has encountered a wide variety of security problems and been involved in various political situations, they'd never seen anything like what was happening in Tunisia.

"We've had to deal with ISPs in the past who have tried to filter or block our site," Sullivan said. "In this case, we were confronted by ISPs that were doing something unprecedented in that they were being very active in their attempts to intercept user information."

If you need a parable for the potential and pitfalls of a social-media enabled revolution, this is it: the very tool that people are using for their activism becomes the very means by which their identities could be compromised. When the details are filled in on the abstractions of Clay Shirky and Evgeny Morozov's work on the promise (former) and danger (latter) of Internet activism, the ground truth seems to be that both had their visions play out simultaneously.

At Facebook, Sullivan's team decided to take an apolitical approach to the problem. This was simply a hack that required a technical response. "At its core, from our standpoint, it's a security issue around passwords and making sure that we protect the integrity of passwords and accounts," he said. "It was very much a black and white security issue and less of a political issue."

The software was basically a country-level keystroke logger, with the passwords presumably being fed from the ISPs to the Ben Ali regime. As a user, you just logged into some part of the cloud, Facebook or your email, say, and it snatched up that information. If you stayed persistently logged in, you were safe. It was those who logged out and came back that were open to the attack.

Sullivan's team rapidly coded a two-step response to the problem. First, all Tunisian requests for Facebook were routed to an https server. The Https protocol encrypts the information you send across it, so it's not susceptible to the keylogging strategy employed by the Tunisian ISPs.

The second technical solution they implemented was a "roadblock" for anyone who had logged out and then back in during the time when the malicious code was running. Like Facebook's version of a "mother's maiden name" question to get access to your old password, it asks you to identify your friends in photos to complete an account login.

They rolled out the new solutions to 100% of Tunisia by Monday morning, five days after they'd realized what was happening. It wasn't a totally perfect solution. Most specifically, ISPs can force a downgrade of https to http, but Sullivan said that Facebook had not seen that happen.

Though Sullivan is the unflappable type, the Tunisian situation seemed to force him into a bit of reflection. "When you step back and think about how Internet traffic is routed around the world, an astonishing amount is susceptible to government access," he noted.

And if governments around the world can, at least hypothetically, compromise users, it makes you wonder, as the Berkman Center's Jillian York has, why Facebook hasn't implemented special tools or processes for activists. The biggest issue is that political dissidents often do not want to use their real names in places where activism can get you killed. Facebook has adamantly opposed activists attempts to use pseudonyms.

"We get requests all the time in a few different contexts where people would like to impersonate someone else. Police wanting to go undercover or human rights activists, say," Sullivan said. "And we, just based on our core mission and core product, don't want to allow that. That's just not what Facebook is. Facebook is a place where people connect with real people in their lives using their real identities."

Does Facebook have to go the extra mile to support activists? Sullivan said that preliminary work has been done to create a special complaint reporting process for NGOs and other activists, a move that would address one long-time complaint.

More generally, though, Facebook certainly don't seem to be under any obligations to provide special treatment. But if Facebook really is becoming the public sphere -- and wants to remain central to people's real sociopolitically embedded lives -- maybe they're going to have to think beyond the situational technical fix. Facebook needs to own its position as a part of The Way the World Works and provide protections for political speech and actors.

Because the protests and overthrow of Ben Ali were just the beginning of this story. Hopes are high, but as we've seen so many times in the global south, the exit of one corrupt dictator usually means the entrance of another. To avoid that fate, politically active Tunisians will be using all of the tools at their disposal, including and maybe especially, Facebook. In fact, Rim said, it's already being used to debate how to create a new government and a better Tunisia.

o