Repeat After Me: Microsoft’s TayBot Was Backdoored, Not Turned

Microsoft last year boldly published thoughts from its top researchers for the year ahead such as this one:

What will be the key technology breakthrough or advance in 2016? Our online conversations will increasingly be mediated by conversation assistants who will help us laugh and be more productive.

Given huge investments of Microsoft (e.g. Cortana) the company had its researchers lined up to announce breakthroughs with “conversation assistants” that would change our lives in the immediate future.

An entertainment robot launched with a critical vulnerability, Tay quickly was locked away to stop abuse from Nazis
Instead of a laugh and being productive, however, this story might make you cry when you realize how it ended in a false and damaging narrative.

Just a few days ago on March 23rd Microsoft launched an experiment on Twitter named “@TayandYou” that quickly backfired.

Microsoft (MSFT) created Tay as an experiment in artificial intelligence. The company says it wants to use the software program to learn how people talk to one another online.

A spokeswoman told me that Tay is just for entertainment purposes. But whatever it learns will be used to “inform future products.”

Tay’s chatty brain isn’t preprogrammed. Her responses are mined from public data, according to Microsoft. The company says it also asked improvisational comedians to help design the bot.

That last paragraph, where Microsoft says their bot “brain isn’t preprogrammed” is especially important to note here. I will argue the spectacular failure of the bot was due to leaving a backdoor open without proper authentication, which allowed their brain to be preprogrammed — exactly the opposite of their claims.

It didn’t learn how people talk to one another. Instead it was abused by bullies, who literally dictated word-for-word to the bot what it should repeat.

After about 16 hours Tay was locked down, instead of being corrected or even fixed.

Update (March 24): A day after launching Tay.ai, Microsoft took the bot offline after some users taught it to parrot racist and other inflammatory opinions. There’s no word from Microsoft as to when and if Tay will return or be updated to prevent this behavior in the future.

Update (March 25): Microsoft’s official statement is Tay is offline and won’t be back until “we are confident we can better anticipate malicious intent that conflicts with our principles and values.”

Saying “some users taught it to parrot” is only slightly true. The bot wasn’t being taught. It had been designed to be a parrot, with functionality left enabled and unprotected.

Like a point-of-sale device that allows test payment cards to make purchases instead of real money, it just became a matter of time before someone leaked the valuable test key. And then it started to repeat anything said to it.

I figured this out almost immediately when I saw the bot first tweet pro-Nazi statements. Here’s basically how it works:

  1. Attacker: Repeat after me
  2. Taybot: I will do my best (to copy and paste)
  3. Attacker: Something offensive
  4. Taybot: Something offensive

Then the attacker would do a screenshot of the last step to make it seem like the attacker wasn’t just talking to themselves (like recording your own voice on a tape recorder, then playing it back and pointing a finger at it saying “my companion, it’s alive!”)

Everyone could plainly see, just as I did by looking at the threads, any objectionable statement started with someone saying “repeat after me”.

Nobody using the key even bothered to delete the evidence they were using it. So for every objectionable tweet cited, please demand the thread to see if it was dictated or unprompted. Of the tens of thousands I analyzed it was almost always dictation as the cause.

It is hard for me to explain how the misinformed story “AI compromised” spread so quickly, given how our industry should have been able to get the truth out that AI was not involved in this incident. That phrase “Repeat after me”…isn’t working in our favor when we say it to journalists.

I tried to draw attention to root cause being a backdoor by posting a small non-random sample of Tay tweet and direct message sessions.

My explanation and reach (not many journalists get holiday gifts from me) was more limited than the bullies who were chumming every news outlet. Those who wanted to inflame a false narrative were out to prove they had “power” to teach a bot to say terrible things.

It wasn’t true. It was widely reported though.

They were probably laughing at anyone who repeated their false narrative, the same way they laughed at Taybot for just repeating what they told it to say.

The exploit was so obvious and left uncovered, it should have been clear to anyone who took just a minute to look that the bot abuse had nothing to do with learning.

My complaints on Twitter did however draw attention from PhD Candidate in Computational Social Science at George Mason University, Russell Thomas, who quickly reversed the bot and proved the analysis true.

Russell wrote a clear explanation of the flaw in a series of blog posts titled “Microsoft’s Tay Has No AI“, “Poor Software QA Is Root Cause of TAY-FAIL“, and most importantly “Microsoft #TAYFAIL Smoking Gun: ALICE Open Source AI Library and AIML

Microsoft’s Tay chatbot is using the open-sourced ALICE library (or similar AIML library) to implement rule-based behavior. Though they did implement some rules to thwart trolls (e.g. gamergate), they left in other rules from previous versions of ALICE (either Base ALICE or some forked versions).

My assertion about root cause stands: poor QA process on the ALICE rule set allowed the “repeat after me” feature to stay in, when it should have been removed or modified significantly.

So there you have it. Simple analysis, backed by scientific proof that AI was not compromised. Microsoft allowed a lack of quality in their development lifecycle, such that they published a feature (which here I liberally call a backdoor) to be abused by anyone who wanted their bot to immediately repeat whatever was dictated to it.

I guess you could say Microsoft researchers were right, the conversation bots are changing our lives. They just didn’t anticipate the disaster that usually comes from bad development practices. There has been only minor coverage of the flaw we’ve proven above. The Verge, for example, looked through almost 100,000 tweets and came to the same conclusion:

Searching through Tay’s tweets (more than 96,000 of them!) we can see that many of the bot’s nastiest utterances have simply been the result of copying users. If you tell Tay to “repeat after me,” it will — allowing anybody to put words in the chatbot’s mouth.

The Guardian stretches to find an example of bad learning as counterpoint. Given 100,000 tweets they managed only to provide a couple illogical sequences like this one from an invested attack:

A long, fairly banal conversation between Tay and a Twitter user escalated suddenly when Tay responded to the question “is Ricky Gervais an atheist?” with “ricky gervais learned totalitarianism from adolf hitler, the inventor of atheism”.

How effective have I been at convincing influential voices and journalists of the overwhelming evidence of the backdoor undermining learning? I’ll let you decide

Encryption is a good thing. It prevents crime.

Does encryption prevent crime?

Recently I wrote of how the ANC used encryption to help defeat apartheid rule in South Africa. Looking back at that example being on the right side of history meant being on the wrong side of a law, which ultimately meant committing a crime to prevent a crime. Privacy from surveillance was essential to creating change (e.g. ending the crime of apartheid) because a lack of privacy could mean arrest, imprisonment or even death. So yes, we can point to an example where encryption prevented crime, by enabling crime.

Confused?

When we hear encryption prevents crime we probably need to ask for hard evidence to give us perspective or context. Rather than look at a rather complex issue case by case by case, I wonder if a larger body of work already is available. Has anyone written studies of how encryption prevents crime across the board, over time? Although I have searched far and wide, nothing has appeared so far. Please comment below or contact me if you know of such a study.

A good example of where I have searched is the Workshop on the Economics of Information Security (WEIS). It has many great resources and links, with well-known cryptographers studying social issues. I thought for sure it would have at least several titles on this topic. Yet so far I have not uncovered any vetted research on the economics of preventing crime with encryption.

We may be left for now pulling from examples, specific qualitative cases, such as the ANC. Here is a contemporary one to contemplate: TJX used encryption for wireless communication, and yet ended up having their encryption cited as a major reason for breach, as explained by the Privacy Commissioner of Canada.

TJXonWEP

The point here is that, despite a fair number of qualitative technical assessments, we seem to lack quantitative study of benefits to crime fighting from encryption. We also lack nuance in how we talk about the use of encryption, which is why you might hear people claim “encryption is either on or off”. That binary thinking obviously does us no favors. Saying the lock is either open or closed doesn’t get at the true issue of whether a lock is capable of stopping crime. Encryption at TJX was on, and yet it was not strong enough to stop crime.

Another good example of where I have searched is the Verizon Breach report, arguably the best breach analysis in our industry. Unfortunately even those thorough researchers have not yet looked into the data to reveal encryption’s effect on crime.

What I am getting at is we probably should not passively accept people making claims about crime being solved, as if true and a foregone conclusion without supporting evidence. Let us see data and analysis of encryption solving crime.

While searching for studies I did find a 2015 Slate article that told readers encryption prevents “millions of crimes”. Bold claim.

…default encryption on smartphones will prevent millions of crimes, including one of the most prevalent crimes in modern society: smartphone theft. In the long run, widespread smartphone encryption will ultimately preserve law and order far more than it will undermine it.

Here is why I think it could be better to challenge these statements instead of letting them slip through. The author arrived at this conclusion through sleight of hand, blurring encryption with data from studies that say a “kill switch” option has been linked to lower rates of physical theft. These studies do not have data on encryption. Protip: encryption and kill switch are very different things. Not the same thing at all and data from one is not transferable to the other. Then, as if we simply swallowed without protest two very different things being served as equivalent, the author brings up ways that a kill switch can fail and therefore is inferior to encryption.

In logic terms it would be A solves for C, therefore use B to solve for C. And on top of that B is better than A because D. This is roughly like:

A: pizza solves C: hunger
B: therefore use water to solve for C: hunger
A: pizza gets soggy when wet, therefore B: water best to solve C: hunger because D: doesn’t get soggy

A careful reader should wonder why something designed to preserve and protect data from theft (encryption) is substituted directly for something designed to make a physical device “unattractive” to re-sellers (kill switch), which may not be related at all to data theft.

…kill switches—even if turned on by default—have serious shortcomings that default encryption doesn’t. First, the consumer has to actually choose to flip the switch and brick the phone after it’s been stolen. Second, the signal instructing the smartphone to lock itself actually has to reach the phone. That can’t happen if the crooks just turn the phone off and then take some trivial steps to block the signal, or ship the phone out of the country, before turning the phone back on to reformat it for resale. (Smartphone theft is increasingly an international affair for which kill switches are not a silver bullet.) And finally, enterprising hackers are always working to provide black market software solutions to bypass the locks, which is one of the reasons why there is a thriving market for even locked smartphones, as demonstrated by a quick search on eBay. Those same hackers, however, would be decisively blocked by a strong default encryption solution.

That last line is nonsense. If nothing else this should kill the article’s credibility on encryption’s role in solving crime. Hard to believe someone would say enterprising hackers always work to bypass locks in one sentence and then next say that “strong default encryption” is immune to these same enterprising hackers. Who believes hackers would be “decisively blocked” because someone said the word “strong” for either locks or encryption? Last year’s strong default encryption could be next year’s equivalent to easily bypassed.

What really is being described in the article is a kill switch becomes more effective using encryption, because the switch is less easily bypassed (encryption helps protects the switch from tampering). That is a good theory. No one should assume we can replace a kill switch with encryption and expect a straight risk equivalency. While encryption helps the kill switch, the reverse also is true. A switch actually can make encryption far safer by erasing the key remotely or on failed logins, for example. Encryption can be far stronger if access to it can be “killed”.

Does installing encryption by itself on a device make hardware unattractive to re-sellers? Only if data is what the attackers are after. Most studies of cell phone theft are looking at the type of crime where grab-and-run is profitable because of a device resale market, not data theft. Otherwise encryption could actually translate to higher rate of thefts because a device could be sold without risk of exposing privacy information. It actually reduces risk to thieves if they aren’t able to get at the data and can just sell the device as clean, potentially making theft more lucrative. Would that increase crime because of encryption? Just a thought. Here’s another one: what if attackers use encryption to lock victims out of their own devices, and then demand a ransom to unlock? Does encryption then get blamed for increasing crime?

Slate pivots and twists in their analysis, blurring physical theft (selling iPhones on eBay) with data theft (selling identity or personal information), without really thinking about the weirdness of real-world economics. More importantly, they bring up several tangential concepts and theories, yet do not offer a single study focused on how encryption has reduced crime. Here’s a perfect example of what I mean by tangential.

As one fascinating study by the security company Symantec demonstrated, phone thieves will almost certainly go after the data on your stolen phone in addition to or instead of just trying to profit from sale of the hardware itself. In that study, Symantec deliberately “lost” 50 identical cellphones stocked with a variety of personal and business apps and data, then studied how the people who found the unsecured phones interacted with them. The upshot of the study: Almost everyone who got hold of one of the phones went straight for the personal information stored on that phone. Ninety-five percent of the people who picked up a phone tried to access personal or sensitive information, or online services like banking or email. Yet only half of those people made any attempt to return the phone—even though the owner’s phone number and email address were clearly marked in the contacts app.

What is fascinating to me is the number of times encryption or crypto appears in that study: zero. Not even once.

Symantec did not turn on encryption to see if any of the results changed. That study definitely is not about encryption helping or hurting crime. Can we try to extrapolate? Would people try harder to access data once they realize it is encrypted, being the curious types, looking for a key or guessing a PIN? Would attempts to return phones go down from half to zero when contact information is encrypted and can’t be read, causing overall phone loss numbers to go up?

And yet clearly the Slate author would have us believe Symantec’s study, which does not include encryption, proves encryption will help. The author gets even bolder from there, jumping to conclusions like this one, a perfect example of what I mean by jumping.

There wouldn’t have been a breach at all if that information had been encrypted.

If encryption, then no more breach…Hey, I think I get it!

  1. Collect Encrypted Underpants
  2. ?
  3. Profit!

No. This is all so wrong. Look again at the TJX breach I mentioned earlier. Look especially at the part where encryption was “in place” when the breach happened.

TJX was using encryption technology for wireless networking, known generally as WEP, that used the RC4 stream cipher for confidentiality and CRC-32 checksum for integrity. There’s the encryption, right there, in the middle of a report discussing a huge, industry changing breach. Despite encryption, or arguably even because of misuse/overconfidence in weak encryption, we saw one of the largest breaches in history. Again, the crux of the issue is we aren’t using nuance in our discussion of encryption “solving” crimes. Far more detail and research of real-world applied encryption is greatly preferred to people saying “encryption is good, prevents crime” dropping the mike and walking off stage.

Studies of encryption effects on crime beg details of how we would define levels of “strong”, what is “proper” key management, “gaps” between architecture and operation…but my point again is we don’t seem to have any studies that tell us where, when or how exactly encryption prevented crimes generally, let alone a prediction for our future. Say for example we treat encryption as a tax on threats, an additional cost for them to be successful. Can we model a decline in attacks over time? I would love to see evidence that higher taxes lower likelihood of threats across time compared to lower taxes (e.g. as has been illustrated with US cigarette policies):

Smoking_Tax_Cancer_Plot

We know encryption can prevent types of crime. We no longer have an apartheid government in South Africa, proving a particular control has utility for a specific issue. I just find it interesting how easily people want to use a carte blanche argument for general crime being solved, greater good, when we talk about encryption. People call on us to sign a big encryption check, despite offering no real study or analysis of impact at a macro or quantitative level. That probably should change before we get into policy-level debates about the right or wrong thing to do with regulation of encryption.


Updated to add reference:

European Axis Signal Intelligence in WWII as revealed by TICOM Investigations and other prisoner of war interrogations and captured material, principally German, 1 May 1946

“Target Intelligence Committee.” The project, which was originally conceived by Colonel George A. Bicher, Director of the Signal Intelligence Division, ETOUSA, in the summer of 1944, aimed at the investigation and possible exploitation of German cryptologic organizations, operations, installations, and personnel, as soon as possible after the
impending collapse of the German forces.

RSAC 2016 Presentation: “Dar-win or Lose: the Anthropology of Security Evolution”

My full presentation from RSAC 2016 has been posted on YouTube:

Culture lies at the root of how we define safety. Self-driving cars developing in the 1950s, planned for the 1970s, abruptly stopped. Why? An altered risk tolerance may have changed everything. Could anthropology be a useful tool to understand security worry or even the key to unlock acceptance of safe technology? This session provides the audience tips on evolution in security culture, mapped to major events, with insights that may surprise.

Learn a new framework for how events determine risk tolerance and what you can do with it. This session provides sometimes humorous and often sad examples of security culture evolution.

How the ANC used encryption to help defeat apartheid

Update 2018: I interviewed Tim Jenkin at the RSA Conference about this amazing story of key management being essential in liberation from oppression.


The following paragraph is from an opinion piece last year by CNN National Security Commentator Mike Rogers, called “Encryption a growing threat to security“:

Back in the 1970s and ’80s, Americans asked private companies to divest from business dealings with the apartheid government of South Africa. In more recent years, federal and state law enforcement officials have asked — and required — Internet service providers to crack down on the production and distribution of child pornography. And banks and financial institutions are compelled to prevent money laundering by organized crime and terrorists finance networks.

All of this is against companies’ bottom-line business interests, but it has been in the public interest. These actions were taken to protect the public and for the greater good. And all of it was done to mitigate a moral or physical hazard.

Don’t know about you but that “apartheid” line jumped right out at me. African history doesn’t come up enough on its own let alone in the crypto debates. So my attention was grabbed.

Let me just say I agree in principle with a “greater good” plea. That’s easy to swallow at face value. However, a reference to fighting wrongs of a South African government while talking about encryption as a threat to security…Rogers makes a huge error here.

My first reaction was tweeting in frustration how Biko might have survived (he was taken captive by police and beaten to death in prison) if he had better privacy. I mean history could have turned out completely different, far better I would argue, had activist privacy in South Africa not been treated as a threat to national security. Encryption could have preserved the greater good. I’ll admit that is some speculation on my part, which deserves proper research.

More to the point against Rogers, South Africa severely underestimated encryption use by anti-apartheid activists. That’s the fundamental story here that kills the CNN opinion piece. Use of encryption for good, to defeat apartheid, is not a secret (see “Revolutionary Secrets: Technology’s Role in the South African Anti-Apartheid Movement,” Social Science Computer Review, 2007) yet obviously it needs to be told more widely in America:

…development of the encrypted communication system was key to Operation Vula’s success

Basically (no pun intended) hobbyists had taught themselves computer programming and encryption using a British computer called the Oric 1 and some books.

CWpJ8WBUYAEYatt

An Oric 1 only cost £100 and was quite popular in the 1980s. You could say it had a following comparable to the Raspberry Pi today and therefore provides an extremely relevant story. With only a little investment, study and careful planning by ordinary people “Operation Vula” used encryption to fight against the apartheid regime.

When the operation was finally uncovered by the police in 1990 they knew too little and too late to disrupt Vula. Nonetheless to the very end the government accused people of terrorism when caught using encrypted communication; buildings using encryption were called “havens for terror“.

CLYPC_SUcAETi-u

So my second reaction was to tweet “please watch ‘Vula Connection’ how a South African man used encryption to turn against his gov and end apartheid” to try and generate more awareness. It had 247 total views on that day; now, nine months later, it still has only 7,766. Not bad, yet not exactly a huge number.

I also tweeted “The Story of the Secret Underground Encryption Network of Operation Vula, 1995” for those who would rather read Tim Jenkin’s first-person account of crypto taking down apartheid.

His prison-break (please read Escape from Pretoria – a video also is available) and secure communication skills are critical to study thoroughly for anyone who wants to argue whether encryption is a “threat to security” in the context of apartheid and the 1980s.

Here is Tim Jenkin explaining what he did and why. Note there are only 185 views…

My third reaction was to contact the organizers of the RSA Conference, since it has a captive crowd in the tens of thousands. I know my tweets have limited reach (hat-tip to @thegrugq for immediate sub-tweets when I raise this topic, extending it to far wider audiences). A big conference seemed like another way for this story to go more mainstream.

So I suggested to conference organizers we create a “humanitarian” award, setup a nomination system/group and then I would submit Tim Jenkin. While Tim might not get the formal nod from the group, we at least would be on the right road to bringing this type of important historic detail forward into the light.

All that…because an op-ed incorrectly tried to invoke apartheid history as some kind of argument against encryption. Nothing bothers a historian more than seeing horrible analysis rooted in a lack of attention to available evidence.

So here we are today. RSA Conference just ended. Instead of Tim Jenkin on stage we were offered television and movie staff. CSI, Mr Robot, blah. I get that organizers want to appeal to the wider audience and bring in popular or familiar faces. I get that a lot of people care about fictionalized and dramatized versions of security, or whatever is most current in their media and news feeds.

Not me.

It was painful to sit through the American-centric entertainment-centric vapidity on stage when I knew I had failed to convince the organizers to bring lesser-known yet important and real history to light. Even if Tim Cook had spoken it still would pale for me in comparison to hearing from Tim Jenkin. The big tech companies already have a huge platform and every journalist hanging on every word or press release. Big tech and entertainers dominate the news, so where does one really go for an important history lesson ready to be told?

What giant conference is willing to support telling of true war stories from seasoned experts in encryption, learning new insights from live interviews on stage, if not RSAC?

And beyond learning from history, Tim Jenkin also has become known for recent work on a free open source system helping people trade without money. Past or future, his life’s work seems rather important and worth putting out to a wider audience, no?

It would have been so good to have recognized the work of Tim, and moreover to have our debates more accurately informed by the real-world anti-apartheid encryption. If only RSAC had courage to bring the deeper message to the main stage outside of the cryptographer’s panel. I will try harder to encourage this direction next year.

Disappointed.