Easy BlueTooth Car Hack: “Press OK to Continue”

Looking at a brand new vehicle console interface for BlueTooth connections we found it prompted the user to select a device name, yet used a limited visual space. The prompt, right in front of the driver on the center console, asks (changed slightly to mask offending vehicle manufacturer):

Would you like to connect…

Then the device name gets inserted immediately after. This led to the natural question whether we could dictate behavior instead of asking the user to make a decision.

We changed a phone name to “Press OK to Continue” put phone into discovery/connect mode and waited in a parking lot. Soon after we had a rogue connection to a car, as a driver thought “Press OK to Continue” was a prompt, not the device name.

That’s a bit of social engineering to fool the human, testing human vulnerability to formatting. To check the device itself before human, you could similarly change the device name to odd characters and test non-human vulnerability to string formats.

American Pro-Slavery History Markers

Charlotte, North Carolina, has a “Confederate” history marker that I noticed while walking on my way into meetings at Bank of America headquarters.

It is in need of major revision, if not removal.

Let me start this story at the end. My searches online for more information eventually found a “NC Markers” program with an entry for L-56 CONFEDERATE NAVY YARD.

Closer to the end of the war…tools and machinery from the yard were moved from Charlotte to Lincolnton. Before the yard could be reassembled and activated in Lincolnton, the war ended. After the war the yard’s previous landowner, Colonel John Wilkes, repossessed the property, for which the Confederate government had never paid him. Where the Confederate Navy Yard once operated, he established Mecklenburg Iron Works. It operated from 1865 until 1875 when it burned.

Please note the very vague “the war ended” phrase in the second sentence.

This supposedly historic account seems to obscure the very simple fact that white supremacists lost the war they started to expand slavery.

I find saying “the war ended” to be an extremely annoying attempt to avoid saying who lost.

To make the problem more clear, compare the above L-56 official account with the UNC Charlotte Special Collections version of the same history:

The exact date of the formation of the Mecklenburg Iron Works is unknown, as is ownership of the firm until its purchase in 1859 by Captain John Wilkes. There is evidence, though, that the firm existed as early as 1846. The son of Admiral Charles Wilkes, John was graduated first in his class at the U.S. Naval Academy in 1847. Following a stint in the U.S. Navy, Wilkes married and moved to Charlotte in 1854. Two years after he purchased the iron works, the Confederate government took it over and used it as a naval ordnance depot. After the Civil War, Wilkes regained possession of the Iron Works, which he operated until his death in 1908. His sons, J. Renwick and Frank, continued the business until 1950, when they sold it to C. M. Cox and his associates.

So many things to notice here:

  1. Confederates appropriated a firm in their war to expand slavery, and possession was returned after they lost that war.
  2. There was a Captain John Wilkes, not Colonel, although neither story says for which side he fought. An obituary lists him as U.S. Navy and says he was active during Civil War
  3. Captain John Wilkes was the son of infamous Union Navy Admiral Charles Wilkes, who was given a court-martial in 1864. Was John, son, fighting for the North with father, or South against him?
  4. There is evidence these Iron Works were established long before the Civil War. NC Markers says “as early as 1846”. The Charlotte library says Vesuvius Furnace, Tizrah Forge and Rehoboth Furnace were operating 35 years earlier, with a picture of the Mecklenburg Iron Works to illustrate 1810.(1)
  5. Wilkes was not just “yard’s previous landowner”, he ran an iron works two years before the Confederate government took possession of it. Did he lose it as he went to fight for the North, or did he give it to help fight for the South? Seems important to specify yet no one does. In any case the iron works was pre-established, used during Civil War and continued on afterwards

The bigger question of course is who cares that there is a Confederate Navy yard in Charlotte, North Carolina? Why was a sign created in 1954 to commemorate the pro-slavery military?

Taking a picture of the sign meant I could show it to an executive business woman I met in Charlotte, and I asked her why it was there. She told me “Democrats put up that sign for their national convention”. She gave this very strangely political answer about the Democrats in her very authoritative voice while being completely wrong. And she both seemed opposed to the sign because of who put it up, yet in no way interested in taking it down. She ended with an explanation that there was no mention of slavery because (yelling at me and walking away) “CIVIL WAR WAS ABOUT TAXES, NOT SLAVERY. I KNOW MY HISTORY”.

I found this also very annoying. Apparently white educated elites in North Carolina somehow have come to believe Civil War was not about slavery. She was not the only one to say this.

What actually happened, I found with a little research, was the North Carolina Highway Historical Marker Program started in 1935. They put up the signs, with no mention of Democrats or political conventions, as you can tell from the link I already gave at the start of this post.

Here is the kind of one-sided “history” the program promotes, calling the preservation of the Union by its own forces an “invasion” and then “occupation”.

Historian pro-tip, you aren’t technically an occupying army when you still are in your own country, even if in an attempted secessionist territory. Otherwise we would still say today that American troops are invading North Carolina when they are assigned to Fort Bragg.

Speaking of Fort Bragg it was opened in 1918 (under racist “America First” President Wilson) and stupidly named after the Confederate General Braxton Bragg.

Bragg is said by historians to have been the worst tactician in the entire Civil War, causing major losses through incompetence that led to defeat of the pro-slavery forces.

Aside from perpetuating his racism, what possible reason would North Carolina have to name a U.S. Army fort after such a terrible enemy military leader known for losing battles let alone wantonly shooting his own men?

How could anyone in North Carolina been proud of Bragg?

Anyway, here is how the NC Markers program explains the official purpose of a CONFEDERATE NAVY YARD sign on the street:

For residents the presence of a state marker in their community can be a source of pride

Source of pride.

Honestly I do not see what they are talking about. What are people reading this sign meant to be proud of exactly? Is a failed attempt by pro-slavery military to create a Navy a proud moment? Confederate yards failed apparently because of huge shortages in raw materials and labor, which ultimately were because of failures in leadership. That is pride material?

What am I missing here?

The sign is dated as 1954. Why this date? It was the year the U.S. Supreme Court struck down “separate but equal” doctrine, opening the door for the civil rights movement. It was the year after Wilkes oldest surviving child died. Does a pro-slavery military commemoration sign somehow make more sense in 1954 (city thumbing nose at Supreme Court or maybe left in will of Wilkes last remaining child) than it does in 2016?

A petition at the University of Mississippi to change one of their campus monuments explains the problem with claiming this as a pride sign:

Students and faculty immediately objected to this language, which 1) failed to acknowledge slavery as the central cause of the Civil War, 2) ignored the role white supremacy played in shaping the Lost Cause ideology that gave rise to such memorials, and 3) reimagined the continued existence of the memorial on our campus as a symbol of hope.

[…]

From the 1870s through the 1920s, memorial associations erected more than 1,000 Confederate monuments throughout the South. These monuments reaffirmed white southerners’ commitment to a “Lost Cause” ideology that they created to justify Confederate defeat as a moral victory and secession as a defense of constitutional liberties. The Lost Cause insisted that slavery was not a cruel institution and – most importantly – that slavery was not a cause of the Civil War.

Kudos to the Mississippi campaign to fix bad history and remove Lost Cause propaganda. The North Carolina sign’s 1950s date suggests there might be a longer period of monuments being erected. When I travel to the South I am always surprised to run into these “proud” commemorations of slavery and a white-supremacy military. I am even more surprised that the residents I show them to usually have no idea where exactly they are, why they still are standing or who put them up.

Anyone who knows me well knows I walked into Bank of America and at the start of the meetings demanded an explanation for the sign outside. The response I heard was “what sign, never seen it” followed after the meeting by a call from someone asking how dare I mention the sign in a business meeting.

My response? How dare you put that sign in front of my meeting and tell me I can’t talk about it being a bad one.

At the very least North Carolina should re-write this sign to be accurate, if they can’t do the more obvious fix of removing it.

Here is my helpful suggestion:

MECKLENBURG IRON WORKS: Established here 1810. Pro-slavery militia in 1862 seized the works in a failed attempt to supply a Navy after their defeat in Portsmouth, Va. Liberated from occupation 1865.

That seems fair. The official “essay” of the NC Markers really also should be rewritten.

For example NC Markers wrote:

…in time it began to encounter difficulty obtaining and retraining trained workers

Too vague. I would revise that to “Southerners depended heavily on immigrants and Northerners for shipyard labor. As soon as first shots were fired upon the Union by the South, starting a Civil War, many of the skilled laborers left and could not be replaced. Over-mobilization of troops further contributed to huge labor shortages.

NC Markers also wrote:

…given its location along the North Carolina Railroad and the South Carolina Railroad, it was connected to several seaboard cities, enabling it to transport necessary products to the Confederate Navy

Weak analysis. I would revise that to “despite creating infrastructure to make use of the Confederate Navy Yard it had no worth without raw materials. Unable to provide enough essential and basic goods, gross miscalculation by Confederate leaders greatly contributed to collapse of plans for a Navy”

But most of all, when they wrote “the war ended” I would revise to say “the Confederates surrendered to the Union, and with their defeat came the end of slavery”.

Let residents be proud of ending the pro-slavery nation, or more specifically returning the Iron Works to something other than fighting for perpetuation of slavery.

So here is the beginning of the story, at its end. Look at this sign on the street in Charlotte, next to Bank of America headquarters:

charlotte-pro-slavery-militia-memorial-sign


(1) 1810 – Iron Industry screenshot from Charlotte – Mecklenburg Library
1810-IronIndustry-Mecklenburg

Elevator Social Engineering

I’ve spent years fiddling with social engineering at a bank of elevators. At first it was just part of the job (getting past security) and now it’s become something more of an analytic game.

Let’s say you have six doors, where you have to push the button and wait for one to open. A crowd forms, three, five, maybe even seven people. Should you try to jump in first when the door opens?

No.

Time and again I find it better to step towards the door and hold it open until it’s completely full. Everyone else will move sheepishly towards the first door they see, or at least the closest open one. Encourage this behavior and help as many people as possible quickly squeeze into a tiny box together. Maybe even push all the floor buttons for them. Then jump out and let the doors close without you inside. The more you pushed in the better.

Pat yourself on the back, push the elevator button again and step alone into the next elevator that opens its doors. Of course the congratulations really depends on how well you estimated flow of arriving passengers and where they’re going (could be a group together choosing a single floor).

It’s a great game of allocation logistics that soon will be replaced by computers assigning people to elevators using basic math. Enjoy it while you can.

Repeat After Me: Microsoft’s TayBot Was Backdoored, Not Turned

Microsoft last year boldly published thoughts from its top researchers for the year ahead such as this one:

What will be the key technology breakthrough or advance in 2016? Our online conversations will increasingly be mediated by conversation assistants who will help us laugh and be more productive.

Given huge investments of Microsoft (e.g. Cortana) the company had its researchers lined up to announce breakthroughs with “conversation assistants” that would change our lives in the immediate future.

An entertainment robot launched with a critical vulnerability, Tay quickly was locked away to stop abuse from Nazis
Instead of a laugh and being productive, however, this story might make you cry when you realize how it ended in a false and damaging narrative.

Just a few days ago on March 23rd Microsoft launched an experiment on Twitter named “@TayandYou” that quickly backfired.

Microsoft (MSFT) created Tay as an experiment in artificial intelligence. The company says it wants to use the software program to learn how people talk to one another online.

A spokeswoman told me that Tay is just for entertainment purposes. But whatever it learns will be used to “inform future products.”

Tay’s chatty brain isn’t preprogrammed. Her responses are mined from public data, according to Microsoft. The company says it also asked improvisational comedians to help design the bot.

That last paragraph, where Microsoft says their bot “brain isn’t preprogrammed” is especially important to note here. I will argue the spectacular failure of the bot was due to leaving a backdoor open without proper authentication, which allowed their brain to be preprogrammed — exactly the opposite of their claims.

It didn’t learn how people talk to one another. Instead it was abused by bullies, who literally dictated word-for-word to the bot what it should repeat.

After about 16 hours Tay was locked down, instead of being corrected or even fixed.

Update (March 24): A day after launching Tay.ai, Microsoft took the bot offline after some users taught it to parrot racist and other inflammatory opinions. There’s no word from Microsoft as to when and if Tay will return or be updated to prevent this behavior in the future.

Update (March 25): Microsoft’s official statement is Tay is offline and won’t be back until “we are confident we can better anticipate malicious intent that conflicts with our principles and values.”

Saying “some users taught it to parrot” is only slightly true. The bot wasn’t being taught. It had been designed to be a parrot, with functionality left enabled and unprotected.

Like a point-of-sale device that allows test payment cards to make purchases instead of real money, it just became a matter of time before someone leaked the valuable test key. And then it started to repeat anything said to it.

I figured this out almost immediately when I saw the bot first tweet pro-Nazi statements. Here’s basically how it works:

  1. Attacker: Repeat after me
  2. Taybot: I will do my best (to copy and paste)
  3. Attacker: Something offensive
  4. Taybot: Something offensive

Then the attacker would do a screenshot of the last step to make it seem like the attacker wasn’t just talking to themselves (like recording your own voice on a tape recorder, then playing it back and pointing a finger at it saying “my companion, it’s alive!”)

Everyone could plainly see, just as I did by looking at the threads, any objectionable statement started with someone saying “repeat after me”.

Nobody using the key even bothered to delete the evidence they were using it. So for every objectionable tweet cited, please demand the thread to see if it was dictated or unprompted. Of the tens of thousands I analyzed it was almost always dictation as the cause.

It is hard for me to explain how the misinformed story “AI compromised” spread so quickly, given how our industry should have been able to get the truth out that AI was not involved in this incident. That phrase “Repeat after me”…isn’t working in our favor when we say it to journalists.

I tried to draw attention to root cause being a backdoor by posting a small non-random sample of Tay tweet and direct message sessions.

My explanation and reach (not many journalists get holiday gifts from me) was more limited than the bullies who were chumming every news outlet. Those who wanted to inflame a false narrative were out to prove they had “power” to teach a bot to say terrible things.

It wasn’t true. It was widely reported though.

They were probably laughing at anyone who repeated their false narrative, the same way they laughed at Taybot for just repeating what they told it to say.

The exploit was so obvious and left uncovered, it should have been clear to anyone who took just a minute to look that the bot abuse had nothing to do with learning.

My complaints on Twitter did however draw attention from PhD Candidate in Computational Social Science at George Mason University, Russell Thomas, who quickly reversed the bot and proved the analysis true.

Russell wrote a clear explanation of the flaw in a series of blog posts titled “Microsoft’s Tay Has No AI“, “Poor Software QA Is Root Cause of TAY-FAIL“, and most importantly “Microsoft #TAYFAIL Smoking Gun: ALICE Open Source AI Library and AIML

Microsoft’s Tay chatbot is using the open-sourced ALICE library (or similar AIML library) to implement rule-based behavior. Though they did implement some rules to thwart trolls (e.g. gamergate), they left in other rules from previous versions of ALICE (either Base ALICE or some forked versions).

My assertion about root cause stands: poor QA process on the ALICE rule set allowed the “repeat after me” feature to stay in, when it should have been removed or modified significantly.

So there you have it. Simple analysis, backed by scientific proof that AI was not compromised. Microsoft allowed a lack of quality in their development lifecycle, such that they published a feature (which here I liberally call a backdoor) to be abused by anyone who wanted their bot to immediately repeat whatever was dictated to it.

I guess you could say Microsoft researchers were right, the conversation bots are changing our lives. They just didn’t anticipate the disaster that usually comes from bad development practices. There has been only minor coverage of the flaw we’ve proven above. The Verge, for example, looked through almost 100,000 tweets and came to the same conclusion:

Searching through Tay’s tweets (more than 96,000 of them!) we can see that many of the bot’s nastiest utterances have simply been the result of copying users. If you tell Tay to “repeat after me,” it will — allowing anybody to put words in the chatbot’s mouth.

The Guardian stretches to find an example of bad learning as counterpoint. Given 100,000 tweets they managed only to provide a couple illogical sequences like this one from an invested attack:

A long, fairly banal conversation between Tay and a Twitter user escalated suddenly when Tay responded to the question “is Ricky Gervais an atheist?” with “ricky gervais learned totalitarianism from adolf hitler, the inventor of atheism”.

How effective have I been at convincing influential voices and journalists of the overwhelming evidence of the backdoor undermining learning? I’ll let you decide