When Can You Trust Cloud Providers?

The Raft of the Medusa by Géricault depicts service provider incompetence of 1816: “Crazed, parched and starved, they slaughtered mutineers, ate their dead companions and killed the weakest”

Our first book detailed the infrastructure risks in cloud environments. It gave basic instructions for how to make it safe to build a cloud.

However, I realized right away that a second book would be necessary as I saw operations going awry. People offering data “services” in cloud environments were doing so unethically.

That’s why since 2013 I’ve been working on tangible, actionable solutions to problems in cloud environments like the immoral SRE.

It has been a much harder book to write because The Realities of Securing Big Data crosses many functional lines in an organization from legal to engineering, sales to operations. A long-time coming now, it hopefully will clarify how and why things like this keep happening, as well as what exactly we can do about it:

We recently found that some email addresses and phone numbers provided for account security may have been used unintentionally for advertising purposes. This is no longer happening and we wanted to give you more clarity around the situation: https://help.twitter.com/en/information-and-ads

…and that led to everyone asking an obvious question.

You may remember a very similar incident last year and wonder why nobody at Twitter thought to test their systems to make sure they didn’t have the same security flaws as a safety laggard like Facebook.

Facebook is not content to use the contact information you willingly put into your Facebook profile for advertising. It is also using contact information you handed over for security purposes and contact information you didn’t hand over at all.

Facebook and Twitter, after flashy high-profile CISO hires and lots of PR about privacy, both have sunken to terrible reputations. They rank towards the same levels as Wells Fargo in terms of customer confidence.

Facebook has experienced a tumultuous time due to privacy concerns and issues regarding election interference, ranked 94th. Wells Fargo ranked 96th. The Trump Organization ranked 98th, considered a “very poor” reputation.

The Drum says even the advertising industry is calling out Twitter for immorality and incompetence:

Neville Doyle, chief strategy officer at Town Square, suggested it was “enormously improbable” that Twitter ‘inadvertently’ improved its ad product with the sensitive data, and blasted the tech giant for being either “either immoral or incompetent”. Either way, he said, it was playing “fast and loose with users’ privacy”. Respected ad-tech and cybersecurity expert Dr Augustine Fou, who was previously chief digital officer at media agency Omnicom’s healthcare division, also branded Twitter’s announcement as “total chickenshit”. Last July, the Federal Trade Commission (FTC) fined Facebook $5bn for improperly handling user data, the largest fine ever imposed on company for violating consumers’ privacy.

The technology fixes ahead are more straightforward than you might imagine, as well as the management fixes.

In brief, you can trust a cloud provider when you can verify in detail a specific set of data boundaries and controls are in place, with transparency around staffing authorizations and experience related to delivering services. Over the years I’ve led many engineering teams to build exactly this, so I’m speaking from experience of what’s possible. I’ve stood in customer executive meetings to detail how controls work and why the system was designed to mitigate cloud insider threats, including executives at the highest levels.

You should be especially concerned if management lacks an open and public resume of prior steps taken over years to serve the privacy needs of others, let alone management that lacks the ability to deconstruct how their control architecture was built from the start to serve your best interests.

What has been hard, especially through the years of Amazon’s “predator bully” subscription model being worshiped by sales teams, is keeping safety oriented around helping others. Tech cultures in America tend to cultivate “leaders” that think of innovation as separation; having no way to relate to the people they are serving.

The tone now seems to be changing as disclosures are increasing and we’re seeing exposure of the wrong things done by people who wanted to serve others while being unable to relate to them. Hoarding other people’s assets for self-gain in a thinly-veiled spin to be their “service provider” should never have been the meaning of cloud.

Study Details Racism in LAPD Traffic Stops

Data in a new LA Times report (and posted to github) reveals that despite whites being found with contraband more often, blacks and latinos are stopped far more often to be searched.

…a black person in a vehicle was more than four times as likely to be searched by police as a white person, and a Latino was three times as likely.

Yet whites were found with drugs, weapons or other contraband in 20% of searches, compared with 17% for blacks and 16% for Latinos. The totals include both searches of the vehicles and pat-down searches of the occupants.

The analysis in the report indicates less evidence was used to prompt a search of latinos and blacks than whites. On top of that, after being stopped and searched, whites also saw better treatment and lower arrest rates.

Blacks and Latinos were more than three times as likely as whites to be removed from the vehicle and twice as likely to either be handcuffed or detained at the curb, the Times analysis found.

About 3% of blacks and Latinos stopped by the LAPD were arrested, compared with 2% of whites.

To put it another way, the city is 9% black yet 27% of people being searched are black; the city is 28% white, yet 18% of those being searched are white.

US Administration Fights to Protect Human Trafficking and Disinformation Platforms

The U.S. already has a reputation for its lax approach to infrastructure regulation that “encouraged the spread of disinformation and supported a powerful forum for harassment and bullying”.

Current occupants of the White House are taking that even further.

American infrastructure is said to be getting legal protections against accountability pushed on foreign trade deals, known as adding in Section 230.

Last year, Congress overwhelmingly approved a bill making it possible to sue online platforms for knowingly facilitating sex trafficking. Lawmakers have raised the prospect of creating additional carve-outs for the online sale of opioids. Critics of Section 230 say they are alarmed by the inclusion of its provisions in trade deals.

In other words despite representatives in U.S. government working to protect the world from clear and documented harms, the White House is headed in an opposite direction by trying to instead protect criminal behavior such as child trafficking operating in the U.S..

This relates directly to other recent news that the American cloud service providers often are abused by men operating them to victimize women and children around the world.

Studies repeatedly show “it’s disproportionately women who are targeted” using cloud services and enslaved.

Seventy-six percent of trafficked persons are girls and women and the Internet is now a major sales platform.

Epstein no longer being protected by powerful American men, found dead in his cell and quickly forgotten, may actually mean he was replaced by technology…and that’s why now it is being made untouchable instead of him.

By allowing lawsuits to proceed as one would normally expect, a court would be able to deliberate and find the right balance between freedoms of expression and clear cases of harm.

“The use of Twitter by the defendants to post allegedly defamatory statements cannot subject the plaintiff to the terms of use agreement and the forum selection clause as it would not subject a plaintiff who did not have a Twitter account to the terms of use agreement,” the ruling states.

Okta SRE Pleads Guilty to Stealing IDs to Violate Women’s Privacy

Once again, cloud services very predictably show why they can be less secure than running your own.

We’ve warned for many years of cloud insider abuse like this, using examples from Uber, Google and Facebook.

In many of these cases it’s male engineers in American technology companies using their power and privilege to stalk and abuse women.

The US Department of Justice has posted details of a 34 year old man who is said to have worked at Yahoo.

In pleading guilty, Ruiz, a former Yahoo software engineer, admitted to using his access through his work at the company to hack into about 6,000 Yahoo accounts. Ruiz cracked user passwords, and accessed internal Yahoo systems to compromise the Yahoo accounts. Ruiz admitted to targeting accounts belonging to younger women, including his personal friends and work colleagues. He made copies of images and videos that he found in the personal accounts without permission, and stored the data at his home. Once he had access to the Yahoo accounts, Ruiz admitted to compromising the iCloud, Facebook, Gmail, DropBox, and other online accounts of the Yahoo users in search of more private images and videos. After his employer observed the suspicious account activity, Ruiz admitted to destroying the computer and hard drive on which he stored the images.

That last sentence is concerning for anyone who has done digital forensics. How was Ruiz tipped off that he was being observed by an internal investigations team?

He wasn’t just a software engineer, he was a Site Reliability Engineer (SRE). And he wasn’t just a Yahoo engineer

LinkedIn profile of Reyes Ruiz, identity thief hired as SRE by Okta

That career path reveals a far worse story than what is being reported right now.

A SRE is a person with deep access inside the cloud provider. They are trusted with the most sensitive data because, in theory, without giving them access a system could become unreliable or go offline.

For example, here’s a single line command in virtual (VMware) environments that exports a copy of an entire server. In a disaster (planning) scenario it could be essential to keeping services running:

Copy-DatastoreItem vmstore:\Datacenter01\StorageArray01\DBNodes\* C:\SREisGod\StolenUserSecrets

Imagine instead, as you can see from the destination path name at the end of that line, any evil SRE just wants to steal ALL the data. SRE staff literally have keys to any kingdom trusting their employer. Even if the data was stored encrypted, using this command it’s decrypted by design.

I’ve repeatedly designed systems to protect against exactly this kind of insider threat and customers need to explicitly ask for proof that one exists. This is a disaster for both Okta and Yahoo if they cannot account for SRE access on their systems, particularly during the hours Ruiz was working.

His eight months at Okta, a widely used identity management company, could be an even bigger problem than Yahoo. Although to be fair the timing is interesting for both cases. Yahoo in 2007 when he joined was the biggest identity provider in the world. In 2018 Okta was claiming to be the leader in this space.

It looks like Okta apparently fired him as soon as the indictments were unsealed that detailed his long-term abuse of being a privileged SRE to game identity management. What Okta hasn’t said is whether they’ve concluded an internal investigation of all his access to identity as an SRE.

This is huge. I can’t overstate enough that an identity management cloud provider, holding the secrets of millions of people, hired an identity thief. It’s like saying a bank hired a bank robber to guard their safe.

Given inside knowledge and access at the service provider he allegedly cracked passwords of thousands of young women, including those he knew and worked with, in order to steal their images. Then he used their identity information to pivot through their cloud accounts that shared the same password to steal more images.

Two lessons here:

One. Okta is a core identity management company that hired a predator who clearly joined companies to commit crimes. Anyone using Okta or a similar service needs to be prepared for this level of insider threat being reported. Although we can pressure Okta on reasons screening didn’t block this hire, we can’t assume screens will be perfect and instead should demand they prove his actions were limited and detected.

Two. Re-use of passwords is what made one evil cloud staff member able to access so many other cloud accounts. Impersonation was possible by Ruiz because users didn’t setup different passwords on each cloud service. Password managers are free and a baseline requirement for users today. Also multi-factor authentication (MFA) would have made SRE theft of user secrets less effective and should be considered another baseline requirement (caveat: nothing is perfect. see new FBI warning on MFA bypass).

There’s a third point about avoiding tipping off suspects during investigations, and preserving evidence, but we don’t have enough details yet on why or how badly that security team at Yahoo was compromised.