Mandiant is reporting that analysis of vCenter logs shows backdoors being installed for at least two years without detection.

While publicly reported and patched in October 2023, Mandiant has observed these crashes across multiple UNC3886 cases between late 2021 and early 2022, leaving a window of roughly a year and a half that this attacker had access to this vulnerability. Most environments where these crashes were observed had log entries preserved, but the “vmdird” core dumps themselves were removed. VMware’s default configurations keep core dumps for an indefinite amount of time on the system, suggesting the core dumps were purposely removed by the attacker in an attempt to cover their tracks.

The lines Mandiant offers for an example are pretty clear, and beg a question of alarms, as well as immutability and availability of core dumps.

2022-01-01T01:31:55.419+00:00| | I125: Notify vMon about vmdird dumping core. Pid : 1558 

2022-01-01T01:31:55.421+00:00| | I125: Successfully notified vMon.

2022-01-01T01:31:55.927+00:00| | I125: Successfully generated core file.

Dumping core is an indicator the environment is untrustworthy. VMware vCenter dumping the LDAP Database “vmdird” core screams security danger. Mandiant illustrates that after the core dump the vpxuser credentials for all ESXi hosts were sent in clear text to the attacker, which sounds very much like what Pentera was ringing alarms about back on March 29, 2022.

Another table is the table ‘vpx_host’ containing details for a user called ‘vpxuser’ and its password phrase. The vpx_host table holds a record for each managed ESXi, each containing a user called “vpxuser” and a unique password phrase. So we retrieved the password phrase, using the command:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc -w -c ‘SELECT user_name, password FROM vc.vpx_host;’

[…]

Once decrypted, the compromised root account vpxuser confirms complete takeover of the ESXi server and a new zero-day is born.

To be fair vpxuser is a non-root account, used in ESXi for privilege elevation. Thus the high-privilege credential leak was used by attackers to install their VirtualPIE/VirtualPITA backdoors into all the ESXi.

Anyone remember those two backdoor names from the VMware report on September 29, 2022?

Mandiant found no evidence that a vulnerability in a VMware product was exploited to gain access to ESXi during their investigations. They have named the malware artifacts as VirtualPITA (ESXi & Linux), VirtualPIE (ESXi), and VirtualGATE (Windows).

Oops, I guess? This all reads to me like Mandiant now sees better what they and VMware missed at least a year ago.

The VMware public advisories on October 24 and 25, 2023 describing VMSA-2023-0023 (Trend Micro credited for the discovery), included a claim there was no known exploitation campaign tied to the critical (9.8) severity vulnerability.

How hard did they look?

The Mandiant report on January 19, 2024 completely blew up this understanding after re-examining vCenter logs going back two years, forcing VMware to update their FAQ.

As of January 18, 2024 VMware is aware of exploitation “in the wild.”

However, note VMware record keeping on this FAQ fails to mention that significant change:

Changelog

2023-10-24, 1930 PDT (-0700): Initial publication.

2023-10-25, 11:50 PDT (-0700): Updates to improve clarity.

2023-10-31, 0930 PDT (-0700): Updates to the VMware Cloud messaging.

Meanwhile the VMware advisory text for VMSA-2023-0023 says they were notified on January 17, a day earlier.

Issue Date:
2023-10-25

Updated On:
2024-01-17

VMware has confirmed that exploitation of CVE-2023-34048 has occurred in the wild.

Log management failure in multiple ways here. This is not how we used to run security at VMware.

Someone was asking me just how many times Okta has been breached recently. Upon looking around I realized there isn’t a simple place to answer such a question.

Is there? On November 29, 2023 Okta published their “October Customer Support Security Incident” but it doesn’t link to any list of previous incidents. Notably, Okta’s official “security advisories” doesn’t seem to include breaches of Okta.

Here’s a few easy examples rattling around the web:

• Nov 2023: “Okta security breach much worse than originally disclosed – all customers’ data potentially affected”
• Nov 2023: “Okta tells 5,000 of its own staff that their data was accessed in third-party breach”
• Sep 2023: “4 Okta customers compromised in social engineering attacks”
• Dec 2022: “Okta confirms another breach after hackers steal source code”, its “fourth breach of the year“.
• Mar 2022: “Okta says 366 customers potentially affected in data breach” where “‘Two Months Is Too Long’: Tenable CEO Slams Okta’s Breach Response”
• Oct 2019: “Okta SRE Pleads Guilty to Stealing IDs to Violate Women’s Privacy [before being hired by Okta]”

And then of course, I have to add in for good measure:

• Aug 2011: “Cloud security different, says Okta”

I wrote that warning in 2011 and here we are twelve long years later looking at the results. Customer Identity and Access Management (CIAM) is now a market segment rife with risks associated with their use:

1. CIAM are attractive targets for attack. Proprietary and “exit-barrier” providers become especially juicy targets as they expect to get away with low safety in proportion to how hard they can make it for their customers to leave them.
2. CIAM can be overly centralized in a way that impacts an entire user access ecosystem, challenging availability architectures that depend on “blast radius” concepts and data boundaries.

Short list, I know.

But let’s be honest here and say what has been true for more than a decade: If you have “unusual” behavior exploiting your CIAM it’s going to come down to usual observations such as where a user is coming from, whether a string of failures concludes with any success (e.g. brute force versus fat-finger events), and how much authorization longevity or reuse is going on (e.g. same session ID with a rotating user agent or different origins).

Okta should publish all their breach reports in one place with all the explanations.

I was thinking about Trevor Noah’s interview by Stephen Colbert, where they discussed the psychological affects of a national anthem, and it reminded me of this research on flag waving.

“Flags are tricky,” Kemmelmeier says. “If you allude to a collective and say, ‘This is us,’ there’s always somebody that’s not included.”

Decades of research has demonstrated that simply assigning a symbol, such as a flag, to an arbitrary group can cause a hardening of attitudes. A study published in 2016 by social psychologists Shannon Callahan and Alison Ledgerwood found that people perceived others as less warm and more threatening if the group was assigned a flag. “A consistent picture emerges,” writes David Smith, a psychology lecturer from Robert Gordon University in Aberdeen. “Flags bond insiders but make outsiders feel unwelcome.”

Here’s the interview where the comedians joke about America being the only country that plays a national anthem all the time when foreigners aren’t present… presumably to target some insiders so they will be perceived more like outsiders.

What they’re poking at is a nativist (anti-immigrant) sentiment of “America First”, which in 1915 became an official slogan of the flag-waving KKK (e.g. blueprint for Nazi Germany). The message was if you don’t renounce any/all other identities and declare yourself “America First” you are to be tortured, lynched and mutilated for all to see what happens to any “outsiders” trying to live in America.

Food for thought, on a regular day anywhere in Germany you will never see a national flag waving in Germany. However, with extreme irony in America you will see German flags waving on houses in a way they would never attempt if they were still living in Germany.

And fruit flies like a banana.

One of the remarkable things about studying the gaslight methods of Hitler, is how similar they are to Elon Musk.

Let’s set the stage properly by acknowledging that Musk’s family emigrated from Canada because in WWII they wanted to help promote a white nationalist genocidal dream of an Apartheid South Africa. They idolized Hitler and his methods of coercion and control.

Hitler, for example, would issue a decree that guillotines are barbaric and you’re not supposed to use them, right before he’d order them installed in all his prisons to murder tens of thousands of his political opponents.

Elon Musk thus regularly says things like he hates the people who hate. Get it? Dictators see themselves as above the law.

Right now he is promoting the brand of hate he likes (Nazism), while demanding others stop promoting hate he dislikes (his perceived opponents).

“You’re not supposed to be fostering hate,” he pointed out.

As owner of the social media platform X [formally known as Twitter], Musk — a self-described “free-speech absolutist” [who censors all speech he dislikes]— has faced heavy scrutiny for allowing users to share sometimes bigoted content [all of the time].

In November, he was condemned by the White House for commenting on an antisemitic post that blamed Jewish people for pushing hatred against white people, writing, “You have said the actual truth.”

“You’re not supposed to be fostering hate” should be read as…

Elon Musk wants to be in sole control of who gets to foster what hate and when. His love of Nazism is unmistakable, right down to the very Hitleresque gaslight tactic.

He will say he is opposed to any mention of genocide, while also saying he’s opposed to all censorship and treats genocide as a matter of different perspectives.

Elon Musk stated that there are “two sides to China’s repression of the Uyghur people”. …stated Mr. Hidayet Oghuzhan, President of IUETO. “Any company conducting business in East Turkistan is complicit in the ongoing genocide. However, Elon Musk’s actions are particularly despicable.”

Elon Musk apparently benefits directly from the genocide he coincidentally pretends he is unfamiliar with.

Nazis love visiting Auschwitz. They really do. Trolls of white hate love Elon Musk for exactly the kind of negative performance he just put on for them.

What better stage can Elon Musk possibly find to promote his particular brand of hate?

…I see almost no antisemitism. […] Two-thirds of my friends are Jewish. I have twice as many Jewish friends as non-Jewish friends.

He says he sees no antisemtitism while walking around saying the most antisemitic things. Oh so clever.

He goes to a genocide memorial to say he knows nothing about the past (“somewhat naive about all this”), and he sees no harms today. Maybe he should have worn a giant Pepe frog pin on his jacket to show his true campaign.

Instead he used an obvious antisemtitic phrase to put a big Nazi bow on top.

Such edge. So hateful. Very gaslit.