Mandiant is reporting that analysis of vCenter logs shows backdoors being installed for at least two years without detection.
While publicly reported and patched in October 2023, Mandiant has observed these crashes across multiple UNC3886 cases between late 2021 and early 2022, leaving a window of roughly a year and a half that this attacker had access to this vulnerability. Most environments where these crashes were observed had log entries preserved, but the “vmdird” core dumps themselves were removed. VMware’s default configurations keep core dumps for an indefinite amount of time on the system, suggesting the core dumps were purposely removed by the attacker in an attempt to cover their tracks.
The lines Mandiant offers for an example are pretty clear, and beg a question of alarms, as well as immutability and availability of core dumps.
2022-01-01T01:31:55.419+00:00|| I125: Notify vMon about vmdird dumping core. Pid : 1558 2022-01-01T01:31:55.421+00:00| | I125: Successfully notified vMon. 2022-01-01T01:31:55.927+00:00| | I125: Successfully generated core file.
Dumping core is an indicator the environment is untrustworthy. VMware vCenter dumping the LDAP Database “vmdird” core screams security danger. Mandiant illustrates that after the core dump the vpxuser credentials for all ESXi hosts were sent in clear text to the attacker, which sounds very much like what Pentera was ringing alarms about back on March 29, 2022.
Another table is the table ‘vpx_host’ containing details for a user called ‘vpxuser’ and its password phrase. The vpx_host table holds a record for each managed ESXi, each containing a user called “vpxuser” and a unique password phrase. So we retrieved the password phrase, using the command:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc -w
-c ‘SELECT user_name, password FROM vc.vpx_host;’ […]
Once decrypted, the compromised root account vpxuser confirms complete takeover of the ESXi server and a new zero-day is born.
To be fair vpxuser is a non-root account, used in ESXi for privilege elevation. Thus the high-privilege credential leak was used by attackers to install their VirtualPIE/VirtualPITA backdoors into all the ESXi.
Anyone remember those two backdoor names from the VMware report on September 29, 2022?
Mandiant found no evidence that a vulnerability in a VMware product was exploited to gain access to ESXi during their investigations. They have named the malware artifacts as VirtualPITA (ESXi & Linux), VirtualPIE (ESXi), and VirtualGATE (Windows).
Oops, I guess? This all reads to me like Mandiant now sees better what they and VMware missed at least a year ago.
The VMware public advisories on October 24 and 25, 2023 describing VMSA-2023-0023 (Trend Micro credited for the discovery), included a claim there was no known exploitation campaign tied to the critical (9.8) severity vulnerability.
How hard did they look?
The Mandiant report on January 19, 2024 completely blew up this understanding after re-examining vCenter logs going back two years, forcing VMware to update their FAQ.
As of January 18, 2024 VMware is aware of exploitation “in the wild.”
However, note VMware record keeping on this FAQ fails to mention that significant change:
Changelog
2023-10-24, 1930 PDT (-0700): Initial publication.
2023-10-25, 11:50 PDT (-0700): Updates to improve clarity.
2023-10-31, 0930 PDT (-0700): Updates to the VMware Cloud messaging.
Meanwhile the VMware advisory text for VMSA-2023-0023 says they were notified on January 17, a day earlier.
Issue Date:
2023-10-25Updated On:
2024-01-17VMware has confirmed that exploitation of CVE-2023-34048 has occurred in the wild.
Log management failure in multiple ways here. This is not how we used to run security at VMware.