On September 3rd at 18:52 the Microsoft Security Response twitter account sent out this notice:

We’re in the process of moving all DigiNotar CAs to the Untrusted Root Store which will deny access to any website using DigiNotar CAs

This was four days after they had sent out a security advisory on SSL fraud linked to DigiNotar.

Mozilla’s first announcement was a day before Microsoft’s and was followed up with a bulletin that said all confidence was lost in DigiNotar’s security management.

Mozilla has a strong history of working with CAs to address shared technical challenges, as well as responding to and containing breaches when they do arise. In an incident earlier this year we worked with Comodo to block a set of mis-issued certificates that were detected, contained, and reported to us immediately. In DigiNotar’s case, by contrast, we have no confidence that the problem had been contained. Furthermore, their failure to notify leaves us deeply concerned about our ability to protect our users from future breaches.

This must come as welcome news to those who use Mozilla and Microsoft products. The security teams at these companies are responding to an authority’s poor security management practices by removing their trust of that authority.

Mozilla even has a convenient page for manual deletion of the DigiNotar CA certificate if you want to take care of it sooner or confirm their update has worked.

These two companies are moving quickly while other major brands seem to be silent. I guess if I owned anything made by Apple I would be concerned right now and wondering why they are so quiet. I explained at the start of my HOPE presentation in 2010 that after 21 years of using Apple as my primary system I will no longer use their products. Hopefully (pun not intended) some of those attending my presentation can now see what I was talking about.

But let’s not get too distracted by the browsers. They have had a problem demonstrating trust in certificates since forever. Five years ago I remember arguing with big groups from Microsoft, Google and Yahoo! over the crisis of certificates in browsers. There was much hand-wringing over the problem of trust with numerous proposals put forward on why we should get rid of the useless lock icon and instead use things like bright green and red URLs — Extended Validation (EV) certificates. This was meant to help put confidence back in browsing but it really spoke to the fact that the certificate system was being easily circumvented. In that case it simply was from manipulating the user experience.

Now we see again the certificate system easily circumvented. We should not be shocked but rather resolved to fix the breach. This time, however, it is an “authority” of certificates that has proven itself to be useless. The problem is more complicated because a certificate authority (CA) is supposed to be worthy of the title. They should know, if anyone, the importance of taking precautions to protect their data.

Let this be an excellent time for everyone to stop and ask themselves if they have ever said “that company would go out of business if they did not have good security so I don’t need to verify”. Pilots use checklists and doctors use compliance checklists. They too would be out of business if they failed, yet even the most intelligent and sophisticated professionals in this world go through a validation program before they take risks.

So the real security issue here is in regard to the confidence in an authority to be managed properly. It is especially important to focus on the response to an incident. It is not that the certificate system has been broken (have you ever seen or heard of a forged drivers license) but how companies respond to the discovery of a flaw in their system of authority and trust. The Fox-IT DigiNotar public report version 1 was released on September 5th and it makes DigiNotar look reckless and negligent for several reasons.

1. Weak record keeping. DigiNotar was not maintaining a record of certificates that they issue, which is an obvious requirement to be a certificate authority.

The rogue certificate found by Google was issued by the DigiNotar Public CA 2025. The serial number of the certificate was, however, not found in the CA system’s records. This leads to the conclusion that it is unknown how many certificates were issued without any record present.

2. Weak infrastructure management. There are many examples to illustrate this point, from open wireless access to failure to notice breach traces left behind by attackers, from a flat Windows domain to webservers missing patches, from a lack of antivirus to no centralized logging…

In August, DigiNotar installed a new web server. It’s fair to assume these hacker traces where copied from the previous web server install. […] A number of malicious/hacker software tools was found. These vary from commonly used tools such a the famous Cain & Abel tool to tailor made software. […] In the text [of the tailor made software] the hacker left his fingerprint: Janam Fadaye Rahbar. […] The attacker(s) had acquired the domain administrator rights. Because all CA servers were members of the same Windows domain, the attacker had administrative access to all of them. […] The network has been severely breached. All CA servers were members of one Windows domain, which made it possible to access them all using one obtained user/password combination. The password was not very strong and could easily be brute-forced. The software installed on the public web servers was outdated and not patched.

3. Failure to respond and alert others to breach. There is evidence of breaches going back to 2009 that DigiNotar apparently did not investigate. This incident has the following timeline, showing a discovery on July 19th and then no response for weeks — until after the attackers had penetrated the servers and generated certificates that caused an external alert.

06-Jun-2011 Possibly first exploration by the attacker(s)
17-Jun-2011 Servers in the DMZ in control of the attacker(s)
19-Jun-2011 Incident detected by DigiNotar by daily audit procedure
[…]
10-Jul-2011 The first succeeded rogue certificate (*.Google.com)
[…]
22-Jul-2011 Start investigation by IT-security firm (not confirmed)
[…]
29-Aug-2011 GOVCERT.NL is notified by CERT-BUND

The pig in the grass house just had it blown down. I do not see yet that we should be looking at an urgent and complete overhaul of the entire certificate system. That would be a great long-term goal of course but it is not yet justified, based on this breach of embarrassingly weak defenses. We should instead be taking a better look at how companies manage and sell elements of PKI, as I wrote about recently.

Over the next weeks and months other CA companies will be out to validate or build up their brick house and prove (through compliance assessments) that they know how to be a responsible authority. I think there are still CAs around that can do it, not least of all because we know the mistakes made by DigitNotar. The focus will be to find management failures to follow even basic and well-known security practices. The sophistication of the threat is not nearly as interesting or daunting as DigiNotar’s lack of response and a lack of awareness. We should not forget that we are talking about a company whose business depended on good security, like a restaurant depends on a clean kitchen, and yet….

On that notion, I wonder when Apple will wake from its slumber to issue a response.

Users can revoke a certificate using Keychain, but if they happen to visit a site that uses the more-secure Extended Validation Certificates, the Mac will accept the EV certificate even if it’s been issued by a certificate authority marked as untrusted in Keychain.

“When Apple thinks you’re looking at an EV Cert, they check things differently,” Sleevi said in an interview Wednesday. “They override some of your settings and completely disregard them.”

Designed as a way to reassure Web surfers that they’re not being phished, Extended Validation Certificates turn the browser address bar green.

It sounds to me like the Apple fix for user experience trust issues with certificates (adding EV support) now limits their ability to handle trust issues with certificate authorities. Perhaps they did not realize that by working to resolve one trust issue that there would be future trust issues ahead? Did they think attackers would sit still? That seems rather ironic. If their EV design turns out to be the problem it just adds fuel to the fire in my belly about the direction at Apple. Consumers will be best served by companies that have the following attributes in their incident management:

• Fast response
• Transparency and admission of facts
• Fixes that acknowledge lessons learned and show process of improvement

This DigiNotar breach thus offers a global litmus to measure our current products, contracts and services. Who should we trust and have they responded yet?

Update: I should have also mentioned that this is a shot across the bow of large cloud providers. They rely more on the certificate authority model because it is a highly efficient way to establish (a certain level of) trust for millions of users. The smaller providers and traditional IT organizations have an advantage in that they can make updates to provide trust for their users far more easily and through secondary non-public channels (e.g. distribute secrets, revoke/replace certs).