There’s been a lot written about the most recent Comodo certificate compromise including two Mozilla Security Blog posts on the topic, but I have yet to see a concise timeline of the events. As a former Mozilla security release coordinator, I’ve been following this topic closely and wanted to write up my thoughts, as well as a full timeline.
A good write up of the issue is available on the Mozilla Security Blog, as well as on the Tor blog, where Jacob Appelbaum did excellent detective work to find this issue long before it was publicly disclosed. I also want to mention bug 642395 in which details are emerging about a hacker claiming to be responsible for the compromise.
Timeline
- 15 March, 18:00-20:00 – Certificates issued.
- T+0d, 0h, 15m – Comodo revokes certificates.
- T+1d, 1h, 32m – Mozilla informed of issue with initial list of certificates.
- T+1d, 4h, 33m – Google lands initial fix in Chrome’s tree.
- T+1d, 13h, 29m – Mozilla bug filed.
- T+1d, 21h, 59m – Comodo confirms most major browser vendors aware of the issue.
- T+1d, 23h, 44m – Chrome update with initial fix available.
- T+2d, 1h, 38m – Mozilla lands initial fix on main development trunk and Firefox 4 branch.
- T+2d, 13h, 29m – Comodo informs Mozilla of two additional certificates to block.
- T+2d, 15h, 33m – Mozilla lands additional fix on main trunk and Firefox 4 branch.
- T+2d, 19h, 59m – Google lands additional fix in Chrome’s tree.
- T+3d, 16h, 45m – Confirmation that Apple is aware of the issue.
- T+3d, 23h, 20m – Mozilla lands initial fix and additional fix on Firefox 3.5 and 3.6 branches.
- T+6d, 17h, 44m – Firefox 4 with fixes available.
- T+7d, 6h, 30m – Firefox 3.5.18 and 3.6.16 with fixes available.
- T+7d, 7h, 12m – Mozilla announces certificate issue, without details.
- T+7d, 20h, 44m – Microsoft issues fixes.
- T+9d, 1h, 16m – Chrome update with additional fix available.
- T+9d, 19h, 23m – Mozilla announces details of certificate issue.
Some notes about the above timeline:
- All times in UTC.
- T+0 is 15 March, 20:00 since that’s seemingly when the last certificate was issued.
- The “initial fix” listed is a patch blacklisting the initial 7 certificates that Comodo informed vendors about.
- The “additional fix” listed is a patch blacklisting the additional 2 certificates that Comodo informed vendors about.
- Details about when vendors other than Mozilla were alerted to the issue are hard to find.
There’s a lot to say about this event, much of which has already been said. Before talking about timeline, I think it’s important to call out Comodo for both their good and bad work in this instance.
To save face, Comodo could have simply revoked the certificates and dealt, in private, with the RA that issued the certificates. Those outside of the open source world know how hard it is to come clean, publicly, for something that can be kept private. Kudos to them for contacting browser vendors and ensuring a fix made it out fast.
That said, this isn’t the first problem Comodo has had. Previously Comodo allowed issuance of a www.mozilla.com certificate, allowing domain verification to be done by their RA.
(I could also mention bug 526560 but that wouldn’t be entirely fair to Comodo since other CAs are doing the same thing. While this is blatantly against Mozilla’s CA Policy, Mozilla has decided not to enforce such issues. The open bug on enforcing section 7 is 567193.)
Of course everyone is focusing on Comodo right now. I’d like to focus on the browser vendors and their reactions to this threat.
From the timeline, it’s fairly clear that one browser vendor has taken the longest getting this issue fixed. No surprise here, that vendor is Apple. I yearn for the day when Apple takes security seriously. Unfortunately, I think I’ll be yearning for a long, long time.
It’s also clear that Google responded fastest, issuing a fix to its users less than 48 hours after the attack. While we don’t know for sure when they were contacted, we can assume it was around the time Mozilla was contacted. It’s clear they went into overdrive and released a stable version of Chrome blacklisting the bad certificates less than 24 hours after being informed of the issue. Sadly, Google didn’t know that Comodo would later realize they had failed to disclose two additional fraudulent certificates to browser vendors. They issued a fix for those two certificates seven days later.
Mozilla has often worked that fast to fix critical security issues, but in this case didn’t. While they quickly decided to rebuild the Firefox 4 release candidate to include the blacklisted certificates, it still took a full six days before a fix was in the hands of users, in the form of Firefox 4 and later Firefox 3.5.18 and 3.6.16. During this time, all users were theoretically at risk.
Comodo has said that there is no evidence of any of the bad certificates being used in the wild, based on their OCSP responder logs. Of course, OCSP pings can be stopped with a MitM attack, something any state-driven attack — as Comodo claims this is likely to be — could easily do. (Read more on revocation and its shortfalls.)
Mozilla also decided not to disclose this issue publicly until a fix was release. They have since apologized for waiting so long but I think the bigger story is how long it took to get a fix out in the first place. While Google jumped to protect Chrome users, Mozilla waited almost six days before issuing a fix to users.
Attacks like this are often targeted at a specific group of users and this one was likely the same. We will likely never know all the details but there are a few questions and takeaways from this event that we should look at closely and take very seriously.
- Revocation clearly doesn’t work right now. At what point will browsers fail on revocation errors?
- Mozilla has always held openness and security as two of its main mantras. In this instance, they failed at both, not informing users of a targeted attack immediately and not issuing a security fix for almost six days. Sometimes waiting for everyone else isn’t “responsible disclosure.”
- Comodo has never had full control over its RAs – something likely true of many CAs – and is increasingly causing critical security issues for users worldwide. The larger your network of RAs, the larger your threat vector.
I’m actually a bit disappointed at Mozilla’s performance during this event and I hope they take such compromises more seriously in the future. Regardless, there are lessons to be learned at each step of the way by all parties involved.