General Musing

A company was having intermittent trouble with their new authenticated SSL, it wasn’t that they experienced trouble with the certificates, which came from a large international CA, or the authentication. There was a bug which caused the OCSP check on some certificates to fail. And after it had failed the first time for a certificate it would continue to fail for that certificate until the application server had been restarted. As this was a mission critical application for their customers between 8am-6pm they had taken to restarting the servers at 7am to ensure that there would be less issues during the day. This was obviously not a permanent solution so the vendor was called to fix the issue.

Having been send by the vendor the my suspicion immediately fell on the web application, the vendor code had changed little and went through rigorous testing. And besides from anything it was implemented across thousands of companies and this was the first the vendor had heard of an issue like this. The company had build their own Java web application which passed the OCSP checks to an OCSP daemon implemented by the vendor. Using vendor provided libraries they had implemented the certificate authentication in the Java web application. Never the less my point of entry would need to be the vendor OCSP as that was generating the error messages.

Firstly I would need to set up the Web Application so it would start generating exceptions to a certificate. The company had a certificate which consistently caused the error to occur on the test server, although it was an expired certificate this wasn’t an issue for the OCSP as it was configured to ignore the expiry date and concentrate only on whether certificates were revoked.

Starting from a baseline I set every daemon in the chain to the highest log level, restarted every daemon and started writing a script to automate starting the browser with the certificate in place. 2 hours, many cups of coffee and thousands of clicks and browser windows later I suddenly got the exception state.

Firstly I checked the web application and OCSP daemon logs, I was indeed getting a certificate exception in the web application and the OCSP logs produced an exception, but it wasn’t a revoked certificate. I checked the previous link in the chain which was the OCSP library. This produced two types of log one for the communication – which showing the same exceptions coming from the OCSP daemon, but no errors when sending the certificates to the OCSP server – and one which contained the last BER message which was send to the OCSP daemon. I had already grabbed a BER message from before the exception occurred and compared the two which showed that there was a difference in the expiry date of certificate in the BER messages. This was strange as the certificate which was used to generate these messages was the same certificate.

I was baffled, what could account for changes between the messages? I requested the code for the OCSP library from the development department of the vendor. Which they promptly refused. I just decompiled the library so I could at least continue my investigation, this is when I discovered some strange code. I fired up my browser to investigate the issues and discovered that the large international CA produced non-standard certificates, not that they didn’t comply to x509, but they used UTF-8 rather than ASCII encoding for their fields. For the library this meant that it needed to jump through a number of additional hoops to produce the BER message which it would send to the OCSP daemon. This also meant that the issue could be traced to a function in the vendor library which garbled the UTF-8 expiry date in certain cases.

I rewrote the function for this client to ensure that the bug couldn’t occur and recompiled the class which I replaced in the vendor’s jar and uploaded it to the test server. 2 hours, many cups of coffee and thousands of clicks and browser windows later we were unable to reproduce the issue. I documented the issue in a site visit report send it to the vendor development team and the customer. I didn’t hear from the customer again.

Image source: