The CloudFlare Security incident that affects Salesforce Security

Google Engineers earlier in the week identified an issue with CloudFlare were it was possible to see other website session data from other websites using the CloudFlare service. Cloudflare is a website security and caching service used by a huge amount of websites, we actually use the CloudFlare service for the LondonsCalling.net website to help secure it and also manage peak demand for the site in the weeks leading up to the event (although this issue doesn’t affect anyone who has purchased tickets as this is provided by Eventbrite).

So what happened?

Because CloudFlare is a multi-tenant service (multiple sites all using the same service), an issue could expose data that it shouldn’t. In this case, it was due to a buffer overrun; this is when a piece of code accidently moves into some memory in that it shouldn’t, and accesses information in memory which should only be accessed by another process (in this case a different website).

Google contacted Cloudflare via Twitter, not a usual way of communicating with a company about a security incident but as it was late on Friday and the issue needed to be resolved quickly. Cloudflare seemed to respond very quickly and activated their global kill feature at CloudFlare which disabled the affected features on their platform while they started working on a fix.

Ormandy the Google researcher that found the issue wrote.

“We keep finding more sensitive data that we need to cleanup. I didn’t realize how much of the internet was sitting behind a Cloudflare CDN until this incident, I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”

How does this effect Salesforce?

So I would be highly surprised if Salesforce is using CloudFlare. In fact, I ran a couple of random checks on core Salesforce services as well as non-core services like www.salesforceusergroups.com and didn’t find that it was being used. BUT if you are connecting to Salesforce from a website that is using CloudFlare then in theory OAuth tokens, session keys, cookies, plain text, etc. may have been compromised.

OAuth is used by websites needing to authenticate with Salesforce, this then allows the website/service to access/modify your Salesforce data or just to simply authenticate and nothing more. For example, workbench.developerforce.com is essentially a separate service from the core Salesforce platform running on Amazon Web Services. When you login into it Workbench it authenticates with Salesforce and workbench receives an OAuth token from Salesforce which it uses to access your Salesforce data, so the service doesn’t have access to your username and password.

If the workbench service was using CloudFlare and someone exploited the issue, it could be theoretically possible for someone to come across the OAuth token and re-use it to access Salesforce data.

What can I do to mitigate this?

At the moment its a bit of a race against time. The issue has been around since at least September 2016 and the issue may have been fixed at CloudFlare but there are search engines and websites that cache website data that may still have compromised data in them. This cached data could have confidential session keys/OAuth Tokens etc. from other websites. Google has been manually purging its cache, and other search engines are following suit. There are rumours that Google has also expired Google Account sessions which have resulted in people being asked to re-authenticate into Google (this has happened to me on one of my Google accounts) but Google has denied the two issues are linked.

We’re still waiting to hear from Salesforce Security on their advice, but as a precautionary measure, I’m advising my customers to re-authenticate any service that has integrations into Salesforce that stores Salesforce credentials/tokens. Re-Authenticating should then refresh the authentication tokens and invalidate the old tokens so if they are cached anywhere they can’t be used.

The importance of bounds testing (it’s more common than you think!)

The route course in the CloudFlare service came down to just one character in a piece of code, >= rather than == which resulted in the buffer overrun. Over the years I’ve seen quite a few bounds issues like the CloudFlare issue, but the following customer issue has stuck in my mind, as it had one of the greatest impacts:

I was working with a client rationalising their global marketing data and campaigns, they sold a lot of consumer products globally but the marketing was very siloed, and they wanted a single view of the customer to see what products they interacted with and spot trends.

They had a business rule on their email marketing campaigns that they would only send emails to customers after they had been on their marketing lists for more than three months. Unfortunately, we spotted a mistake in their rule which meant they were only sending emails to people who had been on their marketing lists for less than three months, essentially automatically unsubscribing their customers from their marketing lists after three months, not ideal. They had their greater & less than symbols the wrong way around, a tiny mistake on the face of it but it had far reaching effects.

I try to drill into developers the importance of bounds testing in code as well as declarative functionality. If you are using a >, <, == or any operator then write unit tests or user acceptance tests around the bounds of the expression. Eg if you were checking that a value was greater than 100 you could test the values 99, 100 & 101. You have then tested the bounds of the expression.

Summary

It does look like CloudFlare we’re quick to resolve the issue and sites that cache websites are working on clearing out the websites affected by this security hole. CloudFlare has said that the leakage affected 0.00003% of requests coming into CloudFlare which doesn’t sound that much, but Cloudflare has a massive customer base including dating websites and password managers which host particularly sensitive data. That’s a lot of data which is potentially cached and now searchable…

better to be safe than sorry!