Best Practice, News

The CloudFlare Security incident that affects Salesforce Security

I have contacted Salesforce Security for comment, but at the moment I’m waiting on a reply.

29 March Update – Salesforce Security replied saying they were looking into it but then never replied again, which was disappointing. I did speak to Salesforce Security at CeBIT last week, we worked through the risk and there would be a chance that the OAuth tokens may be cached. But, as OAuth tokens time out the risk is now negligible.

Google Engineers earlier in the week identified an issue with CloudFlare were it was possible to see other website session data from other websites using the CloudFlare service. Cloudflare is a website security and caching service used by a huge amount of websites, we actually use the CloudFlare service for the LondonsCalling.net website to help secure it and also manage peak demand for the site in the weeks leading up to the event (although this issue doesn’t affect anyone who has purchased tickets as this is provided by Eventbrite).

So what happened?

Uber exposed data

Because CloudFlare is a multi-tenant service (multiple sites all using the same service), an issue could expose data that it shouldn’t. In this case, it was due to a buffer overrun; this is when a piece of code accidently moves into some memory in that it shouldn’t, and accesses information in memory which should only be accessed by another process (in this case a different website).

Google contacted Cloudflare via Twitter, not a usual way of communicating with a company about a security incident but as it was late on Friday and the issue needed to be resolved quickly. Cloudflare seemed to respond very quickly and activated their global kill feature at CloudFlare which disabled the affected features on their platform while they started working on a fix.

Ormandy the Google researcher that found the issue wrote.

“We keep finding more sensitive data that we need to cleanup. I didn’t realize how much of the internet was sitting behind a Cloudflare CDN until this incident, I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”

How does this effect Salesforce?

Exposed data from Fitbit

So I would be highly surprised if Salesforce is using CloudFlare. In fact, I ran a couple of random checks on core Salesforce services as well as non-core services like www.salesforceusergroups.com and didn’t find that it was being used. BUT if you are connecting to Salesforce from a website that is using CloudFlare then in theory OAuth tokens, session keys, cookies, plain text, etc. may have been compromised.

OAuth is used by websites needing to authenticate with Salesforce, this then allows the website/service to access/modify your Salesforce data or just to simply authenticate and nothing more. For example, workbench.developerforce.com is essentially a separate service from the core Salesforce platform running on Amazon Web Services. When you login into it Workbench it authenticates with Salesforce and workbench receives an OAuth token from Salesforce which it uses to access your Salesforce data, so the service doesn’t have access to your username and password.

If the workbench service was using CloudFlare and someone exploited the issue, it could be theoretically possible for someone to come across the OAuth token and re-use it to access Salesforce data.

What can I do to mitigate this?

MaxMind’s response

At the moment its a bit of a race against time. The issue has been around since at least September 2016 and the issue may have been fixed at CloudFlare but there are search engines and websites that cache website data that may still have compromised data in them. This cached data could have confidential session keys/OAuth Tokens etc. from other websites. Google has been manually purging its cache, and other search engines are following suit. There are rumours that Google has also expired Google Account sessions which have resulted in people being asked to re-authenticate into Google (this has happened to me on one of my Google accounts) but Google has denied the two issues are linked.

We’re still waiting to hear from Salesforce Security on their advice, but as a precautionary measure, I’m advising my customers to re-authenticate any service that has integrations into Salesforce that stores Salesforce credentials/tokens. Re-Authenticating should then refresh the authentication tokens and invalidate the old tokens so if they are cached anywhere they can’t be used.

The importance of bounds testing (it’s more common than you think!)

The route course in the CloudFlare service came down to just one character in a piece of code, >= rather than == which resulted in the buffer overrun. Over the years I’ve seen quite a few bounds issues like the CloudFlare issue, but the following customer issue has stuck in my mind, as it had one of the greatest impacts:

I was working with a client rationalising their global marketing data and campaigns, they sold a lot of consumer products globally but the marketing was very siloed, and they wanted a single view of the customer to see what products they interacted with and spot trends.

They had a business rule on their email marketing campaigns that they would only send emails to customers after they had been on their marketing lists for more than three months. Unfortunately, we spotted a mistake in their rule which meant they were only sending emails to people who had been on their marketing lists for less than three months, essentially automatically unsubscribing their customers from their marketing lists after three months, not ideal. They had their greater & less than symbols the wrong way around, a tiny mistake on the face of it but it had far reaching effects.

I try to drill into developers the importance of bounds testing in code as well as declarative functionality. If you are using a >, <, == or any operator then write unit tests or user acceptance tests around the bounds of the expression. Eg if you were checking that a value was greater than 100 you could test the values 99, 100 & 101. You have then tested the bounds of the expression.

Summary

It does look like CloudFlare we’re quick to resolve the issue and sites that cache websites are working on clearing out the websites affected by this security hole. CloudFlare has said that the leakage affected 0.00003% of requests coming into CloudFlare which doesn’t sound that much, but Cloudflare has a massive customer base including dating websites and password managers which host particularly sensitive data. That’s a lot of data which is potentially cached and now searchable…

better to be safe than sorry!

Apex, Best Practice, Development, Dreamforce

You can you use DML in a for loop!

Some people may have have seen ‘Apex Ten Commandments’ that myself and Codefriar did at Dreamforce a while back.

One slide we didn’t know if we should put in or not was to come right at the end saying “Thou shalt feel free to break these rules as wisdom merits”. But the issue with this slide was two fold. The majority of developers the ten commandments was aimed at should be following the rules. But for more advanced developers you do at times need to twist the rules and they don’t make sense. Adding the slide could mean that junior developers think of the commandments are only loose guidelines. In the end we left it out but the following year Codefriar kept it in… and in hindsight a better idea 🙂 Read more

Apex, Best Practice, Development

Thou shalt not leave the Salesforce critical updates until the last minute!

When I do org reviews its amazing how many people ignore the critical updates in Salesforce. You know that little pop-up box that says “You have critical updates do you want to view them? Yes | No?”, this appears when a new critical update gets added to your org (search for ‘critical updates’ in the setup menu), and with the Summer ’15 release there are a few new ones and one may need particular attention for some of us.

Read more