Cloudflare took down our website after trying to force us to pay 120k$ within 24h
TL;DR:
We've been on the Cloudflare Business plan ($250/month) for years. They suddenly contacted us and asked us to either pay them $120k up front for one year of Enterprise within 24 hours or they would take down all of our domains. While this escalated up our business we had 3 sales calls with them, trying to figure out what was happening and how to reach a reasonable contract in a week. When we told them we were also in talks with Fastly, they suddenly "purged" all our domains, causing huge downtime in our core business, sleepless nights migrating away from CF, irreparable loss in customer trust and weeks of ongoing downtime in our internal systems.
Backstory
I'm a SysOps engineer at a fairly large online casino. (I think this article is relevant regardless of whether you think that in general casinos are ethical or not, I’m just mentioning it for context). We have around 4 million monthly active users. We had been happy Cloudflare customers since 2018 on the "Business" plan which has some neat features and costs $250/month for "unlimited" traffic.
Now admittedly, $250 is probably fairly low for the amount of traffic we were pushing through Cloudflare. We mainly use CF for the CDN (caching all our static content) and DDOS protection, for which it works pretty well. It’s easy to use and you don’t usually have to think about it much.
I had read a few articles on Hacker News about how at some point Cloudflare contacts you, asking you aggressively to move to "Enterprise" with custom pricing. But I wasn't expecting it to go this horribly.
April 19, 2024
In April, we received this email from Cloudflare:
This sounds like there's some issue with our website. We scheduled a call with their “Business Development” department. Turns out the meeting was with their Sales team, and they didn't have any “serious issues” to report at all. They asked us whether we would like to consider Enterprise. We politely declined, a bit confused as to the tone of the email.
May 3, 2024
Two weeks later, we received another email:
Now this needs a bit of context on what they are talking about. We do have multiple domains that mostly act as mirrors to our main domain. We have these for a few reasons. One is that since we are a casino, we have different regulatory requirements we need to comply with in many countries. For example, many games are only available in some countries. Some countries we block completely. Then we have a few different domains that remove certain game groups or site features - for example our social features (chat, user tipping / interaction) or our sportsbook. Another is that we use them to target different global user groups and affiliates and track conversions long-term. This also means that if a country DNS-blocks our main domain, a secondary domain may still be available. This could arguably be seen as a violation of the Cloudflare TOS, as they wrote above.
In any case, we receive >95% of our traffic through the main domain that’s been unchanged since our founding, and were happy to resolve this issue in whatever way, including by removing any affected secondary domains from Cloudflare.
We sent them info about our domains and tried to get more information from them about the issue and who from our team we should get involved, but they refused to give us anything apart from a date for a call.
May 7, 2024
So we scheduled another call, now with their "Trust and Safety" team. But it turns out, we were actually talking to Sales again.
They said they could offer us an amazing contract for $10k per month with all kinds of great features. We tried figuring out how exactly this was related to the TOS problem and how to resolve the situation. We asked them which domains were affected by their “rotation” concerns. They didn't give us an answer. We asked which of the Enterprise features we actually had to get.
They would not offer us anything apart from a full deal for $10k per month, which would magically resolve the issue. They were not interested in any other resolution.
They said we had 24h to sign their contract because they had to “get back to Trust & Safety". We asked to pay monthly. They said we need to sign a yearly commitment and pay the full year up front. It felt like extortion. Pay us $120k until tomorrow or we destroy your business.
After the call, they sent us this:
Note the email implies a monthly payment might be possible. When we asked for clarification, we were told we must pay the full year upfront.
We do not need most of those features they mention. I understand asking us to do BYOIP to remove their liability for our domains, but the rest is all things we don’t need or are purely “nice to have”.
We managed to buy a week of time by letting it escalate to our CEO and CTO and having them talk directly with Cloudflare.
But still, they didn't care about any other resolutions to the issue and refused to give us any other contract options. Finding numbers online is difficult, but if you squint your eye a bit (compare with this post: https://news.ycombinator.com/item?id=29333160 and this post https://news.ycombinator.com/item?id=31336515 ) 80TB of traffic might have a reasonable price of $150-$2000 per month. Note that 80TB is the number they tried to sell us, I don’t know if it is accurate since they removed all our access to historical analytics.
During this time we also looked into alternatives and set up a test domain and call with Fastly, since they seemed to be a reasonable competition.
May 16, 2024
In another call, trying to negotiate a reasonable contract, our CEO told Cloudflare Sales we were also talking with a competitor. I would have thought this is obvious, who wouldn't look for alternatives when getting slapped with a $120k invoice? But a few hours after the call, this happened:
Cloudflare had suddenly deleted all of our domains. All of our DNS records, caching setup, rate limits, whitelists, gone. Our public website, our incoming emails (including support emails from our customers) and our internal infrastructure, our authentication configuration on Cloudflare Access, down.
They also sent us this email:
The email says “this [...] does not impact current services”, so we frantically wrote them a support ticket but got no response. So we called in our SysOps team and started migrating our main site to Fastly. We had the basics after a few hours, but even then, a "NS" DNS entry change apparently takes a pretty arbitrary time to propagate everywhere, from 1h to 48 hours. We’re still recovering from the aftermath.
At some point, Cloudflare responded to our ticket with this:
“Trust and Safety” never reached out to us, and our account remains locked.
My tips for when Cloudflare reaches out to you
First of all, congrats! Your business must have become pretty successful. How exactly did CF decide to “ask” you to switch to Enterprise?
Maybe...
...you hit 10TB of traffic per month
...their lava lamps went into a specific astral alignment
...a sales rep realized that they haven't hit their quarterly quota yet
In the end, who knows? Cloudflare has absolutely no information on when they will force you into custom billing, but when they start "urgently" needing to talk to you you're probably not going to get out until you have a juicy custom contract with them. There's a reason why they have no public information anywhere on traffic limits or Enterprise pricing. Their Sales team will use anything (like having multiple domains) as fuel to force your whole account to Enterprise , no matter if it is fixable in a simple way.
The price they give you is going to be purely based on what they think you might pay, not on any measurable metric or feature set.
We tried asking how the price is going to be affected if we have less traffic, but they refused to say anything except 80TB is included (we have a large amount of callback traffic that uses IP whitelists and thus doesn’t actually need to go through a CDN, we just never spent time optimizing it since unlimited traffic was included).
We tried saying that we don't need any number of the 14 features that are included, they said all those amazing features are included whether we want them or not.
Numbers found on Hacker News threads (links above) suggest that the prices vary by at least one order of magnitude for the same services.
We tried saying our different domains (like internal ones) don't all need Enterprise, they said the whole account is Enterprise.
If they think you are flakey (maybe if you have alternatives?), they will give you an unreasonable deadline and force you into paying the year up front.
They will use any excuse as a reason for why you suddenly "need" enterprise, even if you're happy with the feature set of Business.
We're not the only ones that got their business threatened by CF’s aggressive sales tactics:
Just because you're paying $250/month, don't expect any kind of courtesy or (non-sale) responses to support mails. If you want CF to respond to you outside of Sales, the only way is apparently to give them negative press.
Be ready to move away from Cloudflare within 24 hours.
Never register domains directly on CF. If you do this and they block you, I have no idea how you can get your domain back in a reasonable time frame. Luckily we only had our NS pointed to CF and thus could move away with ~3-24h of downtime for most users.
Don't use any custom caching rules on CF. CF by default ignores most / all standard HTTP cache headers except for an arbitrary set of extensions and encourages you to create custom rules on CF. Instead, set CF to "Cache: Always" (this does not actually mean always) and "Respect Origin Headers". That way the rules will work for other caching proxies.
Don't use any proprietary Cloudflare products like Zero Access or Workers. We heavily used Zero Access for authentication in internal products, and now we have to rebuild all this infrastructure from scratch with massive downtimes. Only use their technology where it is compatible with third-party standards.
Make backups of your configuration on Cloudflare. It's an unexpectedly large pain to recreate all those configurations, including various sending email services (SPF, DKIM, …), site verification DNS entries, ip lists, rate limiting rules, etc.
Make sure you understand the impact of CF’s business model on you: Either you’re leeching off Cloudflare (customers on the free/business plan), or Cloudflare is leeching off you (intransparent Enterprise pricing). There is no in-between, and at some point the time comes to switch.
And: Consider whether you need Cloudflare at all.
CF only managed (for us) pretty large DDOS attacks. If you have some a bit more vulnerable attack surfaces (for example, an uncached unauthenticated API request that eats up 100ms of CPU time, and can thus use up your cores with just 10-100 requests per second), Cloudflare is not even going to detect it. Especially since all semi-professional “DDOS attack as a service” groups seem specifically specialized for Cloudflare-backed services, including “Under Attack Mode” workarounds etc.
This happened to us at BMW. Legal threatened to sue for millions in business and strangely everything was restored with NO FEEDBACK.
Hi, we've had the exact same thing happen to us, the BYOIP, the trust and safety team but it was sales, we can talk and discuss about how to NOT use cloudflare?