How Meta's Crawler Spiked My Vercel Bill with 11 Million Requests

The Incident: facebookexternalhit Gone Wild

Specifically, the User-Agent strings were dominated by facebookexternalhit/1.1 and Meta-ExternalAgent. These are the crawlers Meta (Facebook/Instagram) uses to generate link previews and, increasingly, to scrape content for AI training.

While good SEO and rich link previews are essential for any business, 11 million requests in a 30-day window is not normal behavior. It was hammering every single route, asset, and API endpoint repeatedly.

The Vercel Cost Model

Here is where it hurts. Vercel, like many serverless platforms, charges based on usage. While static asset bandwidth is cheap, Edge Middleware and Serverless Function invocations are not unlimited.

If you are using Middleware for authentication, geolocation, or A/B testing (which I do heavily), every single request that hits your site triggers that middleware. Even if the bot is just pinging your homepage, your code runs, and the meter ticks.

For 11 million requests, those milliseconds add up to a significant unexpected bill.

How to Stop the Bleeding

If you find yourself in this situation, here is the triage process I used to fix it.

1. The Polite Way: robots.txt

The first line of defense is telling the bot to go away. I updated my robots.txt file to explicitly disallow these agents:

User-agent: facebookexternalhit
User-agent: Meta-ExternalAgent
Disallow: /

The Problem: This relies on the bot being polite. While Meta usually respects this, propagation takes time, and during a glitch (which this seemed to be), they might ignore it completely.

2. The Application Layer: Blocking in Middleware

Since Vercel's Edge Middleware was the cost center, I added logic to reject these User-Agents immediately.

// middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';

export function middleware(request: NextRequest) {
  const userAgent = request.headers.get('user-agent') || '';
  
  if (userAgent.includes('facebookexternalhit') || userAgent.includes('Meta-ExternalAgent')) {
    return new NextResponse(null, { status: 403 });
  }
  
  return NextResponse.next();
}

The Catch: This still counts as an invocation! You are still paying Vercel to say "No" to Meta. It's cheaper than rendering the full page, but at 11 million hits, it's still not free.

3. The Infrastructure Layer: Vercel Firewall & Cloudflare

This is the real solution. You need to block the traffic before it executes your code.

• Vercel Firewall: Vercel has a built-in firewall where you can block traffic by User-Agent or IP. This stops the request at the edge before it hits your functions.
• Cloudflare (Recommended): I sit all my projects behind Cloudflare. By creating a WAF (Web Application Firewall) rule to challenge or block requests containing specific User-Agents, the traffic never even reaches Vercel's infrastructure. This reduced the billable requests to zero.

Lessons for Developers

1. Observability is Key: Set up billing alerts. Vercel allows you to set "Spend Management" limits. Enable them. If I hadn't checked my dashboard, this could have gone on for months.
2. Middleware is Not Free: Remember that middleware runs on every request unless configured otherwise via a matcher. Be careful what you put there.
3. Defense in Depth: Don't rely on your application code to filter bad traffic. Use infrastructure-level tools like Cloudflare or AWS WAF to scrub traffic before it costs you money.

As AI agents and crawlers become more aggressive, "defensive hosting" is becoming a required skill for modern web development. Don't let a bot eat your profit margin.

Practical review notes

The way I would make this practical is to turn the advice into a decision path. How Meta's Crawler Spiked My Vercel Bill with 11 Million Requests needs clear checks, failure modes, and one next action the reader can apply immediately.

My review path is simple: connect the advice to one real workflow, make the risk visible, change only what is needed, and keep proof that the change worked.

Decision checklist

• State the decision the reader needs to make.
• Add one concrete example.
• Explain the common mistake.
• Give a validation step.
• Link the advice to the next practical action.

Risks to test

• The post stays at opinion level.
• The reader cannot tell what to do next.
• The examples are not specific enough.
• The validation step is missing.

Next useful improvement

To make this stronger over time, I would add proof from the workflow itself: a screenshot, log excerpt, metric table, source link, or concrete QA result.

For a shorter post, I would add depth through one tested example rather than filler. One good edge case or validation note is more useful than another generic overview.

• One real example from the workflow.
• One edge case that breaks the simple advice.
• One metric or signal to watch after the change.
• One clear action the reader can take today.

One practical scenario

For How Meta's Crawler Spiked My Vercel Bill with 11 Million Requests, I would keep one concrete example in the page so the advice does not stay abstract. The example should show the starting state, the decision being made, the check I would run, and the signal that tells me the change worked. That makes the content more useful for readers and more defensible for SEO/AEO because it demonstrates practical experience instead of repeating a general claim.

• Starting state: what the store, app, workflow, or codebase looks like before the change.
• Decision point: what the reader needs to choose or fix.
• Validation: the command, screenshot, metric, support ticket, or QA step that proves the change.
• Risk: the edge case that could still fail in production.
• Follow-up: the next improvement I would make after the first pass is stable.

What to do next

The next step is deliberately small: test the idea on one real example, keep before/after evidence, then decide whether it deserves broader rollout.

Review path for meta-crawler-vercel-billing-spike:
1. Pick one real example.
2. Apply the checklist.
3. Record before/after evidence.
4. Watch one metric or failure signal.
5. Keep or revert based on the result.