How to rate-limit requests per IP in PrestaShop without blocking Google

Monday morning. The shop went down twice on Sunday, the access log shows three addresses that each requested eighty thousand pages, and you block them on the server. Peace of mind. On Tuesday there are forty new addresses doing exactly the same thing.

Blocking by address is everyone's first reaction and the one that lasts the least. Not because it is a bad idea, but because the problem is not who is asking but how much they ask. And that is measured, not guessed.

Why blocking addresses never ends

A crawler going through your catalogue does not come from one address: it comes from a network, or several, and it rotates. When you cut one off, it carries on with the next without missing a beat. Same with names: they show up as an ordinary browser, change identity every week and three new ones appear every month. The block list grows and the load does not drop.

What they cannot disguise is the pace. A person who is shopping opens a category, ticks two filters, turns one page and goes to a product page: three or four expensive searches in the whole visit. A crawler makes three or four per second, from every address it uses. Counting how many expensive searches each origin makes in a minute tells the two apart without needing to know their names.

What rate limiting per IP is, and why per network too

Limiting per IP means setting a ceiling: how many requests the shop accepts from the same address within a time window — sixty seconds, usually. Below the ceiling, everything goes through; above it, the request gets a "come back in a while" that costs nothing to serve.

With that alone, a crawler spreading the work across two hundred addresses of the same network stays under the ceiling on every one of them. That is why the second limit is per network: addresses sharing their first three blocks (what a technician calls a /24) are counted together. A mobile carrier can look like that, so this ceiling sits higher than the per-address one, but rotating stops being free. And a third brake, for the whole shop, acts as an emergency limit: when the total of searches per minute exceeds what your server handles comfortably, only visitors who have already proven to be people get through.

Where the limit goes: server or shop

There are three places where you can count, and they do not see the same thing.

• At the CDN (Cloudflare's rate limiting rules, for instance). It counts per address and per URL pattern. It does not know which request is expensive and which is an image, does not know whether the visitor is signed in, and treats Googlebot like anyone else. For one afternoon's spike it works; to live like that, you either overbrake your customers or underbrake the bots.
• At the web server (Apache's or nginx's rate limiting modules, or fail2ban reading the log). It counts everything coming in, CSS and photos included, so the ceiling has to be so high it barely brakes. And you need to know how to administer a server to touch it, which on shared hosting is not even allowed.
• In the shop itself, before PrestaShop starts searching. It is the only place that knows what is expensive: a filtered search, the search box, a listing page, a change of sort order. Only that counts towards the limit. It knows whether the visitor is signed in — and then does not limit them — and can check whether whoever claims to be Googlebot really is. That is where Anti-Bot Protection for PrestaShop puts it, and that is why the limit can be low without bothering anyone who buys.

How to let Google in

This is the fear that holds most people back, and rightly so: a badly set limit wipes you off Google all by yourself. The answer is not a list of names, because anyone can show up as Googlebot: it is asking the network. Every request claiming to come from Google is checked with a reverse DNS lookup, and if the address does not belong to Google, that request is not from Google, whatever it says.

Verified search engines come in with their own quota, separate from visitors', and receive filter pages marked as not indexable. That is exactly what Google recommends for faceted navigation: filter combinations should not turn into thousands of repeated pages. Once you stop spending its time on them, it starts crawling the ones you care about better.

What happens when someone goes over the limit

A script receives a "too many requests" with the time it may come back, and it costs your database nothing. But a per-address limit has one awkward case: the office with thirty people behind a single connection, or the customer who filters very fast. Those should not be thrown out: they should be checked to be people.

The check that works is the one nobody sees. A proof of work — a small calculation the browser solves on its own in milliseconds, with no puzzles or checkboxes — and a pass for one hour. The pass does not exempt completely, because solving that proof is cheap for a script too: whoever carries one still has a ceiling per address, just ten times higher.

The numbers to start with

With a sixty-second window, thirty expensive searches per address, ninety per network and six hundred for the whole shop cover the vast majority of shops without any customer ever noticing. A fast customer makes twenty in a minute; thirty is already someone not looking at what they filter.

If you do not trust it yet, there is an observation mode: the limit watches and records what it would have stopped, but lets everything through. A week like that and the panel tells you how many requests would have stayed at the door and where they came from, and you decide with data.

How to know the limit is set right

A limit that is not measured is a limit that one day bothers a customer without you knowing. What to look at every week is three things: how many expensive searches were served and how many were stopped, which origins were stopped most and why, and whether any of them is one of yours — your uptime monitor, your price comparison feed, a customer who complained. That one gets a click of trust and is never counted again.

And for the doubts, a checker: paste the address and browser from the log and the page they were asking for, and it tells you exactly what it would do with that request right now and why. It is the way to touch a limit without touching it blind.

All of this — the three ceilings, the cookie bounce, Google verified by DNS, the invisible check, observation mode and the panel with the checker — is what Anti-Bot Protection for PrestaShop does from the minute it is installed, on PrestaShop 1.7.6 to 9 and behind Cloudflare. And if you first want to know whether this is your problem, send us the access log of a bad day through the contact form: we will tell you for free how many of those requests are people and how many are noise.