AI bots on PrestaShop: which to block and which bring sales

A clothing shop with eight hundred models starts falling over every afternoon. The server hasn't changed, neither has the traffic, and new names are showing up in the log: Bytespider, GPTBot, ClaudeBot. The first instinct is to block them all. That's the expensive instinct, and it doesn't fix what's actually breaking.

Which AI bots visit your shop, and what each one does

They aren't all the same, and that's the part almost nobody checks. AI companies run three different robots with three different names, precisely so you can decide separately:

• Training. They collect pages for future versions of the model: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Bytespider (ByteDance, the TikTok company), CCBot. Blocking these costs you nothing in visits.
• Search. They index your shop so it can be cited when somebody asks: OAI-SearchBot, Claude-SearchBot, PerplexityBot. Block these and you stop appearing in the answers.
• A person's request. They fetch your page because somebody just asked about you: ChatGPT-User, Claude-User. They aren't crawling: they're serving a potential customer. OpenAI also warns that, because these are user-initiated actions, its robots.txt rules may not apply.

The block list going round the forums lumps all three groups together. It closes training —which you may well have wanted closed— and on the way out removes you from the index and slams the door on the customer who was asking about your trainers.

Blocking them all costs you sales, and it's measured

Adobe tracks over a trillion visits to US retail sites, and for eleven straight months it has measured the same thing: a visitor arriving from an AI converts around 60 % better than one arriving through any other channel. They leave 37 % more revenue per visit, add to cart 28 % more often, and spend nearly half as long again on the page.

It makes sense: by the time that customer clicks, the comparison has already happened inside the chat. They aren't browsing, they've decided.

And it isn't marginal: AI-referred traffic to retail grew 62 % in a year, and has multiplied twelvefold since late 2024.

Blocking a training crawler saves you bandwidth. Blocking a search crawler erases you from the answer that was about to cite you.

Google-Extended: the only one you can close for free

There's one that's all upside and hardly anyone knows it. Google-Extended decides whether Google may use your text to train Gemini, and it's a permission separate from Googlebot, which is what indexes your shop for ordinary search.

Here's what matters: the AI summaries at the top of Google don't draw on Google-Extended, they draw on Googlebot's normal index. You can tell Google not to train on your product pages and still appear exactly as before in search and in those summaries.

Bytespider ignores your robots.txt

This is where half the advice you'll read falls apart. robots.txt is a request, not a door. It works with whoever chooses to honour it.

The most cited case is Bytespider, ByteDance's crawler —the TikTok company— feeding their Doubao assistant. ByteDance maintains that it respects robots.txt; what administrators report is another matter: some have logged up to 1.4 million requests a day from that single bot, a rate in the order of twenty-five times GPTBot's. For Bytespider, robots.txt isn't enough: it has to be closed at the server.

And it isn't alone. In 2025 Cloudflare publicly accused Perplexity of crawling under undeclared identities when its declared agent was blocked: switching user-agent, posing as an ordinary Chrome on macOS, and rotating addresses outside its published ranges. Cloudflare removed it from its verified bot list. Perplexity denied it.

The practical conclusion: robots.txt works for OpenAI, Anthropic and Google, who honour it and document their agents. For the rest you need real blocking — a server rule by user-agent, or the AI-crawler blocking switch your CDN already offers you if you sit behind one.

The arithmetic that explains why your shop falls over

Back to the clothing shop. Eight hundred models aren't eight hundred addresses. With faceted search on —6 sizes, 12 colours, 20 brands, 4 price bands— every category generates 5,760 combinations of a single value per filter. Across forty categories, 230,000 URLs. And that's counting one option per filter: the moment a customer can tick two sizes at once, the figure runs into millions.

None of those pages really exists. Every one of them fires a heavy query the cache can't keep, because every combination is different.

AI didn't dig that hole. It had been there for years and you were paying for it in wasted Google crawl budget, as we covered when writing about faceted search and CPU. What AI has done is multiply the number of robots walking through it —bots are now more than half of all web traffic— and that's where a shop with modest resources breaks: not from one expensive visit, but from fifty at once.

That's why blocking robots treats the symptom. The ones arriving next month will have different names.

What to do, in order

1. Close the URLs that shouldn't exist for anyone. Filter combinations, internal search results, sort orders, page 47 of a listing, cart and customer account. Not for AI, not for Google. This is what brings the CPU down.
2. Leave the AI search crawlers open. OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User are the ones bringing you the customer who converts 60 % better.
3. Decide about training. GPTBot, ClaudeBot and CCBot are optional. Google-Extended is free to close.
4. Block Bytespider at the server, not in robots.txt: it isn't going to read it.
5. Read the logs for a week before and after. It's the only thing that tells you whether you got it right.

And if the server has nothing left to give

All of the above assumes you have headroom. If you're on shared hosting and the shop falls over every afternoon, you put the fire out first: close the training crawlers and let only the search ones in.

But make it a decision with a date on it, not a permanent state. A shop that has shut the door on AI search crawlers doesn't come up when people ask about it, and that never shows on any chart: it simply doesn't happen.

If you don't know where to start cutting, tell us what catalogue you have, how many facets are switched on and what server it runs on, and we'll tell you what to close, what to leave open and what needs fixing before you touch robots.txt at all.

Tell us how your server is holding up