In early October 2026 a site moved to Hostney with a scraper problem it had been living with for a long time. The scraper was loading more than 300,000 pages a day, almost every one from a different address. On the site’s previous host it kept bringing the server down, and the only way to keep the site up was to put every visitor behind a CDN’s “under attack” page, which stayed on for months.
For most site owners, a scraper like that never shows up as a scraper. It shows up as an email from the host saying the site went over its monthly visits or keeps hitting its CPU limit, with a link to a bigger plan. Scrapers copy content and prices, scanners look for old plugins, login bots try passwords, and this year we’ve seen whole farms of headless browsers that run JavaScript and get through a challenge page the way your own browser would. If you upgrade, they get the extra room too.
We last wrote about our bot protection in February, and most of what we’ve added since started with that one site. Here’s what happened to it, and after that, how to check your own traffic before you agree to an upgrade.
What we found when it arrived#
The scraper spread its requests over as many addresses as it could get. A per-address score has very little to go on with traffic like that, because it hardly ever sees the same address twice. Between October 5 and 11, the scraper answered 35,931 of our checks on that one site from 35,331 different addresses.
What we built for that site over the following three weeks now runs for every site we host.
Checking every visitor#
We started with a per-site switch called “Check every visitor”, which we turned on for that site on October 12. With it on, every page view needs proof from the visitor’s own browser before the page is served. A person passes it once and keeps browsing. The scraper has to pass it again on every new address.
The switch can also turn itself on. When any site we host starts getting read by a rotating pool of addresses, the check comes on within the hour and the site’s owner gets an email about it. After seven quiet days it switches itself off again. If you turn it off yourself, we leave it off for 30 days and don’t second-guess you.
Headless browsers on home internet connections#
The scraper kept getting through the new check, because it was answering with headless Chrome. A headless browser is a real browser engine driven by a script, usually on a server. It runs JavaScript, so it can work through the puzzle on a challenge page like any other browser. In the week to October 19, 2026, more than 45,000 addresses on home broadband and mobile networks in the US, Europe and Australia answered the checks on that site with headless Chrome. They solved every puzzle we gave them. Scored address by address, they looked like 45,000 clean visitors, each seen once or twice.
We caught them by the way they draw the page. A browser running on a server with no graphics card renders everything in software, and a person’s phone or laptop almost never does that. Since October 20, an answer like that from a home or mobile network no longer clears the check. The address gets our interactive check, a checkbox that pays attention to how a person moves and clicks, and extra computing power doesn’t help anyone pass it. When we checked right after switching it on, 220 addresses from that pool had been shown the checkbox, and none of them had made it through to a page. On that site, the number of addresses solving our checks with a software-rendering browser dropped from about 17,000 a day to about 100.
Hosting and cloud networks get the checkbox#
One rule was already in place before the site arrived. Very few people browse the web from a rented server, and a lot of scrapers run on one, so visitors from hosting and cloud networks go straight to the interactive check. The scraper sent plenty of traffic that way too. On that one site, in the 30 days to October 28, 2026, visitors from hosting networks were shown the interactive check 333,820 times, and it was passed 379 times.
If you expect visits from cloud machines on your own site, like your uptime monitor or a partner’s integration, you can turn datacenter checks off for that site, whitelist the addresses, or exempt the single path the integration calls.
Why bots end up on your hosting bill#
Most hosting plans are sized by monthly visits, by how much CPU and memory a site may use, or by both. Bots count against all of it.
A visit counter can’t tell a scraper from a reader. Every page request counts as a visit, and a scraper that switches to a new IP address for every request shows up as a new unique visitor each time. A scraping job that loads 50,000 pages can look like 50,000 people.
CPU limits fill up the same way. When a bot asks for a page, PHP runs and the database answers just as they would for a person. Bots also like the expensive pages: search results, filtered product lists, every variation of every product, and URLs with query strings that the page cache won’t store. A site that serves its real audience without breaking a sweat can spend most of its CPU on a single scraper.
Moving to a bigger plan doesn’t fix the proportions. A scraper that’s only held back by how fast your server answers will happily use any extra room you add.
Your analytics may not show any of it. Simple bots never run the analytics script, so they only appear in the server logs. Headless browsers do run it, so they can inflate your analytics numbers as well, and that makes the case for an upgrade look stronger than it is.
How to check your traffic before you upgrade#
For a first look you only need your access logs and your analytics. If your host doesn’t give you access logs, ask them for the breakdown below before you agree to anything.
- Compare the logs with analytics. Count the page requests in one day’s access log (leave out images, CSS and JavaScript) and compare that with the page views analytics shows for the same day. If the log figure is several times higher, the difference never ran the analytics script.
- See where the requests come from. A local bakery’s site read mostly from the other side of the world is a warning sign. Travellers, VPN users and company networks turn up abroad too, so look at the volume and ignore the odd visit.
- See which networks they use. People browse from home broadband, mobile networks and offices. Requests from cloud and hosting providers are mostly software. Any IP lookup tool will tell you which network an address belongs to.
- Count requests per address, and new addresses per hour. One address making thousands of requests is easy to spot. The harder pattern is thousands of addresses making one or two requests each, within a few minutes of each other, all hitting the same kind of page. That’s a scraper going through a rotating proxy pool like the one on the site above, usually home and mobile connections rented by the hour.
- Look at what they ask for. Search pages, every combination of filters, an archive read page by page in order, the same URL with different query strings, and
/wp-json/endpoints your theme never calls. - Check the clock. People sleep. Traffic that stays flat all night in your visitors’ time zone isn’t coming from them.
A rate limit catches the single busy address and misses the spread-out one, which is why rate limiting alone does not stop bots. If most of what you find looks like the spread-out kind, a bigger plan won’t help you, and neither will blocking addresses one at a time.
What else runs on every site#
Every address that sends requests to a site we host still gets a threat score, and the ones that misbehave are still challenged or blocked, as we described in February. Next to that, every site gets spot checks and its own set of switches.
A small share of ordinary visitors get a check at random. It keeps the system up to date on what real visitors look like, and it catches bots that look clean from the outside. With invisible checks on, the spot check runs inside the page, so nobody sees a challenge page for it. The page loads as usual, a small script from your own domain runs the browser checks in the background, and a visitor who fails gets the regular challenge on their next page.
The automatic layers work without anyone telling them what to look for, but when you already know what’s hitting a site, you can handle it yourself with the site’s own controls:
- rate limit rules for a specific path, such as a search page or an API endpoint a script keeps calling
- blocks for single addresses, ranges and whole countries
- access rules that open a site only to listed addresses, or only during opening hours
- AI crawler blocking, if you don’t want your content in training data
- lockdown, which checks every visitor during an attack, with exemptions for your office, your monitoring and your webhooks
You don’t have to set any of these up for the protection to work. Search engines are let through after we check their addresses against the ranges they publish (we explain how in how we verify search engine bots), and a signed-in WordPress admin working in their own dashboard isn’t held to the rate limits that would otherwise flag bulk edits and uploads.
Seeing what's happening on your site#
You should be able to see what was stopped, where it came from and how much of your traffic it was, so that the next time someone says your site needs more resources, you can check for yourself. We spent a good part of this year on that side of the system.
Every website on Hostney has a Bot detection page in the control panel. For any range from the last 24 hours to 90 days, it shows:
- Protection activity. How many requests were refused and how: challenge pages, blocks and rate limits, per day or per hour.
- Traffic against its usual range. Request volume next to the range we’ve learned is normal for that site at that hour of the week, so a spike stands out.
- Where the attacks come from. The top countries and hosting providers behind the addresses that scored on the site.
- Who solves the challenges. The countries the answers came from, how many came from headless browsers, which checks they met, and how many were sent to the checkbox compared with how many passed it. If your readers are in Germany and most of the solves come from other countries with a high headless share, you’re looking at a scraper.
- Recent threats. Every address that scored on the site, why it scored, and a button to whitelist it.
On the dashboard, the traffic quality figure shows what share of your account’s requests reached your sites without being challenged or blocked. Ellie, the assistant built into the control panel, reads the same data, so you can ask her who’s scraping a site and she’ll walk you through the countries, networks and checks behind it.
You’ll hear about changes as well. When the automatic visitor check switches on for a site, the owner gets an email. If someone on the account turns off a site’s bot protection, datacenter checks or lockdown, the account owner gets an email too, so a teammate’s change can’t sit unnoticed for months.
A refused request doesn't use your plan#
When we challenge or block a request, the web server in front of your site answers it before PHP starts. It never takes a PHP worker, a database connection, or any of the CPU and memory your plan gives your site. A WordPress security plugin can’t do that, because WordPress has to load before the plugin can refuse anyone. We go through how the two work together in Wordfence and server-level security.
On the site from October, most of the scraper’s page loads now end there, as check pages the web server answers without PHP or the database getting involved.
Our plan cards list visits too, from about 10,000 a month on Startup to about 200,000 on Pro. That number is our estimate of what the plan handles with caching on. Nothing on Hostney counts visits for billing, and we don’t charge for going over. If you do outgrow a plan, it should be because your real visitors need more, and the Bot detection page will show you whether that’s the case before you decide. For sites that do need the room, we cover what changes at higher volumes in high-traffic WordPress hosting.
What still gets through#
Some automated traffic will always get through, and it’s worth knowing which kind.
We set the system up to protect visitors first. Challenging a real person by mistake can cost you a page view or a sale, so we’d sooner let the odd bot request through. It uses a little server time, and a bot that keeps coming back runs into its score sooner or later. That’s why most signals add to a score without deciding anything on their own, the checks get harder a step at a time, and an address that stops misbehaving goes back to clean by itself. The trade-off is that a scraper behaving like a single person, at a person’s pace, from one address, is very hard to tell apart from that person. We don’t go after that kind of traffic aggressively, because doing so ends up blocking real visitors.
Every bypass we’ve closed stays with us as a test case that we replay against a real server, so a later change can’t quietly open it again. When a new technique shows up, it goes into that set before the fix ships.
Login attacks have their own write-ups. We looked at what attackers actually type into WordPress login forms in our login telemetry research, and at how the layers respond to a credential attack in what happens when bots find your WordPress login page.
If your site is being scraped right now#
There’s nothing to configure on Hostney. The protection runs from the first request a site gets, and its Bot detection page starts filling up the same day. You can move a WordPress site with our migration plugin during the 14-day free trial and see what your traffic really looks like before you pay for anything.
If you’d like to understand your traffic before moving anything, send our support team what your host told you and we’ll go through it with you.
The figures come from one site we host, from our own challenge records and edge logs, October 2026. We’ve left its name out on purpose.