There's a version of competitor research that is completely defensible: you read the same pages a customer reads, you read them on a schedule instead of when you remember to, and you write down what changed. And there's a version that gets your IP blocked or a letter from someone's lawyer. The uncomfortable part is that both start with the same twenty lines of Python. The difference isn't the code — it's the line you drew before you wrote it.
The takeaway up front: the boundary that matters is access, not technique. If a page is public, served to anyone without a login, and the site's rules don't forbid automated reading, collecting it politely is fair game. The moment you're using automation to get at something the site has decided not to give you — private data, a rate you weren't granted, an account that isn't yours — you've crossed the line, and no amount of clever engineering moves it back.
The three buckets: fair game, ask first, off-limits
Before you automate anything, sort every source into one of three buckets. It takes ten minutes and prevents every bad outcome in this article.
Fair game
- Public pages served to anyone. A competitor's homepage, feature pages, blog, careers page, changelog. If a first-time visitor with no cookies sees it, you can read it on a schedule.
- Published pricing. Prices a company puts on a public page are published deliberately, for exactly this reason — customers compare. Tracking them over time is ordinary market research.
- Your own accounts and properties. Your ad accounts, your analytics, your dashboards, your staging site. Automating something you're authorized to use isn't a competitor-research risk at all; it's just tooling.
- Official APIs, exports, and public ad libraries. Permission is baked in and the schema is stable. Always check for these before you write a scraper.
- Public registries. Company registration records, trademark filings, and similar government data are published to be read.
Ask first (or don't bother)
- Anything behind a signup wall, even a free one — you accepted terms to get in.
- Content a site licenses commercially. If they sell the dataset, taking it for free is a business decision, not a technical one.
- High-volume collection from a small operator. A fast crawl against a three-person company's shared host is a real cost you're imposing. An email asking for a feed is often quicker than the scraper anyway.
Off-limits
- Anything requiring credentials you weren't given. This is the brightest line in the article. Automation aimed at logging into an account that isn't yours isn't research; it's intrusion, and no framing rescues it.
- Personal data about identifiable people. "It was visible on the internet" is not a lawful basis under GDPR or CCPA/CPRA. Keep research at the company level — pricing, positioning, content, public listings — and this problem mostly disappears.
- Paywalled or deliberately restricted content. If they built a wall, going around it is the thing the wall exists to stop.
- Anything a site's terms explicitly forbid, once you know about it.
The tell that you've drifted from bucket one to bucket three is usually emotional: you start thinking of the target site as an adversary. Polite collection doesn't feel like that. If your plan involves outsmarting someone, re-read your buckets.
Rate limits are a contract, even when nobody wrote one down
Most sites never tell you how fast you may crawl. That doesn't make the limit infinite — it means you infer it and stay well underneath. Three mechanisms do almost all the work.
robots.txt is a stated preference, and you honor it. It isn't legislation, but it's the site telling you which paths it's willing to have crawled and, via Crawl-delay, how slowly. Ignoring it is the clearest possible signal of bad faith, and frequently a terms violation on top. Parse it per host, cache it for a day, re-check before each run.
One request at a time, per host, with a real gap. You are not building a search engine. A single connection with a one-to-three second delay is plenty — a fifty-page site takes two minutes. Concurrency is where teams accidentally turn research into a load test.
Conditional requests, so you stop re-downloading things that haven't changed. Store the ETag and Last-Modified you got last time and send them back. A 304 Not Modified costs the origin almost nothing and tells you exactly what you wanted to know.
import time, urllib.robotparser as rp
import requests
UA = "MyCompanyResearchBot/1.0 (+https://example.com/bot; [email protected])"
def allowed(base, path):
r = rp.RobotFileParser()
r.set_url(f"{base}/robots.txt")
r.read() # cache this per host, per day
return r.can_fetch(UA, base + path), (r.crawl_delay(UA) or 2.0)
def polite_get(base, path, etag=None, modified=None):
ok, delay = allowed(base, path)
if not ok:
return None # disallowed: stop, don't "work around" it
headers = {"User-Agent": UA}
if etag: headers["If-None-Match"] = etag
if modified: headers["If-Modified-Since"] = modified
resp = requests.get(base + path, headers=headers, timeout=20)
time.sleep(delay) # the gap is the whole point
if resp.status_code in (429, 503): # they said slow down
time.sleep(int(resp.headers.get("Retry-After", 60)))
return None
return resp # 304 = unchanged, and that's useful
Two details there matter more than the rest. Identify yourself in the User-Agent with a contact address: it turns you from an anonymous bot into someone a sysadmin can email before they block you, and in practice that email arrives instead of the block. And treat 429 and 503 as instructions, not errors to retry through — Retry-After is the server stating its rate limit, and hammering past it is the fastest route to a permanent ban.
Why the aggressive approach costs more than it returns
The case against aggressive scraping is usually made on ethics. For a small team, the arithmetic is more persuasive.
Blocks compound. A polite crawler runs for years untouched. An aggressive one gets its IP range blocked, then its User-Agent fingerprinted, then its proxy pool burned — each costing an engineer's afternoon plus a proxy bill, to produce exactly the same pricing table you'd have gotten by reading one page every six hours.
Your data quality gets worse, not better. Blocked sources fail silently. A dataset with three competitors quietly missing is more dangerous than no dataset, because someone will present it in a planning meeting. Coverage you can sustain beats volume you can't.
Volume rarely changes the decision. Ask what you'd do differently with a hundred pages per competitor per day versus one per week. For pricing, positioning, and content strategy the honest answer is almost always "nothing" — you're tracking changes, and changes are rare.
The legal and reputational tail is real. Terms violations, GDPR exposure on personal data, a reputation as the company that hammers small sites: all low-probability, all disproportionately bad at your size.
For the broader operational version of this — what to collect, why to prefer official APIs, and how privacy law applies to "public" data — the companion piece on automating competitor research and public data covers that ground.
Where anti-bot challenges fit — and where they don't
Here's the genuinely awkward case. You're doing everything right — public pricing page, one request every few seconds, robots.txt honored, User-Agent identifying you — and the page still hands your job a Cloudflare Turnstile or a reCAPTCHA, because the site's edge protection can't tell a polite research bot from a hostile one and challenges both. Your run stalls on the one thing a script can't manufacture, a valid challenge token, and that source silently drops out of your dataset.
That's the narrow, legitimate place where a solving service earns its keep. CaptchaAI is one option worth knowing about for a specific reason: it speaks the legacy 2Captcha-shaped protocol, so it drops into existing research tooling by changing a host rather than rewriting a pipeline. You submit to POST /in.php and poll /res.php with action=get&id=<taskId>, adding json=1 for structured responses instead of legacy plain text.
import time, requests
API = "https://ocr.captchaai.com"
KEY = "YOUR_API_KEY" # 32-character key
def solve_turnstile(site_key, page_url):
r = requests.post(f"{API}/in.php", data={
"key": KEY, "method": "turnstile",
"sitekey": site_key, "pageurl": page_url, "json": 1,
}, timeout=30).json()
if r.get("status") != 1:
raise RuntimeError(r.get("request")) # e.g. ERROR_ZERO_BALANCE
task_id = r["request"]
while True:
time.sleep(5) # documented poll cadence
res = requests.get(f"{API}/res.php", params={
"key": KEY, "action": "get", "id": task_id, "json": 1,
}, timeout=30).json()
if res.get("status") == 1:
return res["request"] # the token your request submits
if res.get("request") != "CAPCHA_NOT_READY":
raise RuntimeError(res["request"]) # ERROR_UNSOLVABLE, etc.
Two practical notes. Its published per-type figures — Cloudflare Turnstile 100% in under 10 seconds, Cloudflare Challenge above 99% in under 15 seconds (returning a clearance cookie plus the matching user agent) — are vendor claims, so verify them on your own sources during a trial rather than budgeting against them. On cost, pricing is thread-based: you buy concurrent threads with unlimited solves per thread and no per-CAPTCHA fee, published from BASIC at $15/mo for 5 threads up to ENTERPRISE at $300/mo for 200. That suits research better than per-solve billing because your bill tracks peak parallelism, which you control, not page count, which the sites decide. The trade-off: a low-volume job pays for capacity it doesn't use, so the smallest tier is usually the right buy.
And the boundary, stated plainly: a solver keeps permitted collection from stalling. It doesn't change which bucket a source is in. Using one to reach a login you don't own, a paywall, or a site whose terms forbid automated access is exactly the abuse this framework rules out — and the fastest way to earn the block you were trying to avoid.
A weekly cadence that beats a big crawl
The setup that works for a small team is unglamorous. Pick five competitors and three signals — pricing page, homepage headline, blog index. Fetch each weekly with conditional requests, so most runs return 304 and cost nothing. Diff against last week's copy and alert only on change. Keep a plain-text log of what changed and when.
That's roughly 15 requests a week. It will never get you blocked, and it catches every pricing move, repositioning, and content push your competitors make — which is the entire point. Scale it only when a specific decision demands data this doesn't give you.
When tracking outgrows a script — hundreds of keywords, backlink monitoring, ad-spend estimates — that's the moment to buy a platform rather than build one; compare the leading SEO tools on Machir to see which fits a small team's budget.
FAQ
Is it legal to scrape a competitor's public website?
It depends on what you take and how. Reading genuinely public pages at a respectful rate, honoring robots.txt and the site's terms, and collecting company-level rather than personal data is ordinary competitive research and broadly defensible. Collecting personal data without a lawful basis, ignoring terms that forbid automated access, or circumventing access controls to reach non-public content is not. If a specific source matters commercially, get advice on that source rather than relying on a general rule.
Does robots.txt legally bind me?
It's a convention, not a statute. But it's the clearest available statement of what a site consents to, and disregarding it is routinely treated as evidence of bad faith, including in terms-of-service disputes. Practically: honor it, and if a path you need is disallowed, ask for access or find another source.
How slow is slow enough?
One request at a time per host with a one-to-three second gap is a safe default, and any Crawl-delay in robots.txt overrides it. Add conditional requests so unchanged pages cost a 304 instead of a full download, and back off immediately on 429 or 503 for whatever Retry-After says.
Is using a CAPTCHA solver on a public page crossing the line?
It depends on what's behind the challenge. If the page is public, you're allowed to read it, and the challenge is edge protection that can't distinguish your polite bot from a hostile one, solving it to continue permitted collection is reasonable. If the challenge guards a login, a paywall, or a site whose terms forbid automation, solving it is how you cross the line — the challenge was the site's answer, and you'd be overriding it.
What should I do if a site asks me to stop?
Stop immediately, and reply. Being contactable is exactly why your User-Agent carries an email address. Most operators who block research bots are reacting to load, not principle, and a short note about what you collect and how often frequently ends in an allowance or a feed — an outcome unavailable to anyone crawling anonymously.
Where to start
Draw the buckets before you write any code; it's the only step that actually protects you. Then build the smallest polite job that answers one real question, identify yourself in the User-Agent, use conditional requests, and back off the instant a server tells you to. If challenges start eating coverage on sources you're clearly allowed to read, trial a solver like CaptchaAI on a small thread tier and measure how many permitted sources it actually keeps flowing before you commit. Automate the boring part, stay firmly in the fair-game bucket, and let current competitor data drive where you spend next.
