Most scraping projects don't die at the request stage. They die three days later, when the HTML you saved no longer matches the selectors you wrote, Playwright screenshots a captcha instead of a product page, and the "clean" CSV has a price column where half the values are strings like "from $19.99". The fetching was never the hard part.
The hard part is the boring pipeline around it: resilient selectors, dynamic rendering, proxy and header hygiene, rate limiting that doesn't get you banned, and normalization into something a df.groupby() won't choke on. An LLM is genuinely useful here — not as a magic scraper, but as a pair engineer that writes the boilerplate, explains why a selector broke, and turns messy extraction output into pandas code.
Below are 14 prompts organized by the stage they belong to. Each one has a task, the prompt itself, and a realistic result. Treat them as starting points — the quality of the output depends almost entirely on how much context (HTML snippets, error messages, sample rows) you paste in.
Stage 1: Designing the crawler before you write a line
1. The crawler blueprint prompt
Task: turn a vague goal ("scrape competitor prices") into a concrete architecture with components, failure points, and a storage schema.
Prompt:
You are a senior data extraction engineer. I need to scrape [TARGET SITE]
which lists [WHAT]. Constraints: [JS-rendered / paginated / login / etc.].
Produce: (1) a component diagram in text, (2) the crawl order and
politeness policy, (3) a storage schema (columns + types), (4) the top
5 failure modes and how to detect each early.
Be specific about which library handles which step.
Example result: the model returns a four-layer design — discovery (sitemap + pagination), fetch (httpx for static, Playwright only for JS routes), parse (BeautifulSoup for static, page.locator() for rendered), and normalize (pandas). The useful part is the failure table: pagination that silently stops at page 3, a 200 response that's actually a soft-block page, and duplicated records from overlapping category pages.
2. Static vs dynamic decision prompt
Task: decide whether you need Playwright at all.
Prompt:
Here is the raw HTML of a product page fetched with httpx (no JS).
Tell me whether the target data (price, title, rating) is present in
the static HTML or requires JS execution. Show the exact evidence:
which tags contain the values, or which container is empty.
Paste the HTML and the model points at <script id="__NEXT_DATA__"> or an empty <div id="root">. If the data lives in a JSON blob inside a script tag, you skip Playwright entirely and parse the JSON — faster, cheaper, and far less fragile.
Stage 2: Selectors and parsing (BeautifulSoup)
3. Selector generator with fallbacks
Task: get robust CSS selectors plus XPath alternatives.
Prompt:
Given this HTML snippet, write BeautifulSoup code that extracts
product name, price, and rating. For each field give:
- a primary CSS selector,
- an XPath fallback,
- a guard that returns None instead of raising.
Explain which selector is most likely to break on a site redesign.
Example result:
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
card = soup.select_one("div.product-card")
name = card.select_one("h2.title")
price = card.select_one("span.price")
rating = card.select_one("[data-rating]")
record = {
"name": name.get_text(strip=True) if name else None,
"price": price.get_text(strip=True) if price else None,
"rating": rating["data-rating"] if rating else None,
}
The model will usually flag that class names like css-1x2y3z are auto-generated and will rotate — anchor on data-* attributes or semantic tags instead.
data-* attributes and stable IDs over generated class hashes.
4. Parsing structured data from JSON blobs
Task: extract the embedded JSON instead of scraping rendered DOM.
Prompt:
This page embeds product data in a <script type="application/ld+json">
tag. Write Python that pulls it out, handles the case where multiple
blocks exist, and maps fields to a flat dict. Use json and BeautifulSoup.
JSON-LD is a schema.org standard (schema.org/Product) — sites publish it for search engines, which means it's usually cleaner than your hand-written selectors.
Stage 3: Dynamic rendering with Playwright
5. Playwright waiting strategy prompt
Task: replace sleep() with deterministic waits.
Prompt:
My Playwright script uses time.sleep(5) before reading the price.
Rewrite it to wait deterministically: wait for the network to settle,
wait for a specific selector, and fail loudly if the selector never
appears. Use sync_playwright and show the timeout values.
Example result:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto(url, wait_until="domcontentloaded")
page.wait_for_selector("span.price", timeout=15000)
price = page.locator("span.price").first.inner_text()
browser.close()
wait_until="domcontentloaded" plus an explicit wait_for_selector beats networkidle on sites with long-polling analytics scripts, which never let the network go idle.
6. Infinite scroll and pagination prompt
Task: handle "load more" and page-number pagination.
Prompt:
Write a Playwright loop that scrolls until no new items appear
(track the count of .card elements), with a hard cap of 20 iterations
and a 2-second pause. Then show the equivalent for click-based
"Load more" buttons that may disappear.
The model typically adds a seen set keyed on item IDs — the non-obvious insight here is that scroll loops often re-render existing nodes, so counting elements alone gives false positives. Track unique IDs, not counts.
.card elements to decide "did more load?" fails when the DOM virtualizes. Key on a stable item identifier instead.
Stage 4: Anti-bot hygiene, rate limits, and captchas
7. Header and proxy rotation prompt
Task: build a sane rotation layer without pretending to defeat every defense.
Prompt:
Write a rotating proxy + User-Agent provider for httpx. Requirements:
round-robin over a list, retry on 429/403 with exponential backoff and
jitter, and a max of 3 attempts per URL. Log which proxy served which
request. Do not invent headers — use realistic ones.
Example result: a class with a rotate() method, tenacity-style backoff (wait = min(60, 2 ** attempt) + random.uniform(0, 1)), and a note that residential proxies cost roughly an order of magnitude more than datacenter ones — a real trade-off you pay for in latency and money.
8. Rate limiter that respects robots and servers
Task: a token-bucket limiter instead of fixed sleeps.
Prompt:
Implement a token bucket rate limiter: N requests per interval, with
blocking acquire. Show how to wire it into an async httpx client and
why a fixed sleep between requests is worse under retries.
A fixed sleep(1) collapses under retries — five retries fire a burst of five requests. A token bucket smooths the burst.
9. Captcha and challenge handling prompt
Task: detect challenges and hand off, rather than pretending to solve them.
Prompt:
Write a detector that distinguishes: (a) normal page, (b) captcha
challenge, (c) soft block, (d) login wall. Use response text markers
and status codes. Then outline the safe handoff: pause, back off,
notify a human, and resume from a checkpoint. Do not write code that
bypasses captchas.
The honest answer is that automating around captchas is a moving target and often a Terms-of-Service problem. The valuable engineering is detecting the wall fast and checkpointing so a re-run doesn't redo 40,000 pages.
Stage 5: From raw records to a clean DataFrame
10. Price and currency normalization prompt
Task: turn "from $19.99" and "19,99 €" into a float plus a currency column.
Prompt:
Write a pandas function that takes a Series of price strings and returns
two columns: price (float) and currency (ISO code). Handle: thousands
separators, comma decimals, currency symbols, and "from"/"approx"
prefixes. Show the regex and a test with these 6 sample values.
Example result:
import re
def parse_price(s):
if not isinstance(s, str):
return None, None
cur = "USD" if "$" in s else "EUR" if "€" in s else None
num = re.sub(r"[^\d,\.]", "", s)
num = num.replace(",", ".") if num.count(",") == 1 and "." not in num else num.replace(",", "")
try:
return float(num), cur
except ValueError:
return None, cur
11. Deduplication and schema validation prompt
Task: catch duplicate products and rows that violate the schema.
Prompt:
Given a DataFrame with columns url, title, price, scraped_at, write:
(1) dedup on a normalized title+price key, keeping the latest;
(2) a validation report of rows where price is null or non-positive;
(3) a typed schema using pandas dtypes. Explain what each check catches.
This is where most pipelines silently rot — the row count looks fine, but 8% of prices are null and nobody notices until the dashboard flatlines.
12. PDF and multi-format extraction prompt
Task: handle PDFs and HTML mixed in one crawl.
Prompt:
I have a mix of HTML pages and PDF price lists. Write a dispatcher
that routes by content-type: PDFs to pdfplumber, HTML to BeautifulSoup,
and returns a common record shape. Show the pdfplumber table extraction
and a fallback for scanned PDFs (flag them, don't OCR silently).
Scanned PDFs return empty text — flagging them for a separate OCR pass beats a pipeline that quietly emits blank rows.
Stage 6: Monitoring the pipeline so it doesn't die quietly
13. Failure logging and alerting prompt
Task: structured logs and a triage report.
Prompt:
Add structured logging to this scraper: per-URL status, selector-miss
counter, and a run summary (pages attempted, succeeded, soft-blocked,
parsed-empty). Show a JSON log line and a summary that flags runs where
selector-miss rate exceeds 10%.
A rising selector-miss rate is the earliest signal of a site redesign — earlier than a crash, because the scraper keeps "succeeding" with empty fields.
14. Debug-the-broken-selector prompt
Task: the one you'll use most.
Prompt:
Yesterday this selector returned prices; today it returns None.
Here is the old HTML and the new HTML. Diff the structure, tell me
what changed, and give me a new selector plus a fallback that would
survive the change you just found.
Paste both snippets. The model diffs the DOM and usually spots the swapped wrapper or renamed class in seconds — the routine task that eats hours otherwise.
What actually keeps a scraper alive
The library choice matters far less than three habits: detect soft blocks fast, key your dedup on stable IDs, not counts, and watch the selector-miss rate, not just crashes. Prompts 1, 9, and 13 are the ones that pay for themselves — they're about failure detection, which is where serious extraction work lives.
If you want to go deeper into querying and shaping the resulting data, the SQL on Steroids picks up where the DataFrame ends, and the JavaScript/TypeScript workflow prompts cover the front-end side of what you're scraping. Start with prompt 1, paste real HTML, and let the model do the boilerplate — you keep the judgment calls about what's worth extracting.