Web Scraping: Trends and Complexities

Fetching a page and pulling values out of it takes an afternoon. Running that reliably for two years, against a site that changes its markup without warning and would prefer you did not, is a different undertaking entirely — and it is the one people accidentally commit to.

Check whether you need to scrape at all

Work through these before writing anything, because each is dramatically cheaper than a scraper:

• An official API. Slower to get access to, infinitely cheaper to maintain. Stable contracts, documented changes, no blocking.
• A bulk export or data feed. Many organisations publish datasets that nobody discovers because scraping was easier to start.
• A commercial data provider. If the data has commercial value, someone is probably already licensing it. Compare their price against a year of an engineer’s maintenance time before dismissing it.
• A partnership. If you need one company’s data continuously, ask them. It happens more often than teams expect.

The legal and ethical position

This is not legal advice and the specifics vary by jurisdiction, but the landscape has some stable features worth knowing.

Scraping publicly accessible data is not inherently unlawful. US case law around hiQ v. LinkedIn established that accessing public pages does not, by itself, constitute unauthorised access under the Computer Fraud and Abuse Act. That is a narrower finding than it is usually quoted as.

What actually creates exposure:

• Terms of service. Scraping in breach of terms you accepted can be a contractual matter even where it is not a criminal one.
• Authentication. Data behind a login is categorically different from public data. Circumventing access controls is where legal risk escalates sharply.
• Personal data. Under GDPR, scraping personal data means you are processing it, with all the obligations that follow — lawful basis, subject rights, the lot. This catches a great many scraping projects by surprise.
• Copyright. Facts are generally not copyrightable; the specific expression of them is. Republishing scraped article text is a different act from extracting prices.
• Load. Aggressive scraping that degrades a site can be treated as an attack, entirely reasonably.

Why it has got harder

The technical difficulty has increased markedly, and for identifiable reasons.

Client-side rendering

A significant share of the modern web ships an empty HTML shell and builds the page in JavaScript. An HTTP client sees nothing useful.

The obvious answer is a headless browser — Playwright or Puppeteer — which is also fifty to a hundred times more expensive per page in CPU and memory. Before reaching for one, open the network tab: very often the page is calling a JSON endpoint you can call directly, which is faster, more stable and easier to parse than the rendered DOM.

Bot detection

Commercial anti-bot services now fingerprint far more than the user agent: TLS handshake characteristics, HTTP/2 frame ordering, canvas and font rendering, mouse movement, timing patterns. This is an arms race, and it is one where the defenders are better resourced than you are.

You can win it temporarily. You cannot win it permanently, and a scraping strategy whose viability depends on staying ahead of a vendor whose entire business is stopping you is a strategy with an expiry date nobody has written down.

Markup churn

The mundane one that causes the most actual pain. Sites redesign. Class names generated by build tools change on every deploy. A selector that worked for eight months silently matches nothing, or worse, matches the wrong element.

Defences that help: prefer semantic and structural selectors over generated class names; prefer data-* attributes and microdata where present; parse JSON-LD from the page, which is frequently richer and far more stable than the visible markup; and never rely on positional selectors like :nth-child, which break the moment anything is inserted.

Building something that lasts

Separate fetching from parsing

The single most valuable structural decision. Fetch pages and store the raw HTML. Parse from storage in a separate step.

This means a parser bug is fixed by reparsing what you already have rather than by refetching everything — which is faster, cheaper, and does not risk your access. It also gives you a record of what the page actually said on a given date, which is invaluable when someone questions the data.

Validate every extraction

Assert on what you extract, and fail loudly:

• Required fields are present and non-empty.
• Types and formats are as expected — a price parses as a number, a date as a date.
• Values fall in plausible ranges. A price of zero or 900 million is a parsing failure, not a bargain.
• Record counts are within expectations. A page that yielded 200 items yesterday and 3 today has changed.

Alert on silence

The dangerous failure is not the scraper that crashes; that gets noticed. It is the one that runs successfully and returns nothing, or returns plausible nonsense, for three weeks. Alert on zero results, on sudden drops in extraction rate, and on validation failure rates crossing a threshold.

Be a good client

• Rate limit, with jitter, well below what the site could tolerate.
• Send conditional requests using ETag and Last-Modified so unchanged pages cost almost nothing.
• Back off exponentially on 429 and 5xx responses, and honour Retry-After.
• Cache aggressively. Every page you fetch twice is a page you did not need to fetch.
• Run overnight in the target’s timezone.

Where LLMs help, and where they do not

Language models have genuinely changed part of this. Given a page, a model can extract structured data without a hand-written selector, and it survives markup changes that would break a traditional parser. For heterogeneous sources — a hundred sites with a hundred layouts — this is a real advance over writing a hundred parsers.

What has not changed: they are far more expensive per page than a selector, they are slower, they hallucinate plausible values when the data is absent, and they are non-deterministic in a domain where reproducibility matters.

The pattern that works is hybrid. Use a model once to generate selectors for a new source, then run the cheap deterministic parser in production and fall back to the model only when validation fails. You get the adaptability where it is needed and the cost profile of selectors everywhere else.

Boolean Solutions experience with scraping

We have built extraction pipelines for pricing intelligence, catalogue aggregation and research datasets. The pattern is consistent: the first version works in a week and the maintenance runs for years, and teams that budgeted only for the first version end up with a pipeline nobody trusts.

We also regularly find that an API or a licensed feed exists and would cost less than a quarter of the engineering time. If you are planning a data acquisition project, talk to us before you commit to scraping it.

Further reading

• The Robots Exclusion Protocol — what robots.txt actually specifies
• Playwright — the most capable headless browser automation tool currently available
• Scrapy — a mature framework that handles scheduling, throttling and pipelines properly
• ICO: lawful basis for processing — relevant the moment your scraper touches personal data
• Analytics: Planning the Right Database — where scraped data usually needs to land