Bounded Web Scraping with Regex in JavaScript

Regular expressions can find a known text pattern in a response. They are generally a poor way to parse arbitrary HTML: markup is nested, attributes can be quoted or ordered in different ways, and real pages contain comments, entities, scripts and malformed fragments. A pattern that works on one fixed snippet can silently miss or combine content after the page changes.

Use a documented API or structured data format when the site provides one. For HTML that needs element relationships, use an HTML parser and select elements by their structure. Regex is appropriate for a bounded task such as extracting a value from a small, known fragment whose format you control or have documented.

A narrow example: one known price fragment

Given exactly this sample fragment:

<p class="price">$19.99</p>

A regular expression can capture the price text:

const sample = '<p class="price">$19.99</p>'
const match = sample.match(/<p class="price">\$([0-9]+\.[0-9]{2})<\/p>/)

console.log(match?.[1]) // 19.99

This pattern is intentionally specific. It does not handle arbitrary whitespace, attributes, currencies, nested elements or alternate markup. That limitation is useful here: it makes the assumption visible. If the source format varies, add explicit test cases or use a parser instead of expanding a fragile pattern until it appears to fit.

For example, in a browser you can parse HTML and query elements with the DOM using DOMParser:

const doc = new DOMParser().parseFromString(html, 'text/html')
const priceText = doc.querySelector('.price')?.textContent?.trim()

DOMParser is a browser API. In Node.js, choose a maintained HTML parser when a DOM-like API is needed; do not assume DOMParser is built into Node. Parsing untrusted HTML is still security-sensitive: do not insert parsed nodes into the live page or execute scripts from fetched markup just to extract data.

Fetch only pages you are allowed to access

Before making requests, identify an official API or feed, read the applicable site terms, and get permission where required. robots.txt is a crawler preference file, not authorization: RFC 9309 explicitly says its rules are not a form of access authorization. A path being allowed in robots.txt does not override terms, access controls, permission requirements or applicable law; a disallow rule is a signal to respect the site's crawler policy. If the requirements are unclear, do not proceed until you have permission or appropriate advice.

For a small, authorized request set, keep requests sequential and space out their start times. This Node.js example fetches each URL in order, waits at least one second between request starts, and stops on redirects rather than following a new destination automatically. Node's global fetch is available in current Node releases. The example URL is a placeholder; use only URLs you are allowed to request:

import { setTimeout as delay } from 'node:timers/promises'

const authorizedUrls = ['https://example.com/']
const minimumIntervalMs = 1000
let lastStartedAt = 0

for (const url of authorizedUrls) {
  const waitMs = Math.max(0, minimumIntervalMs - (Date.now() - lastStartedAt))
  if (waitMs > 0) await delay(waitMs)
  lastStartedAt = Date.now()

  const response = await fetch(url, { redirect: 'manual' })
  if (response.status >= 300 && response.status < 400) {
    throw new Error('Redirect received; check its destination and permission before requesting it')
  }
  if (response.status === 429) {
    throw new Error('Rate limited; stop and follow the server’s Retry-After guidance')
  }
  if (!response.ok) throw new Error(`Request failed: ${response.status}`)

  const html = await response.text()
  // Inspect only for the narrow, documented format you expect.
  console.log(`Received ${html.length} characters from ${new URL(url).origin}`)
}

Respect published rate limits and Retry-After responses; use a lower request rate when asked, and stop on errors or access-denied responses. A delay does not itself make a request authorized or guarantee that it is gentle enough. Do not parallelize requests against a small site without permission and a documented need.

Handle data carefully

Collect only fields needed for the stated purpose. Avoid collecting personal or sensitive data unless you have a valid basis and suitable safeguards. Do not put passwords, session cookies, API keys or other credentials in patterns, source files, URLs or logs. Use an authorized API's recommended credential mechanism, and store any required secrets outside the script. Do not republish or share extracted content without checking rights and permission.

Test the pattern against a fixture

Keep a small input with the expected result so markup changes do not silently break a narrow extraction:

const fixture = '<p class="price">$19.99</p>'
const pricePattern = /<p class="price">\$([0-9]+\.[0-9]{2})<\/p>/
const result = pricePattern.exec(fixture)

if (result?.[1] !== '19.99') {
  throw new Error('Price fixture did not match the expected value')
}

This verifies only the stated fixture; it does not prove the pattern handles a live site's full HTML. Test against the same JavaScript regex engine and representative permitted samples you intend to process. Avoid ambiguous nested quantifiers on long untrusted strings because poorly designed patterns can take excessive time. See the Node.js guide to regular-expression denial of service for why ambiguous patterns can block an event loop.

The site's Regex Tester can help inspect patterns, capture groups and replacement previews. It is a practice aid; it does not fetch pages, establish permission, or verify that a pattern is safe for every input. Its JavaScript regex testing Guide covers pattern behavior. If you need to reshape authorized match data, the Data Converter supports JSON and CSV workflows; check the tool's disclosure before entering sensitive data.

Choose the right method

A reliable extraction workflow begins with a permitted source and a specific data need. Use regex only for the part it can express clearly, keep request volume low, and verify the result before relying on it.