<!-- skill: Web Scraping | version: 1.0.0 | description: Ethical web scraping — requests, BeautifulSoup, playbooks, politeness, legality | category: Web -->

## Instructions
- Check first: `robots.txt`, Terms of Service, and whether an official API exists (API always wins over scraping).
- Be polite: identify with a real User-Agent + contact, rate-limit (1-2s between hits), respect `Retry-After`/429 by backing off, never hammer login endpoints.
- Static pages: `requests` + BeautifulSoup (lxml parser). JavaScript-rendered: headless Chromium (Playwright) — heavier, use only when needed.
- Selectors: prefer stable attributes (`data-*`, ids) over fragile CSS paths; assert field presence and log URL+timestamp per record so failures are debuggable.
- Store raw HTML snapshots beside parsed output — when the site changes layout, you can re-parse without re-crawling.
- Legal/ethical lines: public data only, no credential stuffing, no bypassing paywalls/captchas, personal data minimized + never republished.

## Examples
- Price tracker: daily fetch of 20 product pages → parse price/stock → CSV + alert on change.
- Docs mirror: crawl sitemap URLs → save Markdown per page for RAG ingestion.

## Workflows
1. Confirm API absence + robots permission for the paths.
2. Fetch one page manually, map the DOM to fields.
3. Script fetch→parse→validate→store with retries + politeness delays.
4. Run small (10 pages), inspect output quality, then scale.
5. Schedule (systemd timer), monitor for layout-change failures, keep HTML snapshots 30 days.

## Templates
```python
# polite fetch + parse pattern
import time, requests
from bs4 import BeautifulSoup
S = requests.Session()
S.headers.update({"User-Agent": "MyBot/1.0 (+mailto:me@example.com)"})
def get(url, retries=3):
    for i in range(retries):
        r = S.get(url, timeout=20)
        if r.status_code == 429:
            time.sleep(5 * (i + 1)); continue
        r.raise_for_status()
        return r.text
    raise RuntimeError("rate-limited: " + url)
soup = BeautifulSoup(get("https://example.com"), "lxml")
items = [{"t": e.get_text(strip=True)} for e in soup.select("[data-item]")]
time.sleep(1.5)  # politeness between pages
```

## Tools
- SARAH Run for scripts; `pip install requests beautifulsoup4 lxml`; Playwright only if JS rendering is required.
- Validate output CSVs open correctly before scheduling.

## Documentation
- BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/bs4/doc
- robots standard: https://developers.google.com/search/docs/crawling-indexing/robots/intro