# Web Scraping Learn how to use Tools and Techniques ## TL;DR Web scraping extracts data from websites using tools like Playwright (best-in-class), Beautiful Soup, and Scrapy. Ethical scraping respects robots.txt, rate limiting, and legal terms; LLM-powered extraction handles unstructured data; and structured data (JSON-LD, microdata) is always preferable when available. ## Introduction The web is full of data. Job listings, product prices, real estate listings, research papers — all trapped in HTML, waiting to be analyzed. Web scraping is the art of extracting that data programmatically. But web scraping exists in a gray zone. Some sites welcome scrapers. Others aggressively block them. Copyright and terms-of-service concerns lurk. Legal liability is real. In 2026, the scraping landscape has evolved. Anti-scraping tools have become more sophisticated. But so have scraping libraries. Playwright has emerged as the dominant browser automation tool. LLMs have opened new possibilities for extracting data from unstructured pages. And increasingly, responsible developers are asking: "Should I scrape, or should I use the public API or structured data instead?" This guide covers ethical scraping techniques, modern tools, and when not to scrape. ## Ethical Scraping: The Foundation Before writing a single line of code, understand the ethical and legal landscape. ### Respecting robots.txt Every website has a `/robots.txt` file. It specifies which parts of the site are available for scraping and how fast you should crawl. ``` # amazon.com/robots.txt User-agent: * Disallow: /s Disallow: /gp/ Disallow: /*/dp/ Disallow: /cart/ # Googlebot is allowed everywhere; others are restricted User-agent: Googlebot Disallow: ``` This says: "General scrapers, don't scrape /s (search), /gp/ (pages), or /*/dp/ (product detail). Googlebot, you're fine." Fetch robots.txt, parse it, and respect its rules: ```python import requests from urllib.robotparser import RobotFileParser rp = RobotFileParser('https://example.com/robots.txt') rp.read() if rp.can_fetch('*', 'https://example.com/page'): print('OK to scrape') else: print('Not allowed by robots.txt') ``` ### Rate Limiting Even if scraping is allowed, don't hammer the site. Add delays between requests. ```python import time import requests urls = ['https://example.com/page1', 'https://example.com/page2'] for url in urls: response = requests.get(url) process_response(response) time.sleep(2) # 2-second delay between requests ``` A well-behaved scraper adds: - 2-5 second delays between requests - A `User-Agent` header identifying your scraper - Backing off if you get 429 (Too Many Requests) responses ### Legal Considerations Scraping isn't inherently illegal, but several factors determine legality: 1. **Terms of Service**: Does the site explicitly forbid scraping? If so, violating ToS has legal risk. 2. **Copyright**: You can scrape public information, but republishing copyrighted content without permission is infringement. 3. **Computer fraud**: Don't bypass authentication or CAPTCHA. Don't use scrapers to commit fraud. 4. **Data protection**: If the data involves personal information (email addresses, phone numbers), GDPR and similar laws apply. **Safe practices:** - Check the site's robots.txt and ToS - Scrape only what you need - Don't republish copyrighted content directly - Use scraped data for analysis, not commercial redistribution - Ask permission when in doubt ## Playwright: The Modern Standard Playwright is the best browser automation tool in 2026. It controls headless browsers (Chrome, Firefox, Safari) and handles JavaScript-heavy sites. ### Basic Usage ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto('https://example.com') title = page.title() print(title) browser.close() ``` Playwright renders JavaScript, handles cookies, and manages browser sessions. It's perfect for single-page apps and dynamic content. ### Scraping Dynamic Content Modern sites load data via JavaScript. Traditional HTTP requests only get the initial HTML. Playwright waits for JavaScript to render, then captures the populated DOM. ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch() page = browser.new_page() # Navigate and wait for data to load page.goto('https://example.com/products') page.wait_for_selector('.product-item') # Wait for products to load # Extract product names and prices products = page.query_selector_all('.product-item') data = [] for product in products: name = product.query_selector('.name').text_content() price = product.query_selector('.price').text_content() data.append({'name': name, 'price': price}) print(data) browser.close() ``` The `wait_for_selector()` is crucial — it ensures the page has finished loading before you extract data. ### Handling Anti-Bot Measures Websites block scrapers to protect their servers. Playwright helps you work around this: ```python # Add realistic delays page.wait_for_timeout(1000) # 1 second delay # Use a realistic User-Agent context = browser.new_context( user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' ) # Rotate through proxy servers (if heavily blocked) context = browser.new_context(proxy={'server': 'http://proxy.example.com:8080'}) # Handle cookies and session state context.add_cookies([{'name': 'session', 'value': 'abc123'}]) ``` But beware: if a site actively blocks scrapers with CAPTCHAs and IP bans, bypassing those measures may violate computer fraud laws. Respect the site's intent to block you. ## Beautiful Soup: Parsing Static HTML For sites that serve complete HTML in the initial response (no JavaScript), Beautiful Soup is lightweight and powerful. ```python from bs4 import BeautifulSoup import requests response = requests.get('https://example.com/products') soup = BeautifulSoup(response.content, 'html.parser') # Extract product elements products = soup.find_all('div', class_='product') for product in products: name = product.find('h2').text price = product.find('span', class_='price').text print(f'{name}: {price}') ``` Beautiful Soup is simpler than Playwright, faster, and uses less memory. Use it for static sites; save Playwright for dynamic ones. ## Scrapy: Enterprise-Scale Scraping Scrapy is a full-featured framework for large-scale scraping projects. It handles: - Concurrent requests (multiple sites simultaneously) - Request scheduling and prioritization - Middleware for handling cookies, proxies, and user agents - Built-in robots.txt and rate limiting support - Exporting data to CSV, JSON, or databases ```python import scrapy class ProductSpider(scrapy.Spider): name = 'products' start_urls = ['https://example.com/products'] def parse(self, response): for product in response.css('div.product'): yield { 'name': product.css('h2::text').get(), 'price': product.css('.price::text').get(), } # Follow pagination links next_page = response.css('a.next::attr(href)').get() if next_page: yield response.follow(next_page, self.parse) ``` Run it: ```bash scrapy crawl products -o output.json ``` Scrapy is overkill for one-off projects but essential for scraping large sites with millions of pages. ## LLM-Powered Data Extraction Large language models can extract structured data from unstructured HTML, dramatically simplifying scraping code. ```python import anthropic html = """

MacBook Pro 16-inch

$1,999

Powerful laptop for developers and creators

""" client = anthropic.Anthropic() message = client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[ { "role": "user", "content": f"Extract product name, price, and description from this HTML:\n\n{html}" } ] ) print(message.content[0].text) # Output: # Product Name: MacBook Pro 16-inch # Price: $1,999 # Description: Powerful laptop for developers and creators ``` LLMs handle messy HTML, variations in structure, and natural language descriptions. They're slower and more expensive than regex parsing but far more reliable. ## Structured Data: The Best Alternative Before scraping, check if the site provides structured data. Many sites embed JSON-LD (Linked Data) in the page. ```html ``` Extract it: ```python import json from bs4 import BeautifulSoup soup = BeautifulSoup(html, 'html.parser') script = soup.find('script', {'type': 'application/ld+json'}) data = json.loads(script.string) print(data['name']) # MacBook Pro ``` JSON-LD is cleaner, faster, and legally safer than parsing HTML. Use it whenever available. ## Checklist for Responsible Scraping - [ ] Check robots.txt — respect crawl delays and disallowed paths - [ ] Read the site's ToS — is scraping explicitly forbidden? - [ ] Add delays between requests (2-5 seconds minimum) - [ ] Identify your scraper with a User-Agent header - [ ] Handle rate limiting gracefully (check for 429 responses) - [ ] Respect copyright — don't republish full content - [ ] Consider using public APIs or structured data instead - [ ] Avoid scraping personal data; if necessary, follow GDPR rules - [ ] Stop if blocked — don't try to bypass CAPTCHAs or IP bans - [ ] Log your scraping activity for auditing ## Conclusion Web scraping is a powerful tool for extracting data from the web. But power comes with responsibility. Always start by checking robots.txt and the site's terms of service. Scrape respectfully — add delays, identify yourself, and back off if blocked. Choose your tool wisely: Playwright for dynamic sites, Beautiful Soup for static HTML, Scrapy for enterprise-scale projects, and LLMs for unstructured data. And before writing any scraper, ask yourself: "Is there a public API or structured data I could use instead?" The line between respectful scraping and abuse is thin. Stay on the right side of it.

#web-scraping

Web Scraping Learn how to use Tools and Techniques

Stay on the Nerd Track