# Web Scraping & Crawling

> Web scraping, crawling, scraping APIs and headless browser automation.

- Parent category: [Developer Tools](https://devshelf.io/category/developer-tools.md)
- DevShelf page: https://devshelf.io/category/web-scraping-crawling

## Resources (16)

- [Extruct AI](https://devshelf.io/resource/extruct-ai.md): An AI-powered research platform that indexes and analyzes company data from the live web based on custom criteria. (pricing: not verified)
- [Firecrawl](https://devshelf.io/resource/firecrawl.md): API to search, scrape, and interact with the live web at scale, turning messy websites into clean, token-efficient markdown for LLMs. (pricing: not verified)
- [GitReverse](https://devshelf.io/resource/gitreverse.md): Tool that reverse engineers codebases or websites into structured prompts for large language models. (pricing: Free)
- [Jina AI](https://devshelf.io/resource/jina-ai.md): Search foundation and AI models for search relevance, embeddings, reranking, and URL-to-markdown reading. (pricing: Freemium)
- [Jsonify](https://devshelf.io/resource/jsonify.md): Automated data agents that monitor websites and apps to extract structured competitive intelligence, pricing, and product data at scale. (pricing: Freemium)
- [Puppeteer](https://devshelf.io/resource/puppeteer.md): A JavaScript library providing a high-level API to control Chrome or Firefox over the DevTools Protocol or WebDriver BiDi. (pricing: Free)
- [RapidProxy](https://devshelf.io/resource/rapidproxy.md): RapidProxy provides over 90 million rotating and static residential IP addresses across more than 200 global regions for web scraping and data collection. (pricing: Pay-as-you-go)
- [Reserp Search API](https://devshelf.io/resource/reserp-search-api.md): Google Search API providing programmatic access to SERP results with a structured schema and 99.8 percent production success rate. (pricing: Freemium)
- [Scrapfly](https://devshelf.io/resource/scrapfly.md): Web scraping API and cloud browser infrastructure with automated proxy rotation, JavaScript rendering, and pay-on-success pricing model. (pricing: Freemium)
- [ScrapingBee](https://devshelf.io/resource/scrapingbee.md): Web scraping API handling proxy rotation, headless browser rendering, and anti-bot defenses for reliable data extraction at scale. (pricing: Freemium)
- [Scrapling](https://devshelf.io/resource/scrapling.md): A Python web scraping framework providing fast HTTP requests, stealth browsers, powerful parsing, concurrent crawling, and an MCP server for AI agents. (pricing: Free)
- [Screenshot Scout](https://devshelf.io/resource/screenshot-scout.md): Website screenshot API that automatically strips cookie banners, chat widgets, and ads while supporting full-page rendering and custom viewports. (pricing: Freemium)
- [SerpApi](https://devshelf.io/resource/serpapi.md): Scrape Google and other search engines from a fast and easy API. (pricing: Freemium)
- [SocialKit](https://devshelf.io/resource/socialkit.md): API service for extracting social media transcripts, posts, profiles, and metrics across platforms like YouTube, TikTok, X, and LinkedIn. (pricing: Freemium, Pay-as-you-go)
- [TwitterAPI.io](https://devshelf.io/resource/twitterapi-io.md): Public social data API for X (formerly Twitter) built for developers and AI agents, offering real-time tweets, user profiles, and search endpoints. (pricing: Pay-as-you-go)
- [Xquik](https://devshelf.io/resource/xquik.md): API and scraper for searching and extracting X (Twitter) tweets, replies, followers, and profiles, with support for AI agents. (pricing: Paid)

---

Source: DevShelf.io, the hand-reviewed catalog of dev tools and resources. Every resource is reviewed by hand before it is listed.