Skip to content

Web Scraping & Crawling

16 resources found

An AI-powered research platform that indexes and analyzes company data from the live web based on custom criteria.

AI-powered

API to search, scrape, and interact with the live web at scale, turning messy websites into clean, token-efficient markdown for LLMs.

Open SourceSelf-hosted

Tool that reverse engineers codebases or websites into structured prompts for large language models.

FreeNo sign-up

Search foundation and AI models for search relevance, embeddings, reranking, and URL-to-markdown reading.

Freemium

Automated data agents that monitor websites and apps to extract structured competitive intelligence, pricing, and product data at scale.

Freemium

A JavaScript library providing a high-level API to control Chrome or Firefox over the DevTools Protocol or WebDriver BiDi.

FreeOpen Source

RapidProxy provides over 90 million rotating and static residential IP addresses across more than 200 global regions for web scraping and data collection.

Pay-as-you-go

Google Search API providing programmatic access to SERP results with a structured schema and 99.8 percent production success rate.

Freemium

Web scraping API and cloud browser infrastructure with automated proxy rotation, JavaScript rendering, and pay-on-success pricing model.

Freemium

Web scraping API handling proxy rotation, headless browser rendering, and anti-bot defenses for reliable data extraction at scale.

Freemium

A Python web scraping framework providing fast HTTP requests, stealth browsers, powerful parsing, concurrent crawling, and an MCP server for AI agents.

FreeAI-poweredOpen SourceSelf-hosted

Website screenshot API that automatically strips cookie banners, chat widgets, and ads while supporting full-page rendering and custom viewports.

Freemium

Scrape Google and other search engines from a fast and easy API.

Freemium

API service for extracting social media transcripts, posts, profiles, and metrics across platforms like YouTube, TikTok, X, and LinkedIn.

FreemiumPay-as-you-go

Public social data API for X (formerly Twitter) built for developers and AI agents, offering real-time tweets, user profiles, and search endpoints.

Pay-as-you-go

API and scraper for searching and extracting X (Twitter) tweets, replies, followers, and profiles, with support for AI agents.

Paid