Crawlee 🕸 The web scraping and browser automation library

The world's most extensive open-source library for building fast and reliable web crawlers in Node.js

Hello world 👋

We're Jan and Jakub, founders of Apify, the web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee, the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production.

For details, see the short video below or read the announcement blog post.

Main features

🖼 Supports headless browsers with Playwright or Puppeteer

⚡️ Supports raw HTTP crawling with Cheerio or JSDOM

🎛 Automated parallelization and scaling of crawlers for best performance

🐾 Avoids blocking using smart sessions, proxies, and browser fingerprints

🗃 Simple management and persistence of queues of URLs to crawl

🗜 Written completely in TypeScript for type safety and code autocompletion

📚 Comprehensive documentation, code examples, and tutorials

💪🏼 Actively maintained and developed by Apify—we use it ourselves!

🤝 Lively community on Discord

Getting started

Visit crawlee.dev or run the following command:

npx crawlee create my-crawler

Show me the code

Liked Crawlee?

💛 You can support the project on GitHub and Product Hunt, or discuss it on Hacker News.