apify/crawlee-python
# Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
$ git clone https://github.com/apify/crawlee-python.git
stars
9,349
forks
777
language
Python
license
Apache License 2.0
What is apify/crawlee-python?
Crawlee is a Python library for building web scrapers and browser automation tools that can extract data from websites in various formats including HTML, PDF, and images. It supports multiple parsing and automation frameworks (BeautifulSoup, Parsel, Playwright) and includes features like proxy rotation and headless browser support, making it suitable for data extraction tasks ranging from simple scraping to feeding data into AI systems. Developers use it when they need a flexible, multi-tool approach to web crawling that can handle both lightweight HTTP requests and full browser automation depending on the target website's requirements.
Links
Topics
Activity
78 open issues · last updated Jul 21, 2026