The best web scraping bots in one place
Bots that collect structured data from sources. 0 bots in this category.
Scraping listings collect structured data from public sources into datasets you can open in a spreadsheet: link inventories, business directories, product pages, job postings, map results. The category is defined by the output - rows with named columns and a source attached - rather than by the technique used to get them. Bots built to be re-run and to report change are filed under Monitoring.
Expect published selectors, an explicit deduplication key, and a failure mode you can see: when detection misses, a run returns zero rows rather than plausible-looking data. Listings also state their access requirements honestly - browser binary, proxies, solver key - and carry a responsible-use notice naming the terms-of-service and data-protection obligations that remain yours.
Scraping listings run headless by default and declare on the page whether a browser binary, proxies or a solver key is needed. The launchers are the same four artifacts everywhere: a Windows .exe and .sh scripts for macOS Apple Silicon, macOS Intel, and Linux, delivered as a single raw file, with the executable bit restored on the shell-script platforms and the macOS quarantine attribute cleared before the first run. Long collection runs are easiest to schedule on Linux or macOS, where the launcher drops straight into cron.
Bots in this category
No bots in this category yet
Scraping bots: frequently asked questions
- Is web scraping allowed?
- Reading publicly published pages is generally possible, but each site's terms of service and robots.txt govern automated access, and the law in your jurisdiction governs what you may do with the result. Every scraping listing carries a responsible-use notice; reviewing it before the first run is your step, not the bot's.
- What happens when a target site redesigns?
- The run returns zero rows rather than invented data: extraction either matches real elements or records nothing, so a failure is visible. Selector repairs ship as a configuration update to the bot's id, which an already-downloaded launcher picks up at its next start on the live update channel.
- Do these bots need proxies or a CAPTCHA solver?
- Only where the listing declares it. Requirements are published per bot: whether proxies are needed, whether a solver key is required at run time, and whether a browser binary is used. Keys and proxy lists are supplied by you at the run prompt and are never embedded in the published configuration.
- What output does a scraping bot produce?
- Datasets written to files beside the launcher, one row per record, with the source page attached where the listing provides it. The field table on the Output tab lists every column, its type, its selector and the transformations applied to it.
- Does a scraper collect personal data?
- Sometimes, and the listing says so. Publicly published business contact details are personal data for sole traders in most jurisdictions. Each listing states its personal-data flag, what it collects, and the lawful-basis obligation that stays with you.