# RTILA Marketplace > RTILA Marketplace is a directory of standalone desktop automation bots. Definition: a standalone automation bot is a compiled launcher executable, named after the bot it runs, which fetches that bot's published configuration from the backend at launch and automates desktop work on the user's machine. Every listing is free and runs locally. This file inlines the full catalogue content; the curated index lives at /llms.txt. ## AppSumo Product Catalog Scraper Search AppSumo and collect deal names, prices, ratings, review counts and countdown data from every listing. - URL: https://marketplace.rtila.com/bots/appsumo-product-catalog-scraper - Category: E-commerce - Tags: appsumo, ecommerce, product-catalog, software-deals, lifetime-deals, price-tracking - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-10-05) - Price: free ### Overview The AppSumo Product Catalog Scraper is an AppSumo scraper: a standalone desktop bot that opens the AppSumo search page, expands the results grid until every deal card is rendered, and pulls roughly thirty catalog fields per listing into the `appsumo_listings` dataset. It never leaves the search page - there are no per-product page visits in this configuration - which is why a full-catalog run finishes in minutes instead of hours. Everything it reads is exactly what a logged-out shopper sees on appsumo.com: deal names, taglines, categories, current and original prices, star ratings, review counts, countdown badges, AppSumo Select flags and card images. This listing is not affiliated with or endorsed by AppSumo. One honesty note on the selectors: they target AppSumo's current public markup and will need updating when AppSumo redesigns its cards - when that happens, a run ends with empty fields or zero rows rather than invented ones. ## What does the AppSumo Product Catalog Scraper do? The run is one expansion phase followed by one extraction. The bot lands on the AppSumo search URL, composed from a single optional settings variable, `search_query`. Leave that variable empty and the search returns the full catalog - 334-plus active products at the time this configuration was written. Type a keyword such as seo or video hosting and the same run collects only the matching slice of AppSumo's deals. Expansion happens in two passes because the deal grid lazy-loads in two ways. First the run presses Escape once to dismiss cookie or promo overlays that would otherwise sit on top of the load-more control. Then it scrolls the page twice with a 2500 ms pause to wake lazy rendering, and enters a click loop: as long as a load-more control is visible - the configuration carries four label variants (Show More Results, Load More, Show more results, View More) to survive minor copy changes - the bot clicks it and waits three seconds for the next batch of cards. When no load-more control remains, a single extraction pass walks every card. The item selector anchors on AppSumo's deal price container inside each card and climbs to the enclosing card element, so the dataset stays bound to real listing cards rather than arbitrary grid children. `product_url` and `product_name` are declared required fields, and rows are deduplicated by `product_url`, so a card rendered twice during expansion lands exactly once. Because everything runs on your own machine, your research footprint stays yours: which categories you size, which competitors you benchmark, which price bands you study never passes through a third-party service's dashboard or logs. ## Who uses an AppSumo scraper? The SaaS founder pricing an AppSumo launch asks "what discount depth do comparable tools open at?" and needs `price_current` next to `price_original` across a whole category. The affiliate and newsletter operator asks "which AppSumo deals expire this week, and which are rated well enough to recommend?" and needs `deal_days_remaining`, `rating` and `review_count` in one dated table. The market researcher asks "how dense is AppSumo's productivity category, and how fast do new launches accumulate reviews?" and re-runs monthly to measure review velocity. The vendor-side competitive analyst asks "do rival products carry the AppSumo Select badge, and at what billing period?" and needs `is_appsumo_select` and `price_period` per row. ## How does an AppSumo catalog scrape run, step by step? Every command in the run links to its reference page. First, [press](https://learn.rtila.com/reference/commands/press) sends the Escape key to the page root, clearing any consent or promo overlay. Second, [infinite_scroll](https://learn.rtila.com/reference/commands/infinite_scroll) runs a capped pass - two scrolls maximum, 2500 ms apart - which is enough to trigger the lazy-rendered grid without chasing the footer endlessly. Third, a [while](https://learn.rtila.com/reference/commands/while) loop checks for a visible load-more button across four label variants, and each iteration [click](https://learn.rtila.com/reference/commands/click)s it and [wait](https://learn.rtila.com/reference/commands/wait)s 3000 ms for the next batch to render. Fourth, [extract_data](https://learn.rtila.com/reference/commands/extract_data) pulls the `appsumo_listings` dataset once, after expansion is complete - card-level extraction, one pass, deduplicated by `product_url`. This release sets no fixed export filename, so rows land in the launcher's local results store and you export them from the results view; nothing is uploaded anywhere except the configuration fetch and download telemetry listed on this page. ## What does the output look like? The table mirrors the published sample row (fourteen of the roughly thirty fields - the dataset carries the rest). Sample values are illustrative placeholders that show the shape of a real row: | product_name | product_slug | tagline | category_name | price_current | price_original | price_currency_symbol | price_period | rating | review_count | deal_badge_type | deal_days_remaining | listing_position | product_url | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | FlowDesk Pro | flowdesk-pro | Visual workflow planner for solo consultants | Productivity | 59 | 240 | $ | lifetime | 4.5 | 312 | deal_ending | 4 | 1 | https://appsumo.com/products/flowdesk-pro | **product_name** is the deal title and a required field - an empty name means the card never rendered properly, and the row is logged rather than trusted. **product_url** and **product_slug** identify the listing; `product_url` is the deduplication key (the appsumo.com/products/ link, with any anchor fragment stripped), and the slug is regex-extracted from it, giving you a short stable join key when you diff two dated snapshots. **tagline** is the clamped card description - the one-line pitch AppSumo shows shoppers. **category_name**, **category_url**, **category_slug**, **category_group_slug** and **category_parent_slug** capture AppSumo's category taxonomy at three levels - the group and parent slugs are parsed straight out of the appsumo.com path structure - and **categories** is a nested list of every category link on the card with its name, URL and depth level. **price_current** and **price_original** are the deal price and the struck-through reference price, both cast to numbers with a null fallback; an empty original means the card shows no discount. **price_currency_symbol**, **price_period** and **price_raw** keep the symbol, the billing period (lifetime, year, month) and the untouched display text for auditing. **rating**, **rating_label**, **review_count**, **review_count_text** and **review_url** are the aggregate social-proof signals pulled from the star image and the reviews anchor - numeric casts fall back to null on brand-new listings, and review text is never collected. **deal_badge_text** is the countdown strip verbatim; **deal_badge_type** normalises it to deal_ending, price_increase or plus_access_ending, empty otherwise; **deal_days_remaining** is the integer day count parsed out of the strip. **is_appsumo_select** counts the AppSumo Select badge image and **appsumo_select_badge_url** keeps its source; **plus_exclusive_label** captures the Plus exclusive suffix on the product name when present. **product_image_url** (tracking query stripped), **product_image_url_resized** and **product_image_alt** carry the card artwork. **listing_position** is the 1-based card order in the results, and **card_text_snapshot** is the full collapsed text of the card - a fallback field that keeps a run useful even when one specific selector breaks. ## Can AppSumo deal data feed competitive research? Yes - with the boundary stated plainly. This bot gives you a dated, structured snapshot of card-level truth on AppSumo: what is listed, in which category, at what price against what reference price, with what social proof and what urgency signals. What it does not do is go deeper or enrich: no product detail pages, so no full descriptions, plan tiers, launch dates or vendor information; no review text; no validation against third-party databases. The intended analysis pattern is diffing two snapshots on `product_url` - price moves, review velocity, badge changes, new launches and quiet removals all fall out of that comparison. For the review stream itself, the sibling AppSumo Product Reviews Scraper is the companion configuration. ## How does lifetime deal research work with AppSumo data? Lifetime-deal economics live in a handful of these columns. `price_period` separates true lifetime deals from disguised subscriptions. Discount depth - `price_original` against `price_current` - shows how aggressively a category prices its AppSumo launches. `deal_days_remaining` and `deal_badge_type` turn the catalog into a prioritised expiry calendar for roundups or last-look competitive reviews. `is_appsumo_select` and `plus_exclusive_label` segment the editorially promoted slice from the open catalog, which is where pricing norms for featured placement become visible. Run the snapshot weekly, join on `product_url`, and the panel data builds itself. ## How does it compare to hosted AppSumo scraping services? Copy-pasting cards by hand produces a handful of rows an hour and no consistency at all. Hosted robot and actor marketplaces do offer AppSumo collectors, but the structure is the opposite of this bot: they meter by the result or by the month, the data lands in their cloud under their retention terms, and your search history - the categories you size, the competitors you watch - lives on their servers. Here the marginal cost of a bigger search is your own time and bandwidth, the datasets stay on your disk, and the workflow configuration is yours to keep, modify and re-run. The trade is real: no dashboards, no built-in scheduling, no cloud-side retry. You get an artifact and a workflow you own outright. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the downloaded .exe directly - there is no installation step; SmartScreen may ask for confirmation on first launch, so verify the published checksum, then choose More info and Run anyway. On **macOS**, two launchers are published - pick the one matching your machine's chip (Apple menu, About This Mac: Apple Silicon or Intel) - then make it executable with `chmod +x`, clear the Gatekeeper quarantine with `xattr -d com.apple.quarantine /path/to/BOT_appsumo-product-catalog-scraper.sh` and launch. On **Linux**, `chmod +x ./BOT_appsumo-product-catalog-scraper.sh` then `./BOT_appsumo-product-catalog-scraper.sh`. At first run the launcher asks for the optional `search_query` value - empty for the full AppSumo catalog, a keyword for a slice - and starts the run. To repeat a snapshot on a cadence, schedule the launcher with your OS scheduler (Task Scheduler, cron or launchd). ## What do I check when a run returns zero rows? Work in order. Zero rows: did the AppSumo search page render in an ordinary browser, or did a consent wall, region lock or challenge greet the fresh session - the single Escape press dismisses common overlays, not all of them. A suspiciously small row count: the load-more loop exits when none of its four button labels matches, so a copy change on that control truncates the run silently - compare the row count against what appsumo.com shows interactively before trusting a snapshot. Rows present but one column empty across every row: that field's selector most likely changed in an AppSumo redesign; check one card by hand. Empty rating or review_count on scattered rows: completely normal for new listings - the numeric casts fall back to null rather than failing the row. The contract is that failure is loud - empty values and zero rows, never fabricated data. ## What are the bot's limits? It reads AppSumo search-result cards only: no detail-page fields, no review text, no photos beyond card images, no vendor or company data. It ships without proxy rotation or CAPTCHA solving enabled - if a challenge stops the run, the run stops; nothing bypasses it. Expansion is bounded by the load-more labels it recognises, so an AppSumo redesign can truncate coverage, which is why the row-count check above matters. One run covers one search term. It is a snapshotter, not a monitor - nothing re-runs, diffs or alerts on its own. Deduplication applies within a run on `product_url`; merging across runs is your step. ## What is responsible AppSumo data collection? The fields collected here are public commercial data - product names, prices, aggregate ratings - and this configuration deliberately stops short of review text and reviewer identities, which keeps personal data out of the dataset entirely. That does not remove your obligations: automated collection may conflict with AppSumo's terms of service, so review those terms with open eyes before each run and let the acceptable-use policy govern it. Keep request volume at what a patient human could sustain - the built-in waits already pace the run that way. Date every snapshot, and if you republish derived analysis, do so in ways AppSumo permits. This bot is an independent tool, not affiliated with or endorsed by AppSumo. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | search_query | text | no | | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **appsumo_listings** (item selector: `xpath=//div[@data-testid='deal-price-container']/ancestor::*[contains(@class,'bg-white') and contains(@class,'rounded')][1]`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | product_name | text | no | css=span.overflow-hidden.text-ellipsis.whitespace-nowrap.font-bold,css=a[href*='/products/']:not([href*='#']) | | product_url | attribute | no | css=a[href*='/products/']:not([href*='#']),css=a[href*='/products/'] | | product_slug | attribute | no | css=a[href*='/products/']:not([href*='#']),css=a[href*='/products/'] | | product_image_url | attribute | no | css=img.aspect-sku-card | | product_image_url_resized | attribute | no | css=img.aspect-sku-card | | product_image_alt | attribute | no | css=img.aspect-sku-card | | is_appsumo_select | count | no | css=img[alt='AppSumo Select'] | | appsumo_select_badge_url | attribute | no | css=img[alt='AppSumo Select'] | | deal_badge_text | text | no | css=div.absolute.bottom-0.left-0 span | | deal_badge_type | text | no | css=div.absolute.bottom-0.left-0 span | | deal_days_remaining | text | no | css=div.absolute.bottom-0.left-0 span | | category_name | text | no | css=a.relative.z-1.underline | | category_url | attribute | no | css=a.relative.z-1.underline | | category_slug | attribute | no | css=a.relative.z-1.underline | | category_group_slug | attribute | no | css=a.relative.z-1.underline | | category_parent_slug | attribute | no | css=a.relative.z-1.underline | | categories | list | no | css=a[href]:not([href*='/products/']):not([href^='#']):not([href^='javascript']) | | tagline | text | no | css=div.line-clamp-3 | | rating | attribute | no | css=img[alt*='stars'] | | rating_label | attribute | no | css=img[alt*='stars'] | | review_count_text | text | no | css=a[href*='#reviews'] | | review_count | text | no | css=a[href*='#reviews'] | | review_url | attribute | no | css=a[href*='#reviews'] | | price_raw | text | no | css=[data-testid='deal-price-container'] | | price_current | text | no | css=#deal-price | | price_currency_symbol | text | no | css=#deal-price | | price_period | text | no | css=#deal-price-suffix | | price_original | text | no | css=#deal-price-original | | plus_exclusive_label | text | no | css=span.overflow-hidden.text-ellipsis.whitespace-nowrap.font-bold,css=a[href*='/products/']:not([href*='#']) | | listing_position | index | no | | | card_text_snapshot | text | no | css=self | ### Downloads - Windows x86_64: `BOT_appsumo-product-catalog-scraper.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/appsumo-product-catalog-scraper/windows - macOS Apple Silicon: `BOT_appsumo-product-catalog-scraper.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/appsumo-product-catalog-scraper/macos - macOS Intel: `BOT_appsumo-product-catalog-scraper.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/appsumo-product-catalog-scraper/macos_intel - Linux x86_64: `BOT_appsumo-product-catalog-scraper.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/appsumo-product-catalog-scraper/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=appsumo-product-catalog-scraper` Configuration document: `https://marketplace.rtila.com/api/bots/appsumo-product-catalog-scraper/config.json` - Encoding: plain - Digest: ee189862126d94ade71f79d44d01f59772432ba91442ec502bd1b5ed3fa70822 - Bytes: 11892 ```json { "name": "AppSumo Product Catalog Scraper", "settings": { "urls": [ "https://appsumo.com/search/?query=${encodeURIComponent(variables.search_query || '')}" ], "headless": false, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": false, "variables": [ { "name": "search_query", "value": "", "required": false, "is_list": false } ] }, "datasets": { "appsumo_listings": { "item_selector": "xpath=//div[@data-testid='deal-price-container']/ancestor::*[contains(@class,'bg-white') and contains(@class,'rounded')][1]", "properties": [ { "name": "product_name", "type": "text", "selector": "css=span.overflow-hidden.text-ellipsis.whitespace-nowrap.font-bold,css=a[href*='/products/']:not([href*='#'])", "transformations": [ { "type": "trim" } ] }, { "name": "product_url", "type": "attribute", "selector": "css=a[href*='/products/']:not([href*='#']),css=a[href*='/products/']", "attribute": "href", "transformations": [ { "type": "replace" }, { "type": "trim" } ] }, { "name": "product_slug", "type": "attribute", "selector": "css=a[href*='/products/']:not([href*='#']),css=a[href*='/products/']", "attribute": "href", "transformations": [ { "type": "regex", "pattern": "/products/([^/?#]+)", "default_value": "" } ] }, { "name": "product_image_url", "type": "attribute", "selector": "css=img.aspect-sku-card", "attribute": "src", "transformations": [ { "type": "replace" } ] }, { "name": "product_image_url_resized", "type": "attribute", "selector": "css=img.aspect-sku-card", "attribute": "src" }, { "name": "product_image_alt", "type": "attribute", "selector": "css=img.aspect-sku-card", "attribute": "alt" }, { "name": "is_appsumo_select", "type": "count", "selector": "css=img[alt='AppSumo Select']" }, { "name": "appsumo_select_badge_url", "type": "attribute", "selector": "css=img[alt='AppSumo Select']", "attribute": "src" }, { "name": "deal_badge_text", "type": "text", "selector": "css=div.absolute.bottom-0.left-0 span", "transformations": [ { "type": "trim" } ] }, { "name": "deal_badge_type", "type": "text", "selector": "css=div.absolute.bottom-0.left-0 span", "transformations": [ { "type": "trim" }, { "type": "replace" }, { "type": "replace" }, { "type": "replace" }, { "type": "replace" } ] }, { "name": "deal_days_remaining", "type": "text", "selector": "css=div.absolute.bottom-0.left-0 span", "transformations": [ { "type": "regex", "pattern": "([0-9]+)\\s+day", "default_value": "" }, { "type": "cast" } ] }, { "name": "category_name", "type": "text", "selector": "css=a.relative.z-1.underline", "transformations": [ { "type": "trim" } ] }, { "name": "category_url", "type": "attribute", "selector": "css=a.relative.z-1.underline", "attribute": "href" }, { "name": "category_slug", "type": "attribute", "selector": "css=a.relative.z-1.underline", "attribute": "href", "transformations": [ { "type": "regex", "pattern": "/([^/?#]+)/?$", "default_value": "" } ] }, { "name": "category_group_slug", "type": "attribute", "selector": "css=a.relative.z-1.underline", "attribute": "href", "transformations": [ { "type": "regex", "pattern": "appsumo\\.com/([^/?#]+)", "default_value": "" } ] }, { "name": "category_parent_slug", "type": "attribute", "selector": "css=a.relative.z-1.underline", "attribute": "href", "transformations": [ { "type": "regex", "pattern": "appsumo\\.com/[^/?#]+/([^/?#]+)", "default_value": "" } ] }, { "name": "categories", "type": "list", "selector": "css=a[href]:not([href*='/products/']):not([href^='#']):not([href^='javascript'])", "properties": [ { "name": "name", "type": "text", "selector": "css=self", "transformations": [ { "type": "trim" } ] }, { "name": "url", "type": "attribute", "selector": "css=self", "attribute": "href" }, { "name": "level", "type": "index", "selector": "" } ] }, { "name": "tagline", "type": "text", "selector": "css=div.line-clamp-3", "transformations": [ { "type": "trim" } ] }, { "name": "rating", "type": "attribute", "selector": "css=img[alt*='stars']", "attribute": "alt", "transformations": [ { "type": "regex", "pattern": "([0-9]+(?:\\.[0-9]+)?)", "default_value": "" }, { "type": "cast" } ] }, { "name": "rating_label", "type": "attribute", "selector": "css=img[alt*='stars']", "attribute": "alt", "transformations": [ { "type": "trim" } ] }, { "name": "review_count_text", "type": "text", "selector": "css=a[href*='#reviews']", "transformations": [ { "type": "regex", "pattern": "([0-9][0-9,]*\\s+reviews)", "default_value": "" }, { "type": "trim" } ] }, { "name": "review_count", "type": "text", "selector": "css=a[href*='#reviews']", "transformations": [ { "type": "regex", "pattern": "([0-9][0-9,]*)\\s+reviews", "default_value": "" }, { "type": "replace" }, { "type": "cast" } ] }, { "name": "review_url", "type": "attribute", "selector": "css=a[href*='#reviews']", "attribute": "href" }, { "name": "price_raw", "type": "text", "selector": "css=[data-testid='deal-price-container']", "transformations": [ { "type": "trim" } ] }, { "name": "price_current", "type": "text", "selector": "css=#deal-price", "transformations": [ { "type": "regex", "pattern": "([0-9][0-9,]*(?:\\.[0-9]{1,2})?)", "default_value": "" }, { "type": "replace" }, { "type": "cast" } ] }, { "name": "price_currency_symbol", "type": "text", "selector": "css=#deal-price", "transformations": [ { "type": "regex", "pattern": "([$£€])", "default_value": "" } ] }, { "name": "price_period", "type": "text", "selector": "css=#deal-price-suffix", "transformations": [ { "type": "replace" }, { "type": "trim" } ] }, { "name": "price_original", "type": "text", "selector": "css=#deal-price-original", "transformations": [ { "type": "regex", "pattern": "([0-9][0-9,]*(?:\\.[0-9]{1,2})?)", "default_value": "" }, { "type": "replace" }, { "type": "cast" } ] }, { "name": "plus_exclusive_label", "type": "text", "selector": "css=span.overflow-hidden.text-ellipsis.whitespace-nowrap.font-bold,css=a[href*='/products/']:not([href*='#'])", "transformations": [ { "type": "regex", "pattern": "[-–—]\\s*(Plus exclusive)\\s*$", "default_value": "" } ] }, { "name": "listing_position", "type": "index", "selector": "" }, { "name": "card_text_snapshot", "type": "text", "selector": "css=self", "transformations": [ { "type": "replace" }, { "type": "trim" } ] } ], "required_fields": [ "product_url", "product_name" ] } }, "commands": [ { "command": "press", "params": { "selector": "html", "key": "Escape" } }, { "command": "infinite_scroll", "params": { "do": [], "max_scrolls": 2, "scroll_delay": 2500 } }, { "command": "while", "params": { "condition": { "selector": "css=button:has-text('Show More Results'),button:has-text('Load More'),button:has-text('Show more results'),button:has-text('View More')", "type": "element_visible" }, "do": [ { "command": "click", "params": { "button": "left", "selector": "css=button:has-text('Show More Results'),button:has-text('Load More'),button:has-text('Show more results'),button:has-text('View More')" } }, { "command": "wait", "params": { "timeout": 3000 } } ] } }, { "command": "extract_data", "params": { "dataset": "appsumo_listings" } } ], "authors": [] } ``` ### FAQ **How do I scrape AppSumo to CSV?** Run the launcher, leave the search_query field empty to collect the full AppSumo catalog or type a keyword to collect a slice, and the bot expands the search results grid and extracts every deal card into the appsumo_listings dataset. Rows land in the launcher's local results store; export them from the results view - CSV is the common working format for this table, and JSON and XLSX exports are available too. No selector work is required from you. **What data can I collect from AppSumo listings?** Roughly thirty card-level fields per deal: product name, URL and slug, tagline, the full AppSumo category taxonomy (name, URL, three slug levels and a nested categories list), current price, original price, currency symbol, billing period, raw price text, rating, rating label, review count, review count text, review URL, deal badge text, normalised badge type, days remaining, the AppSumo Select flag and badge URL, the Plus exclusive label, three image fields, listing position and a full card text snapshot. **How do I scrape the whole AppSumo catalog?** Leave the search_query field empty at launch. The start URL is the AppSumo search page with an empty query, which returns the entire active catalog - 334-plus products at the time this configuration was written, and it grows as new deals launch. The bot scrolls the grid, clicks every load-more control until none remain, then extracts all cards in one pass, deduplicated by product URL. **How do I scrape only one AppSumo category or keyword?** Type a keyword into the search_query field at launch - something like seo or video hosting. The bot URL-encodes it into the AppSumo search query, so the run collects only matching results. One run covers one query; for several categories, run the launcher once per term and keep the exports separate, then join them later on product_url if you need one master table. **How do I track AppSumo deal prices and review counts?** The bot is a snapshotter, so tracking means re-running it on a cadence and diffing the results yourself. Schedule the launcher weekly with your OS scheduler (Task Scheduler, cron or launchd), export each run with a date, and join the snapshots on product_url - the dataset's deduplication key and the most stable identifier in the row. Price moves show up as changes in price_current or price_original; review velocity shows up as the delta in review_count between two dated runs. **Can I scrape AppSumo deal countdown timers?** You can scrape the countdown strip, at day granularity. deal_badge_text keeps the visible strip verbatim, deal_badge_type normalises it to deal_ending, price_increase or plus_access_ending (empty when the strip shows something else), and deal_days_remaining parses the integer day count out of the text. There is no exact expiry timestamp on the card, so none is invented - the row tells you which day to check by hand. **What is the AppSumo Select flag in the output?** is_appsumo_select counts the AppSumo Select badge image on the deal card, so it reads 0 for ordinary listings and 1 or more for editorially selected ones; appsumo_select_badge_url keeps the badge image source. The separate plus_exclusive_label field captures the Plus exclusive suffix on the product name for membership-gated deals. Together they let you segment the curated slice of the catalog from the open listings. **How long does a full AppSumo catalog scrape take?** Minutes, not hours. The bot never visits individual product pages - it expands the search grid with two capped scrolls and a load-more click loop that waits three seconds per batch, then extracts every card in a single pass. A full-catalog run of several hundred listings typically finishes in well under twenty minutes; a keyword-filtered search is faster. Compare that with detail-page crawlers, where run time scales with one page visit per listing. **Does the scraper collect AppSumo review text?** No - only aggregate signals, deliberately. The dataset captures rating, rating_label, review_count, review_count_text and the review_url link, never review bodies and never reviewer names. That keeps personal data out of the output entirely (this listing is flagged piiHandled false). If you need the reviews themselves, the sibling AppSumo Product Reviews Scraper collects them as a separate configuration with its own compliance bar. **Is scraping AppSumo allowed?** The data collected here is public commercial data - product names, prices, aggregate ratings - but automated collection may still conflict with AppSumo's terms of service, and that review is yours to do before each run. Keep request volume at what a patient human could sustain (the built-in waits already pace the run that way), date every snapshot, and do not republish derived data in ways AppSumo prohibits. This bot is not affiliated with or endorsed by AppSumo. **Why are some price or rating fields empty?** Empty is normal and expected in three cases: brand-new listings carry no reviews yet, so rating and review_count fall back to null through their cast defaults; not every card shows a struck-through original price or a countdown badge; and the Plus exclusive label only exists on membership-gated deals. The transformations fall back to empty values rather than failing the row, and the failure contract is loud - empty values or zero rows, never fabricated data. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## AppSumo Product Reviews Scraper Collect AppSumo review text, ratings, dates, reviewer flags and founder replies from every review page. - URL: https://marketplace.rtila.com/bots/appsumo-product-reviews-scraper - Category: Market Research - Tags: appsumo, product-reviews, review-analysis, voice-of-customer, market-research, software-deals - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-10-06) - Price: free ### Overview The AppSumo Product Reviews Scraper is an AppSumo reviews scraper: a standalone desktop bot that takes a list of AppSumo product URLs, normalizes each one to the review tab, walks every paginated review page through to the last, and extracts thirty-three fields per review into the `reviews` dataset. Where the sibling catalog bot reads deal cards, this one goes into the review stream itself - the full review body, the star rating, the displayed and ISO-normalised dates, the reviewer's username and verification badges, the AppSumo Plus membership flag, any founder reply, and the helpful and share counts - across many products in a single run. This listing is not affiliated with or endorsed by AppSumo. The selectors target AppSumo's current public review-card markup and will need updating when that markup changes - when it does, a run ends with empty fields or zero rows rather than invented ones. ## What does the AppSumo Product Reviews Scraper do? The run is built for scale across products and depth within each. You supply `product_urls` - a required list variable, one AppSumo product URL per entry. The configuration ships with a single example product URL as its default; replace it with your own list before a real run. The engine fans the list out into its URL queue, one worker task per product, so a dozen products become a dozen independent collection passes. Each pass is self-normalising: it accepts any AppSumo product URL shape - with or without the reviews path, with or without a trailing slash or query string - and rewrites it to the canonical review tab at page one before collecting. Within a product the bot paginates. It first reads AppSumo's own pagination controls to discover the last page number, then loops from page one up to that last page - bounded by the optional `max_pages_per_product` safety cap - extracting the `reviews` dataset on every page. Because review pages can overlap and re-render, the dataset deduplicates on the combination of `review_id`, `review_text` and `reviewer_username`, and declares `review_id` and `review_text` as required fields, so a card that never rendered its body is logged rather than silently trusted. Because everything runs on your own machine, your research footprint stays yours: which products you audit, which competitors you benchmark, which complaint themes you mine never passes through a third-party service's dashboard or logs. ## Who uses an AppSumo reviews scraper? The voice-of-customer researcher asks "what do AppSumo buyers actually praise and complain about across this category?" and needs the full `review_text` plus `review_rating` for every review, not a sample. The SaaS founder launching on AppSumo asks "which recurring defects show up in one-star reviews of comparable products, and did those founders respond?" and needs `review_rating`, `review_text` and `has_founder_reply` side by side. The competitive analyst asks "how responsive is the competition to negative reviews, and how fast do they reply?" and needs `founder_reply_date_iso` measured against `review_date_iso`. The agency building deal-market datasets for clients asks "can I hand over a dated, deduplicated, structured review file per product?" and needs exactly the `reviews` export. Each of them needs a defensible artifact they produced themselves rather than a screenshot. ## How does an AppSumo review run work, step by step? Every command in the run links to its reference page. The pass opens on the queued product URL. First, [execute_script](https://learn.rtila.com/reference/commands/execute_script) reads the live location into `source_url`, and a second execute_script parses the product slug out of the path into `product_slug`. Third, [goto](https://learn.rtila.com/reference/commands/goto) navigates to the canonical AppSumo review tab for that slug at page one. Fourth, [wait_for_selector](https://learn.rtila.com/reference/commands/wait_for_selector) blocks - up to 45 seconds - until either the page heading or the first review card renders, so extraction never races the page. Fifth, a third execute_script scans the pagination links - any whose accessible label reads "Go to page" followed by a number - and returns the highest into `last_page`. Sixth, [set_variable](https://learn.rtila.com/reference/commands/set_variable) seeds `page` at one and [extract_data](https://learn.rtila.com/reference/commands/extract_data) pulls page one's reviews. Finally a [while](https://learn.rtila.com/reference/commands/while) loop continues while the current `page` is below both `last_page` and the `max_pages_per_product` cap: each iteration increments `page`, re-navigates to that numbered review page, waits for a review card, and extracts again. This release sets no fixed export filename, so rows land in the launcher's local results store and you export them from the results view; nothing is uploaded anywhere except the configuration fetch and download telemetry listed on this page. ## What does the output look like? The table mirrors the published sample row (fourteen of the thirty-three fields - the dataset carries the rest). Sample values are illustrative placeholders that show the shape of a real row: | product_slug | review_id | review_title | review_text | review_rating | review_date_iso | reviewer_username | is_verified_buyer | is_plus_member | has_founder_reply | founder_reply_text | helpful_count | scraped_page_number | review_position_on_page | |---|---|---|---|---|---|---|---|---|---|---|---|---|---| | flowdesk-pro | 184207 | Exactly what my calendar workflow needed | Set it up in an afternoon and it replaced three manual steps. | 5 | 2026-03-04 | dana.k | 1 | 0 | 1 | Thank you Dana - glad it saved you time! | 12 | 1 | 1 | **product_slug** and **source_url** tie every row back to its AppSumo product - the slug parsed from the path and the original queued URL captured before normalisation. **scraped_page_url** is the exact review page the row was read from, **scraped_page_number** is the pagination number parsed out of it (defaulting to 1), and **review_position_on_page** is the 1-based position of the card within that page. **review_id** is the numeric identifier extracted from the review's own permalink and a required field; **review_url** and **review_slug** keep the full permalink and its slug. **review_title** is the headline and **review_text** is the full body, whitespace-collapsed and trimmed, also a required field. **review_text_length** is the character count of that body, computed by a small script transform - handy for filtering thin or empty reviews. **review_rating_raw** keeps AppSumo's star accessibility label verbatim (the text ending in " stars") and **review_rating** is the numeric value cast to a float with a null fallback. **review_date** is the displayed date string, **review_date_iso** is that date normalised to yyyy-mm-dd by a parser with a month-name fallback (empty when unparseable), and **review_meta_row_text** is the whole collapsed meta line for auditing. **reviewer_username**, **reviewer_profile_url** and **reviewer_avatar_url** identify who wrote the review - personal data, handled accordingly. **is_verified_buyer** and **is_plus_member** are count fields (0 when absent) for the verified-purchase check icon and the AppSumo Plus membership badge; **reviewer_verified_label** captures the verified wording, **reviewer_member_since** the "Member since" month and year, and **reviewer_badges_text** the collapsed badge row. **has_founder_reply** is a count flag for a creator response; when one is present, **founder_reply_label**, **founder_username**, **founder_avatar_url**, **founder_reply_date** and **founder_reply_date_iso** describe it and **founder_reply_text** carries the response body. **helpful_count** and **review_share_count** are the engagement numbers parsed from the card, both cast to integers defaulting to 0. ## Can an AppSumo review scraper capture founder replies? Yes - and it is one of the more useful columns here. A creator response is captured as a flag plus its full detail: `has_founder_reply` tells you a reply exists, and `founder_username`, `founder_reply_text`, `founder_reply_date` and `founder_reply_date_iso` tell you who replied, what they said and when. The analytical payoff is latency: subtract `review_date_iso` from `founder_reply_date_iso` and you have a reply-time per review, which becomes a responsiveness benchmark when you compare how quickly AppSumo vendors answer their one-star reviews against their five-star ones - directly useful if you are planning your own launch's review strategy. What the bot does not do is interpret any of it - there is no sentiment score, no topic tag, no translation. It hands you the raw text and the dates; the analysis is your step. ## How does reviewer identity and verification data work? Each review row carries who wrote it and how AppSumo labels them. `reviewer_username`, `reviewer_profile_url` and `reviewer_avatar_url` are the identity fields; `is_verified_buyer` counts the verified-purchase check icon and `is_plus_member` counts the AppSumo Plus badge, so both read 0 when absent and 1 or more when present. `reviewer_verified_label` keeps the verified wording, `reviewer_member_since` parses the "Member since" month and year, and `reviewer_badges_text` collapses the whole badge row for auditing. Because these fields identify real people, this listing is flagged `piiHandled` true - treat them as personal data, with the lawful-basis and takedown duties that follow. ## How does it compare to hosted AppSumo review services? Reading reviews by hand produces a handful of rows an hour, no consistency, and nothing you can diff next month. Hosted robot and actor marketplaces do list AppSumo review collectors, but the structure is the opposite of this bot: they charge per result or per month, the review text and reviewer identities land in their cloud under their retention terms, and your audit history - which products, which competitors - lives on their servers. Here the marginal cost of a deeper back-catalog is your own time and bandwidth, the personal data stays on your disk under your control, and the workflow configuration is yours to keep, modify and re-run. The trade is real: no dashboards, no built-in sentiment analysis, no scheduling, no cloud-side retry. You get an artifact and a workflow you own outright. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the downloaded .exe directly - there is no installation step; SmartScreen may ask for confirmation on first launch, so verify the published checksum, then choose More info and Run anyway. On **macOS**, two launchers are published - pick the one matching your machine's chip (Apple menu, About This Mac: Apple Silicon or Intel) - then make it executable with `chmod +x`, clear the Gatekeeper quarantine with `xattr -d com.apple.quarantine /path/to/BOT_appsumo-product-reviews-scraper.sh` and launch. On **Linux**, `chmod +x ./BOT_appsumo-product-reviews-scraper.sh` then `./BOT_appsumo-product-reviews-scraper.sh`. At first run the launcher asks for the `product_urls` list - one AppSumo product URL per line, replacing the shipped example - and the optional `max_pages_per_product` cap, then starts the run. Keep `product_urls` a list: a single string collapses the fan-out. To repeat an audit on a cadence, schedule the launcher with your OS scheduler (Task Scheduler, cron or launchd). ## What do I check when a run returns zero rows? Work in order. Zero rows: did the review page render in an ordinary browser, or did a consent wall, region lock or challenge greet the fresh session - the run waits up to 45 seconds for a heading or a review card, and neither appearing means the page never loaded. Only page one collected for every product: the pagination-label scan most likely stopped at 1, so `last_page` read 1 and the while loop never advanced - check one product by hand against its real page count on appsumo.com. A single product collected when you queued many: `product_urls` was passed as a string instead of an array, collapsing the fan-out. Rows present but a column empty across every row: that field's selector most likely changed in an AppSumo redesign; check one review card by hand. Empty rating or ISO date on scattered rows: completely normal - the casts and date parser fall back to null or empty rather than failing the row. The contract is that failure is loud - empty values and zero rows, never fabricated data. ## What are the bot's limits? It reads AppSumo review pages only: no product specifications, prices, plan tiers or vendor data (the sibling AppSumo Product Catalog Scraper covers the deal-card side), and no photos beyond reviewer and founder avatars. It ships without sentiment scoring, topic tagging or translation - the review text is raw and the analysis is yours. It has no proxy rotation or CAPTCHA solving enabled, so if a challenge stops the run, the run stops; nothing bypasses it. Pagination is bounded by the labels it recognises and by `max_pages_per_product`, so an AppSumo redesign of the pagination controls can truncate depth silently - which is why the page-count check above matters. It is a snapshotter, not a monitor - nothing re-runs, diffs or alerts on its own. Deduplication applies within a run on the review id, text and username triple; merging across runs is your step. ## What is responsible AppSumo review collection? This bot crosses into personal data by design: review bodies, usernames, profile links and avatars identify people, so establish a lawful basis under GDPR, UK GDPR or your local equivalent before storing or analysing the output, and honour takedown and opt-out requests immediately and permanently. Never republish reviewer identities, and never use the output for spam or for profiling that harms individuals. Review AppSumo's terms of service with open eyes - automated collection may conflict with them, and the decision to run is yours to own. Keep request volume at what a patient human could sustain; the per-page waits and the `max_pages_per_product` cap already pace the run that way. Date every snapshot, and if you republish derived analysis, do so in aggregate and in ways AppSumo permits. This bot is an independent tool, not affiliated with or endorsed by AppSumo. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | product_urls | text | yes | - | Declared in project settings. | | max_pages_per_product | text | no | - | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **reviews** (item selector: `css=[data-testid='review-card-wrapper']`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | product_slug | text | no | variable=product_slug | | source_url | text | no | variable=source_url | | scraped_page_url | page_url | no | | | scraped_page_number | page_url | no | | | review_position_on_page | index | no | | | review_id | attribute | no | css=a[target='_blank'][href*='/reviews/'] | | review_url | attribute | no | css=a[target='_blank'][href*='/reviews/'] | | review_slug | attribute | no | css=a[target='_blank'][href*='/reviews/'] | | review_title | text | no | css=div.space-y-1 p.font-header | | review_text | text | no | css=div.text-midnight [data-testid='toggle-text'] | | review_text_length | text | no | css=div.text-midnight [data-testid='toggle-text'] | | review_rating_raw | attribute | no | css=[alt$=' stars'] | | review_rating | attribute | no | css=[alt$=' stars'] | | review_date | text | no | css=div.flex-wrap.text-grace span | | review_date_iso | text | no | css=div.flex-wrap.text-grace span | | review_meta_row_text | text | no | css=div.flex-wrap.text-grace | | reviewer_username | text | no | css=[data-testid='discussion-user-info'] a.font-bold | | reviewer_profile_url | attribute | no | css=[data-testid='discussion-user-info'] a.font-bold | | reviewer_avatar_url | attribute | no | css=[data-testid='discussion-user-info'] img | | is_plus_member | count | no | css=[data-testid='plus-badge'] | | is_verified_buyer | count | no | css=[data-testid='discussion-user-info'] svg[data-icon='circle-check'] | | reviewer_verified_label | text | no | css=[data-testid='discussion-user-info'] | | reviewer_member_since | text | no | css=[data-testid='discussion-user-info'] | | reviewer_badges_text | text | no | css=[data-testid='discussion-user-info'] | | has_founder_reply | count | no | css=[data-testid='discussion-creator-reply'] | | founder_reply_label | text | no | css=[data-testid='creator-reply-label'] | | founder_username | text | no | css=[data-testid='creator-username'] | | founder_avatar_url | attribute | no | css=[data-testid='creator-avatar'] | | founder_reply_date | text | no | css=[data-testid='creator-posted-date'] | | founder_reply_date_iso | text | no | css=[data-testid='creator-posted-date'] | | founder_reply_text | text | no | css=[data-testid='discussion-creator-reply'] [data-testid='toggle-text'] | | helpful_count | text | no | css=[data-testid='discussion-helpful-buttons'] | | review_share_count | text | no | css=[data-testid='review-card-wrapper'] | ### Downloads - Windows x86_64: `BOT_appsumo-product-reviews-scraper.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/appsumo-product-reviews-scraper/windows - macOS Apple Silicon: `BOT_appsumo-product-reviews-scraper.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/appsumo-product-reviews-scraper/macos - macOS Intel: `BOT_appsumo-product-reviews-scraper.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/appsumo-product-reviews-scraper/macos_intel - Linux x86_64: `BOT_appsumo-product-reviews-scraper.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/appsumo-product-reviews-scraper/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=appsumo-product-reviews-scraper` Configuration document: `https://marketplace.rtila.com/api/bots/appsumo-product-reviews-scraper/config.json` - Encoding: plain - Digest: 460d4665b5fbcafc90102dcdddbedf3414baff951c8be9dccd4556ce1e13a1b8 - Bytes: 13664 ```json { "name": "AppSumo Product Reviews Scraper", "settings": { "urls": [ "${product_urls}" ], "headless": false, "navigation_timeout": 60000, "max_concurrent_workers": 1, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": false, "variables": [ { "name": "product_urls", "required": true, "is_list": false }, { "name": "max_pages_per_product", "required": false, "is_list": false } ] }, "datasets": { "reviews": { "item_selector": "css=[data-testid='review-card-wrapper']", "properties": [ { "name": "product_slug", "type": "text", "selector": "variable=product_slug", "transformations": [ { "type": "trim" } ] }, { "name": "source_url", "type": "text", "selector": "variable=source_url", "transformations": [ { "type": "trim" } ] }, { "name": "scraped_page_url", "type": "page_url", "selector": "" }, { "name": "scraped_page_number", "type": "page_url", "selector": "", "transformations": [ { "type": "regex", "pattern": "[?&]page=(\\d+)", "default_value": "1" }, { "type": "cast" } ] }, { "name": "review_position_on_page", "type": "index", "selector": "" }, { "name": "review_id", "type": "attribute", "selector": "css=a[target='_blank'][href*='/reviews/']", "attribute": "href", "transformations": [ { "type": "regex", "pattern": "-?(\\d+)\\/?$", "default_value": "" }, { "type": "trim" } ] }, { "name": "review_url", "type": "attribute", "selector": "css=a[target='_blank'][href*='/reviews/']", "attribute": "href", "transformations": [ { "type": "trim" } ] }, { "name": "review_slug", "type": "attribute", "selector": "css=a[target='_blank'][href*='/reviews/']", "attribute": "href", "transformations": [ { "type": "regex", "pattern": "\\/reviews\\/([^\\/?#]+)\\/?$", "default_value": "" } ] }, { "name": "review_title", "type": "text", "selector": "css=div.space-y-1 p.font-header", "transformations": [ { "type": "replace" }, { "type": "trim" } ] }, { "name": "review_text", "type": "text", "selector": "css=div.text-midnight [data-testid='toggle-text']", "transformations": [ { "type": "replace" }, { "type": "trim" } ] }, { "name": "review_text_length", "type": "text", "selector": "css=div.text-midnight [data-testid='toggle-text']", "transformations": [ { "type": "script", "code": "return String((value || '').length);", "default_value": "0" }, { "type": "cast" } ] }, { "name": "review_rating_raw", "type": "attribute", "selector": "css=[alt$=' stars']", "attribute": "alt", "transformations": [ { "type": "trim" } ] }, { "name": "review_rating", "type": "attribute", "selector": "css=[alt$=' stars']", "attribute": "alt", "transformations": [ { "type": "regex", "pattern": "([\\d.]+)", "default_value": "" }, { "type": "cast" } ] }, { "name": "review_date", "type": "text", "selector": "css=div.flex-wrap.text-grace span", "transformations": [ { "type": "trim" } ] }, { "name": "review_date_iso", "type": "text", "selector": "css=div.flex-wrap.text-grace span", "transformations": [ { "type": "script", "code": "var t=(value||'').trim();if(!t)return '';var d=new Date(t);if(!isNaN(d.getTime()))return d.toISOString().slice(0,10);var m=t.match(/^([A-Za-z]{3,9})\\.?\\s+(\\d{1,2}),\\s*(\\d{4})$/);if(!m)return '';var mo={jan:1,feb:2,mar:3,apr:4,may:5,jun:6,jul:7,aug:8,sep:9,oct:10,nov:11,dec:12}[m[1].slice(0,3).toLowerCase()];if(!mo)return '';return m[3]+'-'+('0'+mo).slice(-2)+'-'+('0'+m[2]).slice(-2);", "default_value": "" } ] }, { "name": "review_meta_row_text", "type": "text", "selector": "css=div.flex-wrap.text-grace", "transformations": [ { "type": "replace" }, { "type": "trim" } ] }, { "name": "reviewer_username", "type": "text", "selector": "css=[data-testid='discussion-user-info'] a.font-bold", "transformations": [ { "type": "trim" } ] }, { "name": "reviewer_profile_url", "type": "attribute", "selector": "css=[data-testid='discussion-user-info'] a.font-bold", "attribute": "href", "transformations": [ { "type": "trim" } ] }, { "name": "reviewer_avatar_url", "type": "attribute", "selector": "css=[data-testid='discussion-user-info'] img", "attribute": "src", "transformations": [ { "type": "trim" } ] }, { "name": "is_plus_member", "type": "count", "selector": "css=[data-testid='plus-badge']" }, { "name": "is_verified_buyer", "type": "count", "selector": "css=[data-testid='discussion-user-info'] svg[data-icon='circle-check']" }, { "name": "reviewer_verified_label", "type": "text", "selector": "css=[data-testid='discussion-user-info']", "transformations": [ { "type": "regex", "pattern": "(Verified(?:\\s+purchaser)?)", "default_value": "" }, { "type": "trim" } ] }, { "name": "reviewer_member_since", "type": "text", "selector": "css=[data-testid='discussion-user-info']", "transformations": [ { "type": "regex", "pattern": "Member since:\\s*([A-Za-z]{3,9}\\.?\\s+\\d{4})", "default_value": "" }, { "type": "trim" } ] }, { "name": "reviewer_badges_text", "type": "text", "selector": "css=[data-testid='discussion-user-info']", "transformations": [ { "type": "replace" }, { "type": "trim" } ] }, { "name": "has_founder_reply", "type": "count", "selector": "css=[data-testid='discussion-creator-reply']" }, { "name": "founder_reply_label", "type": "text", "selector": "css=[data-testid='creator-reply-label']", "transformations": [ { "type": "trim" } ] }, { "name": "founder_username", "type": "text", "selector": "css=[data-testid='creator-username']", "transformations": [ { "type": "trim" } ] }, { "name": "founder_avatar_url", "type": "attribute", "selector": "css=[data-testid='creator-avatar']", "attribute": "src", "transformations": [ { "type": "trim" } ] }, { "name": "founder_reply_date", "type": "text", "selector": "css=[data-testid='creator-posted-date']", "transformations": [ { "type": "trim" } ] }, { "name": "founder_reply_date_iso", "type": "text", "selector": "css=[data-testid='creator-posted-date']", "transformations": [ { "type": "script", "code": "var t=(value||'').trim();if(!t)return '';var d=new Date(t);if(!isNaN(d.getTime()))return d.toISOString().slice(0,10);var m=t.match(/^([A-Za-z]{3,9})\\.?\\s+(\\d{1,2}),\\s*(\\d{4})$/);if(!m)return '';var mo={jan:1,feb:2,mar:3,apr:4,may:5,jun:6,jul:7,aug:8,sep:9,oct:10,nov:11,dec:12}[m[1].slice(0,3).toLowerCase()];if(!mo)return '';return m[3]+'-'+('0'+mo).slice(-2)+'-'+('0'+m[2]).slice(-2);", "default_value": "" } ] }, { "name": "founder_reply_text", "type": "text", "selector": "css=[data-testid='discussion-creator-reply'] [data-testid='toggle-text']", "transformations": [ { "type": "replace" }, { "type": "trim" } ] }, { "name": "helpful_count", "type": "text", "selector": "css=[data-testid='discussion-helpful-buttons']", "transformations": [ { "type": "regex", "pattern": "(\\d+)", "default_value": "" }, { "type": "cast" } ] }, { "name": "review_share_count", "type": "text", "selector": "css=[data-testid='review-card-wrapper']", "transformations": [ { "type": "regex", "pattern": "Share(\\d+)", "default_value": "" }, { "type": "cast" } ] } ], "required_fields": [ "review_id", "review_text" ] } }, "commands": [ { "command": "execute_script", "params": { "script": "return location.href;" }, "output_variable": "source_url" }, { "command": "execute_script", "params": { "script": "return (location.pathname.split('/products/')[1] || '').split('/')[0];" }, "output_variable": "product_slug" }, { "command": "goto", "params": { "url": "https://appsumo.com/products/${product_slug}/reviews/?ref=pdp&page=1" } }, { "command": "wait_for_selector", "params": { "selector": "css=h1, [data-testid='review-card-wrapper']", "timeout": 45000 } }, { "command": "execute_script", "params": { "script": "var as=document.getElementsByTagName('a');var last=1;for(var i=0;ilast)last=n;}}}return last;" }, "output_variable": "last_page" }, { "command": "set_variable", "params": { "name": "page", "value": 1 } }, { "command": "extract_data", "params": { "dataset": "reviews" } }, { "command": "while", "params": { "condition": { "type": "script", "expression": "((variables.page - 0) || 0) < ((variables.last_page - 0) || 0) && ((variables.page - 0) || 0) < ((variables.max_pages_per_product - 0) || 999)" }, "do": [ { "command": "set_variable", "params": { "name": "page", "value": "${(variables.page - 0) + 1}" } }, { "command": "goto", "params": { "url": "https://appsumo.com/products/${product_slug}/reviews/?ref=pdp&page=${page}" } }, { "command": "wait_for_selector", "params": { "selector": "css=[data-testid='review-card-wrapper']", "timeout": 45000 } }, { "command": "extract_data", "params": { "dataset": "reviews" } } ] } } ], "authors": [] } ``` ### FAQ **How do I scrape AppSumo product reviews to CSV?** Run the launcher, paste one or more AppSumo product URLs into the required product_urls list (one URL per line), optionally set a per-product page cap, and the bot fans the list out, normalizes each URL to the AppSumo review tab, paginates to the last page and extracts every review card into the reviews dataset. Rows land in the launcher's local results store; export them from the results view - CSV is the common working format, and JSON and XLSX exports are available too. No selector work is required from you. **What review data does the AppSumo scraper collect?** Thirty-three fields per review: the product slug and source URL, the scraped page URL and number, the review position, review id, url and slug, the title, the full review text and its character length, the raw star label and the numeric rating, the displayed date and an ISO-normalised date, the meta row text, the reviewer username, profile url and avatar, verified-buyer and AppSumo Plus member flags, the verified label and member-since text, the badge row, whether a founder replied plus the reply label, username, avatar, date, ISO date and text, and the helpful and share counts. **How does the bot handle multiple AppSumo products at once?** product_urls is a required list variable, and the engine fans it out into its URL queue with one worker task per entry, so a dozen products become a dozen independent collection passes in a single run. It accepts any AppSumo product URL shape - with or without the reviews path, with or without a trailing slash or query string - and normalizes each to the canonical review tab. One caution baked into the configuration: the value must stay a JSON array. Passed as a single string, the fan-out breaks and the run collapses to one worker-replicated URL. **How does AppSumo review pagination work?** For each product the bot first reads the page's own pagination controls - it scans links whose accessible label reads Go to page followed by a number and takes the highest - storing that in last_page. It then loops from page one up to last_page, re-navigating to each numbered review page URL, waiting for a review card to render, and extracting. The loop stops at the last detected page or at the max_pages_per_product cap, whichever comes first. **Can I limit how many review pages are crawled per product?** Yes. max_pages_per_product is an optional safety cap on review pages crawled per product. It defaults to 999, which is effectively unlimited, so a full review history is collected. Set it to 2 for a fast smoke test, or to any number to bound run time and request volume on products with very deep review back-catalogs. **Does the scraper collect reviewer names and profiles?** It does, deliberately, and that raises the compliance bar. The dataset captures reviewer_username, reviewer_profile_url and reviewer_avatar_url, plus verified-buyer and AppSumo Plus member flags and a member-since value. These identify people, so this listing is flagged piiHandled true: establish a lawful basis before storing or analysing them, honour takedowns, and never republish reviewer identities or use them for spam. **Can it capture AppSumo founder replies to reviews?** Yes. has_founder_reply is a count flag for a creator response on the review card, and when one is present the run also captures founder_reply_label, founder_username, founder_avatar_url, founder_reply_date and its ISO-normalised form, and founder_reply_text. Diffing founder_reply_date_iso against review_date_iso gives you a reply-latency measure - a useful responsiveness signal when benchmarking how a vendor handles its negative AppSumo reviews. **How are AppSumo review dates normalised?** Two ways per date. review_date keeps the displayed string verbatim, and review_date_iso runs it through a small script transform that first tries a native date parse and then falls back to a month-name pattern (AppSumo shows dates like March 4, 2026), emitting a clean yyyy-mm-dd value. The same pair exists for founder replies. When a date cannot be parsed the ISO field is left empty rather than guessed, so an empty review_date_iso is an honest unknown, not a failure. **How does deduplication work across review pages?** AppSumo review pages can overlap and re-render as you paginate, so the reviews dataset deduplicates on the combination of review_id, review_text and reviewer_username. That triple is stable for a given review and catches the same card appearing on adjacent pages. review_id and review_text are also declared required fields, so a card that never rendered its body is logged rather than silently trusted. **Is scraping AppSumo reviews allowed?** Automated collection may conflict with AppSumo's terms of service, and review text plus reviewer usernames are personal data in most jurisdictions. You are responsible for reviewing AppSumo's terms and establishing a lawful basis under GDPR, UK GDPR or your local equivalent before running the bot, storing the output, or analysing named individuals. Keep request volume modest and let the acceptable-use policy govern every run. This bot is not affiliated with or endorsed by AppSumo. **Why are some rating or date fields empty?** Empty is normal and expected in a few cases: a review with no star label leaves review_rating null through its cast default; an unparseable or absent date leaves review_date_iso empty; helpful_count and review_share_count fall back to 0 when the card shows none; the AppSumo Plus flag reads 0 for non-members; and the founder-reply fields are empty on every review that has no creator response. The transforms fall back to empty or zero rather than failing the row, and the failure contract is loud - empty values or zero rows, never fabricated data. **How long does a full AppSumo review scrape take?** It scales with products times pages. Each product costs one page visit per review page, paced by human-like navigation and a wait for the review card to render before each extraction. A single product with a few hundred reviews across a handful of pages finishes in minutes; a queue of many products with deep review histories can run toward an hour. Lower max_pages_per_product or queue fewer products to keep runs short. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## Google Trends Keywords Research Automate a Google Trends search and download the related-queries CSV to your desktop, one keyword per run. - URL: https://marketplace.rtila.com/bots/google-trends-keywords-research - Category: SEO - Tags: google-trends, keyword-research, seo, trends-data, csv-export, search-trends - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-09-18) - Price: free ### Overview Google Trends Keywords Research is a google trends scraper: a standalone desktop bot that types one keyword into the public Trends explore page, waits for the related queries widget, and captures the CSV export the page generates, written to `related_query.csv`. ## What does Google Trends Keywords Research do? You give it a keyword - the configuration ships with a required `keyword` field prefilled with the example default "ai adoption by industry" - and it does the rest without another click. The launcher loads the Trends explore start URL pinned in its project settings, types your keyword into the explore search input, presses Enter, waits for the related queries export control to appear on the results page, triggers it, and saves the file the platform produces. There is no dataset extraction step and no reformatting: the artifact is the platform's own CSV, byte-for-byte as generated, landing on your disk as `related_query.csv`. Because everything runs on your desktop, your research interests - which seeds you are expanding, which categories you are probing, which clients you are building keyword backlogs for - never pass through anyone else's servers. One honesty note up front: unlike platform-neutral listings in this directory, this bot is platform-specific by design. Its selectors are pinned to current Google Trends markup, which is why it carries experimental stability: pinned selectors are the normal maintenance cost of automating one specific page. ## Who uses a google trends scraper? The SEO freelancer asks "what long-tails cluster around this seed right now?" and needs a rising-queries list per client niche, dated and defensible. The content marketer asks "is this topic heating up before I commission ten articles?" and needs the rising percentages as a directional signal, not a guess. The market researcher asks "what vocabulary does the audience actually use for this category?" and needs the related query text itself - the words people search, not the words the brief assumed. All three want an artifact they produced themselves: a CSV with the keyword and the collection date attached, sitting in their own folder, rather than a screen inside someone else's subscription. ## How does an seo keyword scraper run work, step by step? Every command in the run is linked to its reference page. First, the launcher navigates to the start URL from the project settings - the Trends explore page with an en-US locale. Then [fill](https://learn.rtila.com/reference/commands/fill) types the value of `${keyword}` into the explore search input, identified by a `data-explore-url` attribute selector, clearing any previous content first. [press](https://learn.rtila.com/reference/commands/press) sends Enter to the same input, submitting the search. [wait_for_selector](https://learn.rtila.com/reference/commands/wait_for_selector) then blocks until the export control of the related queries widget - the button carrying `widget-name="RELATED_QUERIES"` - exists on the page, and [wait_for_download](https://learn.rtila.com/reference/commands/wait_for_download) triggers that control and captures the generated file to `related_query.csv`. That is the whole run: four commands, one required input, one output file. The `keyword` variable is required, so a run without it fails before any navigation rather than silently researching the wrong thing. ## What does the related queries export look like? The bot produces no derived dataset of its own, so the columns below are the platform's export format, shown as an illustration of a rising-queries file: | Query | Rising | |---|---| | ai adoption statistics | +1,250% | | generative ai in manufacturing | Breakout | | ai governance checklist | +350% | **Query** is the related search term exactly as the platform lists it. This is the column you actually research with: the vocabulary your audience uses around your seed. **Rising** (or a relative top index, depending on the widget state) is the platform's own magnitude signal. Rising values are growth percentages against the prior period; "Breakout" means growth too large to measure. Neither is an absolute search volume, and the bot never invents one. The file is written verbatim - no renamed columns, no added fields, no parsed numbers. If you need a numeric column for sorting, split the percentage in your spreadsheet so the transformation is visible and yours. ## Can it serve as a keyword research automation tool? Yes, within its stated scope: it automates the tedious loop of navigating, typing, waiting, and exporting for one seed at a time, and it produces a consistently named, dated artifact you can diff across weeks. What it will not do is replace judgment. Rising percentages are relative, Breakout is unmeasurable, and none of it is volume. The working pattern is a pipeline: run the launcher per seed, store each `related_query.csv` under a filename carrying the keyword and the date, merge the batch in a spreadsheet, and prune. If you need saved histories, alerts on new risers, or grid-style tracking, that is a product category, not this bot - the comparison below sets out the trade. ## Is it a complete trends data download workflow? For the related queries widget, yes - and that is deliberately the only file it produces. It does not export the interest-over-time chart, related topics, or regional interest breakdowns, and it does not loop over regions or languages: the start URL carries a fixed en-US locale. One run, one seed, one widget, one CSV. Each of those boundaries keeps the run predictable; each is also a documented limit rather than a hidden one. ## How does it compare to other ways to get trend keywords? Copying rows by hand works for one seed and collapses at twenty, and it produces inconsistent files nobody trusts a month later. This bot produces the same deterministic file on your own machine for nothing per run, and the workflow configuration is yours to keep. Against subscription research platforms, the difference is structural: those keep your history in their cloud, estimate volumes with proprietary models, and bill monthly. Here the data lands on your disk and the marginal cost of another seed is your own bandwidth. Against metered API services, the same logic holds - per-call pricing versus a launcher you already own. The trade is real: no dashboards, no stored history, no built-in scheduling. You own the artifact and the workflow outright, and you keep your own records. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your platform from this page. Each download is a single raw file - there is no archive and no extraction step on any platform - and the filename carries the configuration id, so do not rename it. On **Windows**, run the `.exe` directly; SmartScreen may ask for confirmation on first launch, and "More info" then "Run anyway" proceeds once you have verified the published checksum. On **macOS**, two launchers are published - pick the one matching your machine's chip (Apple menu, About This Mac: Apple Silicon or Intel), then make it executable with `chmod +x ./BOT_google-trends-keywords-research.sh`, clear the Gatekeeper quarantine with `xattr -d com.apple.quarantine /path/to/BOT_google-trends-keywords-research.sh`, and launch. On **Linux**, the launcher is the same single shell script: `chmod +x ./BOT_google-trends-keywords-research.sh` then run it. The first run presents the required `keyword` field with its example default. For recurring research, schedule the launcher with Task Scheduler, cron, or launchd - the bot has no internal scheduler. ## What do I check when a run downloads no CSV? In order. First: is a consent or cookie interstitial sitting in front of the explore input? The configuration ships no dismissal step, so the fill times out behind it - this is the most common first-run failure, especially from EU addresses. Second: did the platform present a verification challenge or sign-in wall? No solver is configured, so the run stops honestly instead of fighting it. Third: has the page been redesigned? Both pinned selectors - the `data-explore-url` input and the RELATED_QUERIES export button - must match live markup, or the wait step times out. Fourth: did the keyword produce an empty or absent related queries widget? The export control never appears and the wait expires. Fifth: ordinary network and page-load problems. The contract is that failure is loud - a timeout and a missing file, never a stale or fabricated spreadsheet. ## What are the limits of the bot? One keyword per run; no region or language loop; related queries only, never interest-over-time, topics, or geography; relative signals, never absolute volumes; no scheduling, alerting, or history - it is a snapshotter. It reads only what a logged-out visitor sees and holds no credential fields. It carries no CAPTCHA solver, so a challenge ends the run. And its selectors are pinned: when the platform changes markup, the bot needs a configuration edit, which is exactly why its stability is marked experimental. ## What is responsible trends research? Automated interaction with the platform may conflict with its terms of service; review them with open eyes before your first run and own that decision. Keep request rates at what a patient human could sustain - a few runs per session, not a loop. The data collected is aggregate and contains no personal data, but the output still carries obligations: do not use rising-query lists to manufacture engagement or spam content at scale, and cite the collection date whenever you present the numbers, because trend signals drift weekly. The acceptable-use policy applies in full before your first run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | keyword | text | yes | ai adoption by industry | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output Export formats: csv. ### Downloads - Windows x86_64: `BOT_google-trends-keywords-research.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/google-trends-keywords-research/windows - macOS Apple Silicon: `BOT_google-trends-keywords-research.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/google-trends-keywords-research/macos - macOS Intel: `BOT_google-trends-keywords-research.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/google-trends-keywords-research/macos_intel - Linux x86_64: `BOT_google-trends-keywords-research.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/google-trends-keywords-research/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=google-trends-keywords-research` Configuration document: `https://marketplace.rtila.com/api/bots/google-trends-keywords-research/config.json` - Encoding: plain - Digest: 8f0746cdce7fcc682fb0523cad572ee5dc6ea52932667f4b77f71749c68ec688 - Bytes: 1432 ```json { "name": "Google Trends Keywords Research", "settings": { "urls": [ "https://trends.google.com/trends?hl=en-US&geo=" ], "headless": false, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": false, "variables": [ { "name": "keyword", "value": "ai adoption by industry", "required": true, "is_list": false } ] }, "datasets": {}, "commands": [ { "command": "fill", "params": { "clear_before_fill": true, "selector": "css=[data-explore-url^=\"https://trends.google.com/trends/explore\"]", "value": "${keyword}" } }, { "command": "press", "params": { "key": "Enter", "selector": "css=[data-explore-url^=\"https://trends.google.com/trends/explore\"]" } }, { "command": "wait_for_selector", "params": { "selector": "css=[widget-name=\"RELATED_QUERIES\"] button.widget-actions-item.export" } }, { "command": "wait_for_download", "params": { "save_path": "./related_query.csv", "selector": "css=[widget-name=\"RELATED_QUERIES\"] button.widget-actions-item.export" } } ], "authors": [] } ``` ### FAQ **How do I download related queries from Google Trends?** Launch the bot, review the required keyword field (it ships prefilled with an example term), and confirm. The launcher loads the Trends explore page, types your keyword into the search input, presses Enter, waits for the related queries widget export control, triggers it, and captures the CSV the page generates. The file lands in the run output location as related_query.csv with no further clicking. **What does the related_query.csv file contain?** Exactly what the platform export produces for the related queries widget: one row per related query, with the query text and the platform's own magnitude column - a rising percentage or a relative top index, depending on the widget state when the export fires. The bot does not reformat, rename columns, or add fields. Open it in Excel or Sheets as-is. **How many keywords can one run research?** One. Each run reads a single keyword value and produces one related_query.csv. For a list of seeds, run the launcher once per term and keep files named by keyword and date, or schedule repeated launches with your operating system scheduler, passing a different keyword each time. **Why did the run fail on the consent page?** The Trends page can present a consent or cookie interstitial before the explore input renders, particularly from EU addresses. This configuration ships no command to dismiss that interstitial, so the fill step times out and the run fails loudly with no CSV. Clear consent once in your regular browser, run from a region where the page renders directly, or extend the project with a consent-handling step. **Do I need a Google account to run it?** No. The bot reads the public explore experience as a logged-out visitor. If the platform presents a sign-in wall or verification challenge from your address, the run stops rather than attempting to authenticate; there are no credential fields anywhere in the configuration, by design. **Does it export rising or top related queries?** It captures whatever the export control of the related queries widget produces at the moment it fires, which reflects the widget state on a fresh page load. If you need the other mode deterministically, that is a small project edit - a click step on the toggle before the export wait - not a setting on this bot. **Does it collect exact search volume numbers?** No. The platform publishes relative signals - rising percentages and zero-to-one-hundred top indices - not absolute monthly search volumes, and the bot collects only what is published. Treat the CSV as directional research input, not as a volume database. **What happens when Google changes the page layout?** The selectors are pinned to current platform markup, so a redesign makes the wait step time out and the run ends with no file. Failure is loud: you get an error and an absent CSV, never a stale or fabricated spreadsheet. Check the selectors in the configuration against the live page and update them; that is the normal maintenance cost of a pinned-selector bot. **How often can I run the bot?** As often as your use case justifies at a human pace. There is no built-in limiter, so the polite pattern is a few runs per session with gaps between them, matching what a patient researcher would do. High-frequency looping from one address is what triggers verification challenges and blocks. **Can it schedule itself to run daily?** Not on its own - the configuration has no internal scheduler. Schedule the launcher with your operating system: Task Scheduler on Windows, cron on Linux, or launchd on macOS. Each scheduled launch re-reads the keyword input and writes a fresh related_query.csv. **Does the output include the interest over time chart?** No. The bot exports only the related queries widget. The interest-over-time chart has its own export control on the page, and adding it would be a project extension with a second wait-and-download step. The shipped configuration keeps one output file so every run stays predictable. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## Link Crawler Crawl internal links on a website and collect meta titles, status codes and headers into one CSV audit file. - URL: https://marketplace.rtila.com/bots/link-crawler - Category: SEO - Tags: link-crawler, seo-audit, site-crawl, meta-data, internal-links, web-audit - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.1.0 (updated 2026-09-14) - Price: free ### Overview The Link Crawler is a link crawler: a standalone desktop bot that follows every internal link on a target website, records what each page actually serves - meta title, HTTP status, content type, charset, size and selected response headers - into the `page_metadata` dataset, and exports the whole audit as a CSV file. You supply one start URL; the bot walks the internal graph from there. ## What does the Link Crawler do? You point it at a homepage or any entry page. The crawl discovers internal links on that page, fetches them, discovers the links on those pages, and keeps going to a maximum of three hops from the start, stopping at 999 URLs per run. Every page it visits contributes exactly one row: the title element from the head of the document, plus HTTP level facts read from the response itself - the status code, content type, charset, byte size, server header, cache control header, and the x robots tag header when one is sent. Because it runs on your own machine, the audit of a site - which URLs exist, which ones are broken, which ones are quietly marked noindex - never passes through a third party dashboard. One honest note up front: this is a technical metadata collector, not a content analyzer. It records what pages return; it does not read, score or rewrite what they say. ## Who uses a link crawler? The technical SEO consultant asks "are there noindexed pages that should rank, and titles missing where they should not be?" and needs the status and x robots tag columns side by side. The site owner mid migration asks "did every old URL survive the move, and are the redirects actually returning clean status codes?" and needs a dated before and after crawl. The web developer asks "which pages are serving the wrong content type or an unexpected charset?" and needs the header columns without opening a browser tab per URL. All of them want an artifact they produced - a CSV with the full URL on every row - rather than a screenshot of someone else audit UI with a project limit on it. ## How does an internal link crawler run work, step by step? Every command in the run is linked to its reference page. Before the crawl starts, the launcher collects the required project variable `target_url` - the start address, defaulted in the configuration to a documentation site so a first test run is never aimed anywhere unexpected - and that value feeds the start URL queue as `${target_url}`. The single [crawl_links](https://learn.rtila.com/reference/commands/crawl_links) command then does the walking: `link_type` internal, `max_depth` 3, `max_urls` 999, `concurrency` 7, and a `delay` of 1000 milliseconds between requests - seven polite parallel fetches, not a firehose. Only HTML style addresses are followed; PDFs, images and other file links are left alone. Inside the crawl `do` block, an [extract_data](https://learn.rtila.com/reference/commands/extract_data) step runs on every visited page against the `page_metadata` dataset, whose item selector is `css=html` - one record per whole document. The title comes from `css=head title` with a trim transformation; every HTTP fact comes from `variable=` selectors reading the response of the page just fetched (`variable=last_http_status`, `variable=last_content_type`, and so on). Rows de-duplicate on the `url` field, and `meta_title` is the dataset required field, so titleless pages are logged loudly and kept rather than silently dropped. The final export writes the consolidated dataset to CSV. The browser window stays visible during the run - headless mode is off in this configuration - so you can watch the crawl move through the site in real time. ## What does the website crawl report look like? The sample row below is the published sample output for this bot, so the table and the field notes agree by construction: | meta_title | url | status | content_type | charset | content_length | server | cache_control | x_robots_tag | |---|---|---|---|---|---|---|---|---| | RTILA X - Local First AI Web Automation Software | https://rtila.com/features | 200 | text/html; charset=UTF-8 | UTF-8 | 48213 | nginx | public, max-age=3600 | | **meta_title** is the text of the title element exactly as served, trimmed of surrounding whitespace. Empty cells are the interesting ones: they mark pages that shipped no title at load time. **url** is the full address of the visited page, read from the navigation result rather than reconstructed from the link that pointed at it - so redirects are recorded as the destination that actually answered. It is also the de-duplication key. **status** is the HTTP response code. Filter this column for anything outside the 200s and 300s to get the broken page list; 301 and 302 rows let you verify redirect chains after a migration. **content_type** is the served MIME type with any parameter suffix. HTML pages that arrive as `application/octet-stream` or plain text are configuration bugs this column surfaces instantly. **charset** is the character encoding from the response. Mismatched charset values across a site are a common source of mojibake in titles and body text. **content_length** is the response size in bytes as reported by the server. Wildly small values on pages that should be substantial often indicate error shells or stub pages. **server** is the server header when the host chooses to send one. Many hosts suppress it - empty is normal, not a failure. **cache_control** is the caching directive header. Comparing this across templates shows whether your caching policy is applied consistently or page by page by accident. **x_robots_tag** is the indexation directive header, captured verbatim when present. This is the column that catches the silent killers: a stray `noindex` left on production after a staging migration. Empty means no directive was sent at header level, which is the expected state for most pages. ## How does it work as a seo audit crawler? A seo audit crawler ultimately answers "what exists, what answers, and what is allowed to be indexed?" - and this dataset answers all three in one pass. Existence: the URL column is the discovered internal inventory, and any page you expected to find that is absent was not reachable within three hops, which is itself a finding. Health: the status column separates live pages from broken ones. Indexation: the x robots tag column exposes server level exclusion directives. The audit is comparative over time - crawl weekly or before and after a release, then diff the files by URL to see exactly what changed. What it will not do is score page content, analyze backlinks, measure rank, or track Core Web Vitals; those are different product categories, and this bot makes no claim to them. ## How do you use it as a meta title checker? Open the CSV and work the meta_title column three ways. First, filter for blanks - every empty cell is a page serving no title. Second, sort or pivot on the title values to find duplicates; two dozen category pages sharing one template title is invisible while browsing and obvious in a sorted column. Third, add a length formula in your spreadsheet to flag titles outside the range search results display cleanly. The bot deliberately ships the title verbatim rather than "fixed" - trimming whitespace is the only transformation applied - so the checker tells you what the server said, and the editorial judgment stays yours. ## How does this website crawl tool compare to hosted audit alternatives? Hosted crawling services are generally reported to price by project or by URL allowance, keep your crawl history in their infrastructure, and cap exports by plan tier. This bot inverts all three: the crawl runs on your machine, the CSV lands on your disk, the 999 URL bound per run is a politeness setting you can re-run as often as you like, and the workflow configuration is a file you own. The trade is equally honest - there is no dashboard, no stored history, no scheduled re-crawl and no trend charts. You get dated artifacts and a spreadsheet; the analysis layer is whatever you build on top, which for most audits means a pivot table and ten minutes. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the executable; SmartScreen may ask for confirmation on first launch, and "More info" then "Run anyway" proceeds if you trust the published checksum. On **macOS**, Gatekeeper may quarantine the download; right click and "Open", or clear the flag with `xattr -d com.apple.quarantine /path/to/LinkCrawler`, then launch. On **Linux**, the launcher is a shell script: `chmod +x ./link-crawler` then `./link-crawler`. At first run the launcher asks for the `target_url` value - enter the full start address including the scheme. A visible browser window opens while the crawl runs; leave it alone until the export completes. To re-run an audit weekly, schedule the launcher with your operating system scheduler (Task Scheduler, cron, or launchd) and diff the CSVs. ## What do I check when a crawl returns zero rows? In order: is the start URL a live page in an ordinary browser, or a signed in, consent walled or region locked view that a fresh session cannot reach? Did the page load but serve no HTML document the extractor could read - a pure JavaScript shell that renders after load can produce an empty title, though the row itself should still appear? Did the run end immediately because the start address redirected off site, taking the internal only crawl out of scope? The contract is that failure is loud: zero rows means the crawl could not visit pages, never that results were fabricated or quietly swallowed. For a shallow crawl - far fewer rows than the site should have - the likeliest cause is navigation that only exists after script driven rendering; check one of the missing sections manually in the visible browser window before concluding anything. ## What are the limits of the bot? It follows internal links only, to depth 3, up to 999 URLs per run - deliberate bounds that keep a single run polite and finishable. It records nine metadata fields per page and nothing else: no word counts, no heading structure, no image alt text, no inlink counts, and no depth column (a page presence in the file means it was reachable within the bounds; its hop distance is not stored). It processes pages at the load state, so late rendered content and late injected links are invisible to it. It cannot enter logged in areas, and it does not bypass paywalls or consent walls. It is a snapshotter, not a monitor: nothing re runs itself and nothing alerts you - schedule it and diff the files. ## What is responsible site crawling? Crawl only websites you own or have explicit permission to audit; pointing any crawler at a third party site you do not control raises terms of service and, in some jurisdictions, legal questions that are yours to answer before the run, not after. Keep the built in pacing - seven concurrent fetches with a one second delay - and lower it further on small shared hosts that can feel seven parallel requests. Respect x robots tag and robots directives as statements of the publisher intent, and treat the output as what it is: technical metadata about page responses, not a license to republish page content. The acceptable use policy applies in full before your first run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | target_url | text | yes | https://rtila.com | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **page_metadata** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | meta_title | text | no | css=head title | | url | text | no | variable=last_navigation_url | | status | text | no | variable=last_http_status | | content_type | text | no | variable=last_content_type | | charset | text | no | variable=last_charset | | content_length | text | no | variable=last_content_length | | server | text | no | variable=last_server | | cache_control | text | no | variable=last_cache_control | | x_robots_tag | text | no | variable=last_x_robots_tag | Export formats: csv, json, xlsx. ### Downloads - Windows x86_64: `BOT_link-crawler.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/link-crawler/windows - macOS Apple Silicon: `BOT_link-crawler.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/link-crawler/macos - macOS Intel: `BOT_link-crawler.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/link-crawler/macos_intel - Linux x86_64: `BOT_link-crawler.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/link-crawler/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=link-crawler` Configuration document: `https://marketplace.rtila.com/api/bots/link-crawler/config.json` - Encoding: plain - Digest: dbb0b3bf63d1504615d183b9db980ede7ab1fdeb950807136cfa9c3d83065084 - Bytes: 2358 ```json { "name": "Link Crawler", "settings": { "urls": [ "${target_url}" ], "headless": false, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": false, "variables": [ { "name": "target_url", "value": "https://rtila.com", "required": true, "is_list": false } ] }, "datasets": { "page_metadata": { "item_selector": "css=html", "properties": [ { "name": "meta_title", "type": "text", "selector": "css=head title", "transformations": [ { "type": "trim" } ] }, { "name": "url", "type": "text", "selector": "variable=last_navigation_url" }, { "name": "status", "type": "text", "selector": "variable=last_http_status" }, { "name": "content_type", "type": "text", "selector": "variable=last_content_type" }, { "name": "charset", "type": "text", "selector": "variable=last_charset" }, { "name": "content_length", "type": "text", "selector": "variable=last_content_length" }, { "name": "server", "type": "text", "selector": "variable=last_server" }, { "name": "cache_control", "type": "text", "selector": "variable=last_cache_control" }, { "name": "x_robots_tag", "type": "text", "selector": "variable=last_x_robots_tag" } ], "required_fields": [ "meta_title" ] } }, "commands": [ { "command": "crawl_links", "params": { "concurrency": 7, "delay": 1000, "link_type": "internal", "max_depth": 3, "max_urls": 999, "do": [ { "command": "extract_data", "params": { "dataset": "page_metadata" } } ] } } ], "authors": [] } ``` ### FAQ **How do I crawl all internal links on a website?** Enter the start URL when the launcher asks for target_url. The crawl_links command discovers internal links on every page it visits and follows them up to depth 3, capped at 999 URLs per run, extracting one metadata row per page as it goes. External links are recorded as out of scope and not followed, so one run gives you the internal graph of a single site. **How do I find missing meta titles on my site?** Run a crawl and filter the meta_title column for blanks in any spreadsheet. Because meta_title is a required field in the page_metadata dataset, rows with a missing title are logged during the run and still kept in the output, so a blank cell means the page genuinely served no title element at load time rather than the row being dropped. **What is max depth in a link crawl?** Depth counts link hops from the start URL: depth 0 is the page you entered, depth 1 is everything it links to, and so on. This bot stops at depth 3, which is deliberate - pages buried deeper than three clicks from the homepage are usually hard for search engines and users to reach too. The 999 URL cap applies on top of the depth limit, whichever is hit first ends the crawl. **How do I check x robots tag headers across a site?** Every visited page records its x_robots_tag response header into the x_robots_tag column. After the crawl, filter that column for noindex, nofollow or noarchive values to see exactly which URLs are being excluded from indexing by server level directives - the kind of instruction that overrides on page meta tags and is easy to leave behind after a migration. **Can the crawler find broken pages by http status?** Yes. The status column records the HTTP response code for every visited URL, so filtering for 404, 410, 500 and other non 200 values gives you a broken page report in one pass. Soft 404s - pages that return 200 but show an error - are not detected; check suspicious 200 rows by their meta title and content length. **How many urls can one crawl handle?** Up to 999 URLs per run, which is the max_urls bound in the crawl_links configuration. With seven concurrent fetches and a one second delay between requests, a full 999 URL crawl typically finishes in a few minutes on a responsive host. Larger sites can be audited in sections by starting separate runs from different entry URLs. **Does the crawler visit external links?** No. The crawl is configured with link_type internal, so it follows only links that stay on the same site. External links are neither visited nor added to the row count. That keeps the run a site audit rather than a web crawl, and keeps request volume directed at one host you are responsible for treating politely. **How do I export crawl results to CSV?** The final extract_data step consolidates the page_metadata dataset and the standalone launcher writes it out as a CSV file you can open directly in Excel or Sheets - no conversion step. The same data can be exported to JSON or XLSX from the launcher export options if your workflow prefers those formats. **Why are duplicate urls removed from the crawl?** The page_metadata dataset de-duplicates on the url field, and the crawler tracks visited URLs as it goes, so each address appears exactly once even when dozens of pages link to it. Query parameter variants of the same page are treated as distinct URLs, which is intentional - spotting parameter duplication is one of the useful outcomes of an audit. **Does the link crawler work on JavaScript sites?** Partially. Pages are processed at the load state in a real browser, so titles and headers rendered by that point are captured. Sites that render their content or navigation only after load may produce rows with an empty meta_title or a shallow link graph, because links injected later are never discovered. A visible browser window opens during the run so you can see exactly what the crawler sees. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## Local Business Profile Scraper Search a map category and collect phones, websites and ratings from every business profile page. - URL: https://marketplace.rtila.com/bots/local-business-profile-scraper - Category: Lead Generation - Tags: lead-generation, business-profiles, maps-data, local-seo, business-directory, data-collection - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-09-20) - Price: free ### Overview The Local Business Profile Scraper is a business profile scraper: a standalone desktop bot that types a category search into a map platform, scrolls the results pane to the end, then visits every listing's own profile page and extracts eleven detail fields into the `Business_Details` dataset. Where a results-card scraper stops at names and ratings, this bot goes one level deeper - phone numbers, websites, menus, amenities, price ranges and plus codes all live on the individual profile pages, and that is exactly where the run goes to get them. ## What does the Local Business Profile Scraper do? The run happens in two phases. Phase one works on the search results page: the bot fills the map search box with the term you provide, presses Enter, waits for result cards to render, and scrolls the results pane until it stops growing. It then extracts every listing's profile URL into the `Map_Search_URLs` dataset. Phase two iterates over those URLs one by one: for each profile page it waits for the heading to render, then extracts the eleven `Business_Details` fields - including a parsed amenities list and the numeric rating and review count - and appends the row. Because everything runs on your own machine, your research footprint stays yours: which categories you are mapping, which cities you are prospecting, which competitors you are benchmarking never passes through a third-party service's dashboard or logs. One honesty note, in the same spirit as its sibling listing: the selectors shipped in this configuration target the current public markup of a major map platform and are illustrative examples for the reader's own permitted target. They are not an affiliation, not an endorsement, and not a guarantee - when a platform redesigns its profile pages, a run ends with empty fields rather than invented ones. ## Who uses a business profile scraper? The local lead-generation freelancer asks "which plumbers in this city publish a phone number but have no website?" and needs phone and website columns side by side to spot exactly that gap. The market researcher asks "what is the density and rating distribution of coffee shops across these three districts?" and needs category, rating and review count per profile. The franchise development team asks "which territories have weak amenity coverage?" and needs the parsed amenities list. The local-SEO consultant asks "what does the profile data of the top-ranked businesses actually contain?" and needs a dated, defensible artifact - a dataset they produced themselves - rather than a screenshot. ## How does a local business data extractor run work, step by step? Every command in the run links to its reference page. First, [fill](https://learn.rtila.com/reference/commands/fill) types your prompted `search_term` variable - `${search_term}`, default "coffee shops" - into the map search box, clearing any existing text first, and its `then` chain fires a [press](https://learn.rtila.com/reference/commands/press) of the Enter key to submit the search. Second, [wait_for_selector](https://learn.rtila.com/reference/commands/wait_for_selector) blocks until the first result card renders, so extraction never races the page. Third, [infinite_scroll](https://learn.rtila.com/reference/commands/infinite_scroll) drives the results pane with a 1500 ms delay per step and an unbounded scroll count - it stops on its own when the pane height stops changing. Fourth, [extract_data](https://learn.rtila.com/reference/commands/extract_data) pulls every profile link into `Map_Search_URLs`. Finally a [for_each](https://learn.rtila.com/reference/commands/for_each) loop iterates that dataset: for each item, [goto](https://learn.rtila.com/reference/commands/goto) navigates to `${item.PlaceURL}`, a second wait_for_selector confirms the profile heading loaded, and a second extract_data appends the eleven-field record to `Business_Details`. ## What does the output look like? The table mirrors the published sample row exactly: | Title | Category | Amenities | Phone | Website | Menu | Price_Range | Rating | Reviews_Count | Address | Plus_Code | |---|---|---|---|---|---|---|---|---|---|---| | Blue Door Coffee House | Coffee shop | Wi-Fi, outdoor seating, wheelchair accessible | +1 555 010 2030 | https://bluedoorcoffee.example.com | https://bluedoorcoffee.example.com/menu | $$ | 4.5 | 1284 | 482 Example Ave, Springfield | 8FVC2G3M+4X | **Title** is the business name from the profile heading. The run waits for this heading before extracting, so an empty Title is a strong signal the page never rendered properly. **Category** is the platform's own classification. Filter on it after the run - map searches routinely mix in adjacent categories. **Amenities** is parsed out of the profile's about section into one label per line and joined; it captures feature labels like Wi-Fi or wheelchair access, never review content. **Phone** and **Website** are the public contact fields, cleaned of their accessibility-label prefixes by replace transformations. Both are legitimately empty on many profiles - that absence is itself useful lead-generation signal. **Menu** is the menu link where the business publishes one. Mostly relevant for food and drink categories; expect empty elsewhere. **Price_Range** is the first bucket of the displayed price histogram, kept verbatim. **Rating** and **Reviews_Count** are the aggregate signals, extracted by regex from the star label and review label - numbers only, review text is never collected. **Address** is the published street address, prefix-cleaned. Combine it with Title when merging runs, since this release has no deduplication key. **Plus_Code** is the open-location code for the entrance, useful for territories where street addressing is unreliable. ## Can a maps detail page scraper build a lead list? Yes - with one boundary stated plainly. A maps detail page scraper gives you the raw prospect row: who, where, what category, how well rated, and the two contact channels the business chose to publish (phone and website). What this release does not do is enrich: it extracts no email addresses, resolves no social profiles, and validates nothing against third-party databases. The output is a clean starting list, not a finished campaign file, and any outreach from it is your decision and your compliance burden. ## How is this different from a business directory scraper? A business directory scraper works against paginated HTML directories with numbered pages and stable cards. Map results are different animals: they render continuously inside a scrolling pane, the interesting data hides on individual profile pages rather than in the cards, and key fields are wrapped in accessibility labels that need transformation to clean. This bot is built for that shape - unbounded scroll detection, a URL-collection phase, and a per-profile visit loop - which is why it collects fields a card-level scraper never sees. ## How does it compare to hosted collection and manual research? Copy-pasting profiles by hand produces a handful of rows an hour and no consistency at all. Hosted collection services and metered platforms will do the visiting for you, but the structure is the opposite of this bot: they charge per request or per month, the data lands in their cloud under their retention terms, and your search history - your prospects, your territories - lives on their servers. Here the marginal cost of a bigger search is your own time and bandwidth, the datasets stay on your disk, and the workflow configuration is yours to keep, modify and re-run. The trade is real: no dashboards, no built-in scheduling, no history. You get an artifact and a workflow you own outright. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the .exe; SmartScreen may ask for confirmation on first launch - verify the published checksum, then "More info" and "Run anyway" proceeds. On **macOS**, Gatekeeper may quarantine the download; right-click and choose Open, or clear the flag with `xattr -d com.apple.quarantine /path/to/LocalBusinessProfileScraper` and launch normally. On **Linux**, the launcher is a shell script: `chmod +x ./local-business-profile-scraper` then `./local-business-profile-scraper`. At first run it prompts for a single field - the search term, such as "dentists in Austin" - and starts the two-phase run. To repeat a search weekly, schedule the launcher with your OS scheduler (Task Scheduler, cron, or launchd) and pass the same term. ## What do I check when a run returns zero rows or missing fields? Work in order. Zero rows from `Map_Search_URLs`: did the search actually return results in an ordinary browser, or did a consent wall, region lock or challenge greet the fresh session? Did the scroll phase end immediately because the results-pane container no longer matches a redesigned page? Every profile row present but fields empty: the profile-page markup most likely changed - check one profile URL from `Map_Search_URLs` by hand and see whether the page still shows the fields. A few scattered empty phones or websites: completely normal, not a failure. Fewer results than the map shows interactively: the scroll detection stopped when the pane stopped growing; re-run once before concluding anything. The contract is that failure is loud - empty values and zero rows, never fabricated data. ## What are the bot's limits? It collects only what a logged-out visitor sees on search cards and profile pages. No review text, no owner responses, no photos, no opening hours in this release. It has no email extraction and no enrichment step. It ships without proxy rotation or CAPTCHA solving enabled - if a challenge stops the run, the run stops; nothing bypasses it. One run covers one search term, unbounded: there is no visit cap, so scope control means narrowing your query. It is a snapshotter, not a monitor - nothing re-runs or alerts on its own. And `Business_Details` carries no deduplication key, so merging runs is your step, on Title plus Address. ## What is responsible lead generation data collection? A sole trader's name, phone number and address are personal data in most jurisdictions, so establish your lawful basis under GDPR, UK GDPR or your local equivalent before any outreach to collected contacts; honour opt-outs immediately and permanently; and never use the output for spam. Review the map platform's terms with open eyes - automated collection may conflict with them, and the decision to run is yours to own. Keep the collection boundary where this bot sets it: aggregate ratings and counts, published contact channels, never review text. Keep request volume at what a patient human could sustain, and let the acceptable-use policy govern every run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | search_term | text | yes | coffee shops | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **Business_Details** (item selector: ``) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Category | text | no | css=button[jsaction*='category'] | | Title | text | no | css=h1 | | Amenities | html | no | css=div[role="region"][aria-label*="About"] | | Phone | attribute | no | css=[data-item-id*='phone'] | | Website | attribute | no | css=[data-item-id*='authority'] | | Menu | attribute | no | css=[data-item-id='menu'] | | Price_Range | text | no | css=table[aria-label='Price range histogram'] tr:first-child td:first-child | | Rating | attribute | no | css=[role='img'][aria-label*='stars' i] | | Reviews_Count | text | no | css=[aria-label*='reviews' i] | | Address | attribute | no | css=[data-item-id*='address' i] | | Plus_Code | attribute | no | css=[data-item-id='oloc'] | **Map_Search_URLs** (item selector: `css=div[aria-label*='Results for '] div[role='article']`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | PlaceURL | attribute | no | css=a[href*='/maps/place/'] | ### Downloads - Windows x86_64: `BOT_local-business-profile-scraper.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/local-business-profile-scraper/windows - macOS Apple Silicon: `BOT_local-business-profile-scraper.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/local-business-profile-scraper/macos - macOS Intel: `BOT_local-business-profile-scraper.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/local-business-profile-scraper/macos_intel - Linux x86_64: `BOT_local-business-profile-scraper.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/local-business-profile-scraper/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=local-business-profile-scraper` Configuration document: `https://marketplace.rtila.com/api/bots/local-business-profile-scraper/config.json` - Encoding: plain - Digest: 2bd85667352855564dd6b8b289c96fac25e32786fadf693e04b3f6d85b132411 - Bytes: 5046 ```json { "name": "Google Maps Local Business Scraper", "settings": { "urls": [ "https://www.google.com/maps/?hl=en" ], "headless": false, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": false, "variables": [ { "name": "search_term", "value": "coffee shops", "required": true, "is_list": false } ] }, "datasets": { "Business_Details": { "item_selector": "", "properties": [ { "name": "Category", "type": "text", "selector": "css=button[jsaction*='category']" }, { "name": "Title", "type": "text", "selector": "css=h1" }, { "name": "Amenities", "type": "html", "selector": "css=div[role=\"region\"][aria-label*=\"About\"]", "transformations": [ { "type": "script", "code": "const regex = /aria-label=\"([^\"]+)\"/g;\nconst matches = [...value.matchAll(regex)].map(match => match[1]);\nreturn matches .join(\"\\n\");" } ] }, { "name": "Phone", "type": "attribute", "selector": "css=[data-item-id*='phone']", "attribute": "aria-label", "transformations": [ { "type": "replace" } ] }, { "name": "Website", "type": "attribute", "selector": "css=[data-item-id*='authority']", "attribute": "href", "transformations": [ { "type": "replace" } ] }, { "name": "Menu", "type": "attribute", "selector": "css=[data-item-id='menu']", "attribute": "href" }, { "name": "Price_Range", "type": "text", "selector": "css=table[aria-label='Price range histogram'] tr:first-child td:first-child" }, { "name": "Rating", "type": "attribute", "selector": "css=[role='img'][aria-label*='stars' i]", "attribute": "aria-label", "transformations": [ { "type": "regex", "pattern": "([\\d.,]+)", "default_value": "" } ] }, { "name": "Reviews_Count", "type": "text", "selector": "css=[aria-label*='reviews' i]", "transformations": [ { "type": "regex", "pattern": "([\\d,]+)", "default_value": "" } ] }, { "name": "Address", "type": "attribute", "selector": "css=[data-item-id*='address' i]", "attribute": "aria-label", "transformations": [ { "type": "replace" } ] }, { "name": "Plus_Code", "type": "attribute", "selector": "css=[data-item-id='oloc']", "attribute": "aria-label", "transformations": [ { "type": "replace" } ] } ] }, "Map_Search_URLs": { "item_selector": "css=div[aria-label*='Results for '] div[role='article']", "properties": [ { "name": "PlaceURL", "type": "attribute", "selector": "css=a[href*='/maps/place/']", "attribute": "href" } ] } }, "commands": [ { "command": "fill", "params": { "clear_before_fill": true, "selector": "css=input[name='q']", "value": "${search_term}" } }, { "command": "wait_for_selector", "params": { "selector": "css=div[aria-label*='Results for '] div[role='article']" } }, { "command": "infinite_scroll", "params": { "do": [], "max_scrolls": 0, "scroll_delay": 1500, "selector": "css=div[aria-label*='Results for ']" } }, { "command": "extract_data", "params": { "dataset": "Map_Search_URLs" } }, { "command": "for_each", "params": { "source_type": "dataset", "as": "item", "dataset": "Map_Search_URLs", "do": [ { "command": "goto", "params": { "url": "${item.PlaceURL}" } }, { "command": "wait_for_selector", "params": { "selector": "css=h1" } }, { "command": "extract_data", "params": { "dataset": "Business_Details" } } ] } } ], "authors": [] } ``` ### FAQ **How do I scrape business details from map search results?** Run the launcher, type a category and location such as coffee shops in Springfield into the single search prompt, and the bot fills the map search box, scrolls the results pane to the end, collects every listing's profile URL, then visits each one and extracts the eleven Business_Details fields. No selector work is required from you. **What data does the business profile scraper collect?** Eleven fields per profile: Category, Title, Amenities, Phone, Website, Menu, Price_Range, Rating, Reviews_Count, Address and Plus_Code. Everything comes from the publicly rendered profile page - no logged-in views, no review text, no photos. **Why does the bot visit every listing page one by one?** Search-result cards only carry a fraction of the data. Phone numbers, websites, menus, amenities, price ranges and plus codes live on the individual profile pages, so the run collects profile URLs first, then iterates through them with a for_each loop, waiting for each page's heading before extracting. That is also why run time scales with result count. **How long does a full profile run take?** Budget roughly one page visit per listing on top of the scroll phase: a search returning 100 results typically takes 15-30 minutes at human-paced navigation, and large city-wide searches can run an hour or more. Narrow the search term or add a district qualifier to keep runs short. **Does it collect review text?** No, deliberately. The dataset captures the aggregate Rating and Reviews_Count only. Review text is personal data with a much higher compliance bar, and this configuration does not touch it. **Where does the collected business data go?** Rows are stored under the two dataset names from the configuration - Map_Search_URLs for the profile links and Business_Details for the eleven-field records - in the launcher's local results store on your machine. This release sets no fixed export filename, so you export from the results view; nothing is uploaded anywhere except the configuration fetch and download telemetry listed on this page. **Is collecting business profile data allowed?** Automated collection may conflict with the map platform's terms of service, and a sole trader's name, phone and address are personal data in most jurisdictions. You are responsible for reviewing the platform terms and establishing a lawful basis under GDPR, UK GDPR or your local equivalent before running the bot or contacting anyone from the output. **Does the scraper work offline?** No. The search results and profile pages are live web content, so the run needs a network connection throughout. The only other call the launcher makes is the configuration fetch at startup. **Can I limit how many profiles are visited?** This release ships with unbounded scrolling - it collects until the results pane stops growing - and then visits every collected profile. The practical control is your search term: add a neighbourhood, district or narrower category to shrink the result set before the run starts. **How are missing phone numbers or websites handled?** Empty is normal and expected. Not every business publishes a phone number, website or menu, and the transformations fall back to empty values rather than failing the row. Business_Details has no deduplication key in this release, so if a listing appears twice in the results pane it can produce two rows - check Title plus Address when merging. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## Mass Alive Checker Check a URL list in one run: alive or dead, with status codes, redirects, final URLs and server details. - URL: https://marketplace.rtila.com/bots/mass-alive-checker - Category: Utility - Tags: url-status-check, dead-link-checker, bulk-url-checker, http-status-codes, redirect-checker, seo-audit, web-maintenance - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-09-20) - Price: free ### Overview The Mass Alive Checker is a bulk URL status checker: a standalone desktop bot that takes a list of URLs, visits each one in a real browser, and records whether it is alive or dead along with its HTTP status code, final URL after redirects, redirect count, content type, server header and protocol. Results are exported as CSV across three datasets - `Alive_Checker` (the full report), `Alive_URLs` and `Dead_URLs`. ## What does the Mass Alive Checker do? You paste a list of URLs, one per line, and the bot works through the whole list in a single run. It is built for speed and honesty rather than depth: three browser workers run in parallel, and every sub-resource - images, scripts, stylesheets, fonts, media, XHR, fetch, websockets and manifests - is blocked, because a status check only needs the navigation response itself. Each URL is visited inside error handling, so a timeout or a domain that no longer resolves never kills the run; the failure is recorded and the next URL proceeds. The scope is deliberately narrow: it validates a list you supply. It does not crawl sites, discover links, or re-check anything on a schedule. ## Who uses a bulk URL status checker? The SEO consultant cleaning a backlink list asks "which of these three hundred domains still resolve, and which redirect somewhere else entirely?" The webmaster mid-migration asks "do all the old URLs return a 3xx, and do they land where I intended?" The content maintainer asks "which links on our resource pages have quietly died this year?" The developer asks "which endpoints in this list now serve JSON or a PDF instead of HTML?" All four need the same artifact: a dated CSV with one row per URL, a status code, and a final destination - produced on their own machine, with no per-check URL cap and nothing uploaded to anyone else's servers. ## How does a run work, step by step? Inputs come from project settings rather than a mid-run dialog: `urls_to_check` (a required list, one URL per line), `check_status_min` (default 100) and `check_status_max` (default 399). The engine starts at a blank page - initial navigation is skipped - and works through your list. For each URL, a [try_catch](https://learn.rtila.com/reference/commands/try_catch) block wraps a [goto](https://learn.rtila.com/reference/commands/goto) navigation; if the navigation throws, the catch branch logs the failure with a [comment](https://learn.rtila.com/reference/commands/comment) step carrying `${last_error}` and the run continues. After the attempt, the engine's last-navigation metadata (target, status, final URL, redirect count, content type, server, protocol, success flag, error) is read by three [extract_data](https://learn.rtila.com/reference/commands/extract_data) steps: the full `Alive_Checker` report; `Alive_URLs`, filtered to statuses where `Number(item.Status) >= Number('${check_status_min}') && Number(item.Status) <= Number('${check_status_max}')`; and `Dead_URLs`, filtered to everything else. Each dataset is exported as CSV beside the launcher. ## What does the HTTP status code checker output look like? The sample row is the listing's own published sample output, so table and prose agree by construction: | Target_URL | Status | Final_URL | Redirect_Count | Content_Type | Server | Protocol | Navigation_OK | Error | |---|---|---|---|---|---|---|---|---| | https://shop.example.com/old-page | 301 | https://shop.example.com/new-page | 1 | text/html; charset=UTF-8 | nginx | HTTP/2 | true | | **Target_URL** is the URL you supplied, exactly as pasted - the join key back to your original list. **Status** is the HTTP status code observed at the end of navigation: 200 for a clean hit, 301/302 for a redirect, 404/410 for gone, 5xx for server errors. **Final_URL** is where the browser actually landed after all server-level redirects. Diff it against your intended destination to catch mis-mapped redirects. **Redirect_Count** is the number of hops taken. Anything above one is a chain worth collapsing for crawl efficiency. **Content_Type** is the served MIME type - the column that reveals a URL now answering with a PDF, JSON or a generic error page type instead of HTML. **Server** is the server response header when the host publishes one. Empty is normal: many servers omit or mask it, and absence is never treated as a failure. **Protocol** is the negotiated protocol version, useful for spotting URLs still stuck on plain HTTP. **Navigation_OK** is the success flag for the attempt itself. When it reads false, the URL never completed a navigation and the row belongs in Dead_URLs regardless of status. **Error** carries the failure text - timeout, DNS resolution failure, connection refused - and is empty on successful navigations. ## How does it work as a dead link checker for long lists? The split happens inside the run, not in your spreadsheet afterwards: Dead_URLs collects every row whose status falls outside your alive window or whose navigation failed, with the reason attached. Because sub-resources are blocked and three workers run in parallel, long lists finish in minutes rather than hours, and there is no per-run cap on how many URLs you paste - the practical limits are your connection and the patience of the target servers. Hosted tools in this space commonly meter checks per batch; here the marginal cost of a bigger list is your own bandwidth. ## Can it serve as a URL list validator before a migration? Yes, in both directions. Before the switch, run the old URL list and confirm nothing is already dead. After the switch, run it again and validate three things per row: the Status is a 3xx, the Final_URL matches the intended new location, and Redirect_Count is one. Rows that fail any of the three become your fix list, and re-running after fixes produces a clean dated CSV you can attach to the migration report as evidence. It is a snapshot validator, though - nothing re-runs itself, so schedule the launcher through your operating system's scheduler if you want periodic checks. ## How does the bulk redirect checker data help after a redesign? Redesigns break URLs in patterns: old paths 404ing, new paths redirecting to the homepage instead of the right page, chains forming where two redirect rules overlap, and protocol stragglers still answering on HTTP. The Redirect_Count, Final_URL and Protocol columns turn each pattern into a one-line spreadsheet filter. A pivot on Status and Content_Type after a platform rebuild quickly answers "how many old article URLs now serve something other than an HTML page?" ## How does it compare to hosted crawlers and monitoring services? Desktop crawlers discover links themselves and audit whole sites; this bot does the opposite - it validates a list you already have, and never wanders beyond it. Hosted checkers and monitoring services run continuously, keep your URL lists and results in their cloud, and commonly meter how many URLs one check may contain; this bot runs once per launch, entirely on your machine, with the CSVs landing on your disk and no account, upload or per-check quota. The trade is real: no dashboards, no alerting, no history, no crawl discovery. You own the artifact and the workflow outright, and a thousand-URL list costs exactly as much as a ten-URL one. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the executable; SmartScreen may ask for confirmation on first launch, and "More info" then "Run anyway" proceeds if you trust the published checksum. On **macOS**, Gatekeeper may quarantine the download; right-click and "Open", or clear the flag with `xattr -d com.apple.quarantine /path/to/MassAliveChecker`, then launch. On **Linux**, the launcher is a shell script: `chmod +x ./mass-alive-checker` then `./mass-alive-checker`. On first run, paste your URL list into `urls_to_check` (one per line) and adjust the status thresholds if you want anything other than the default 100-399 alive window; the shipped defaults include one deliberately non-existent domain so you can see what a failure row looks like. A visible browser window opens during the run - flip the headless setting to true for unattended use, and pair it with your OS scheduler for recurring checks. ## What do I check when a run returns zero rows? In order: was the list pasted into `urls_to_check` one URL per line, with full `https://` schemes? Did every navigation fail at once - check the visible browser window and your own connection, since a dead local network produces an all-error batch. Are the thresholds sane - `check_status_min` greater than `check_status_max` would classify everything as dead, and an unusual window can hide rows from the dataset you expected them in. If Alive_Checker itself is empty, the runs never recorded navigation metadata; re-run a handful of URLs and watch the browser. The contract is that failure is loud: dead URLs appear in Dead_URLs with their error text, and a row is never fabricated, silently dropped, or marked alive without an observed response. ## What are the limits of the bot? It checks the list you give it - no crawling, no link discovery, no sitemap parsing. It runs once per launch: no scheduling, alerting, history or diffing is built in. There is no retry logic; a flaky host produces one failure row, and re-running is your retry. Because scripts are blocked for speed, JavaScript-driven redirects fired after load may not execute, and pages behind logins, consent walls or bot challenges will report their gate response rather than real content. Server and Protocol columns depend on what hosts choose to publish - empty values are normal. And it sees one moment in time: a 200 now says nothing about next Tuesday. ## What is responsible URL list checking? Check URLs you own or have explicit permission to test. Three parallel workers with blocked sub-resources is already a light footprint, but keep lists batched and re-runs purposeful - do not hammer a small host, and never point the bot at third-party infrastructure as a probe or stress test. Server headers in the output are operational metadata about systems you are entitled to audit; using them to fingerprint and target sites you do not own is out of scope and out of bounds. The acceptable-use policy applies in full before your first run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | urls_to_check | list | yes | https://www.google.com https://example.com https://this-domain-does-not-exist-9x7q.com | Declared in project settings. | | check_status_min | text | no | 100 | Declared in project settings. | | check_status_max | text | no | 399 | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **Alive_Checker** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Target_URL | text | no | variable=last_target_url | | Status | text | no | variable=last_http_status | | Final_URL | text | no | variable=last_final_url | | Redirect_Count | text | no | variable=last_redirect_count | | Content_Type | text | no | variable=last_content_type | | Server | text | no | variable=last_server | | Protocol | text | no | variable=last_protocol | | Navigation_OK | text | no | variable=last_navigation_ok | | Error | text | no | variable=last_navigation_error | **Alive_URLs** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | URL | text | no | variable=last_target_url | | Status | text | no | variable=last_http_status | **Dead_URLs** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | URL | text | no | variable=last_target_url | | Status | text | no | variable=last_http_status | | Error | text | no | variable=last_navigation_error | Export formats: csv. ### Downloads - Windows x86_64: `BOT_mass-alive-checker.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/mass-alive-checker/windows - macOS Apple Silicon: `BOT_mass-alive-checker.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/mass-alive-checker/macos - macOS Intel: `BOT_mass-alive-checker.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/mass-alive-checker/macos_intel - Linux x86_64: `BOT_mass-alive-checker.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/mass-alive-checker/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=mass-alive-checker` Configuration document: `https://marketplace.rtila.com/api/bots/mass-alive-checker/config.json` - Encoding: plain - Digest: bebba5c7baa6c7c1c714429af7b2b0ebdd9479cde7bc057b8a13cfb8209434c0 - Bytes: 4006 ```json { "name": "Mass Alive Checker", "settings": { "urls": [ "${urls_to_check}" ], "headless": false, "max_concurrent_workers": 3, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": true, "variables": [ { "name": "urls_to_check", "value": "https://www.google.com\nhttps://example.com\nhttps://this-domain-does-not-exist-9x7q.com", "required": true, "is_list": true }, { "name": "check_status_min", "value": "100", "required": false, "is_list": false }, { "name": "check_status_max", "value": "399", "required": false, "is_list": false } ] }, "datasets": { "Alive_Checker": { "item_selector": "css=html", "properties": [ { "name": "Target_URL", "type": "text", "selector": "variable=last_target_url" }, { "name": "Status", "type": "text", "selector": "variable=last_http_status" }, { "name": "Final_URL", "type": "text", "selector": "variable=last_final_url" }, { "name": "Redirect_Count", "type": "text", "selector": "variable=last_redirect_count" }, { "name": "Content_Type", "type": "text", "selector": "variable=last_content_type" }, { "name": "Server", "type": "text", "selector": "variable=last_server" }, { "name": "Protocol", "type": "text", "selector": "variable=last_protocol" }, { "name": "Navigation_OK", "type": "text", "selector": "variable=last_navigation_ok" }, { "name": "Error", "type": "text", "selector": "variable=last_navigation_error" } ] }, "Alive_URLs": { "item_selector": "css=html", "properties": [ { "name": "URL", "type": "text", "selector": "variable=last_target_url" }, { "name": "Status", "type": "text", "selector": "variable=last_http_status" } ] }, "Dead_URLs": { "item_selector": "css=html", "properties": [ { "name": "URL", "type": "text", "selector": "variable=last_target_url" }, { "name": "Status", "type": "text", "selector": "variable=last_http_status" }, { "name": "Error", "type": "text", "selector": "variable=last_navigation_error" } ] } }, "commands": [ { "command": "try_catch", "params": { "try": [ { "command": "goto", "params": { "url": "${last_target_url}" } } ], "catch": [ { "command": "comment", "params": { "text": "${last_error}" } } ] } }, { "command": "extract_data", "params": { "dataset": "Alive_Checker" } }, { "command": "extract_data", "params": { "dataset": "Alive_URLs", "filter_condition": "Number(item.Status) >= Number('${check_status_min}') && Number(item.Status) <= Number('${check_status_max}')" } }, { "command": "extract_data", "params": { "dataset": "Dead_URLs", "filter_condition": "!(Number(item.Status) >= Number('${check_status_min}') && Number(item.Status) <= Number('${check_status_max}'))" } } ], "authors": [] } ``` ### FAQ **How do I check if a list of URLs is still live?** Paste the list, one URL per line, into the urls_to_check variable on the launcher input screen and start the run. The bot visits each URL in a real browser, records its HTTP status, and splits the results into an Alive_URLs dataset and a Dead_URLs dataset, both exported as CSV you can open in Excel or Sheets. **How do I find dead links in bulk?** The Dead_URLs dataset isolates every URL whose final status falls outside your alive window (100-399 by default) or whose navigation failed outright, with an Error column explaining timeouts, DNS failures and connection resets. You get the full dead list in one file instead of hunting through the complete report. **What HTTP status codes count as alive?** By default, any status from 100 to 399 counts as alive, which means 3xx redirects land in Alive_URLs because the resource still exists somewhere. If you want only clean 200-level responses treated as alive, set check_status_min to 200 and check_status_max to 299 and redirects will move to the dead list instead. **How do I check redirects for a list of URLs?** Every row of the full Alive_Checker report carries a Redirect_Count and a Final_URL column. Redirect_Count tells you how many hops the URL took, and Final_URL tells you where it ended up, so you can spot both single 301s and long redirect chains without opening a browser for each URL. **Can I customize which status codes are treated as alive?** Yes. The check_status_min and check_status_max variables define the alive window, and both filtered datasets are driven by that same window, so Alive_URLs and Dead_URLs always stay consistent with each other no matter what thresholds you set. **How fast is a bulk URL status check?** The configuration runs three browser workers in parallel and blocks all sub-resources - images, scripts, stylesheets, fonts, media, XHR and websockets - because a status check only needs the navigation itself. That makes per-URL cost close to raw response time; hundreds of URLs typically finish in minutes, with slow hosts and timeouts dominating the total. **Does it capture the final URL after redirects?** Yes, Final_URL records the address the browser landed on after all server-level redirects. One honest caveat: because scripts are blocked for speed, purely client-side redirects fired by JavaScript after page load may not execute, so a JS-driven redirect can show as its initial response rather than its destination. **How do I audit a domain migration for broken links?** Export your old URL list from a sitemap or analytics package, run it through the bot, then check three things in the CSV: old URLs now returning 404 or 410, Final_URL values that do not match the intended new destination, and Redirect_Count values greater than one that reveal chains worth collapsing. Re-run after fixes to confirm. **Does the checker work without a browser window?** The shipped configuration runs with headless set to false, so a visible browser window opens and you can watch each URL being checked - useful for diagnosing slow or blocked runs. For scheduled or background use, flip the headless setting to true and the run proceeds without any window. **What happens when a URL times out or fails to resolve?** Each navigation is wrapped in error handling, so one bad URL never stops the run. The failure is recorded with Navigation_OK set to false and the error text captured in the Error column, and the row lands in Dead_URLs. Failures are always visible in the output; a row is never silently dropped or marked alive. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## On-Page SEO Auditor Audit any URL list for indexability, metadata, readability, links, structured data and accessibility - on your own machine. - URL: https://marketplace.rtila.com/bots/on-page-seo-auditor - Category: SEO - Tags: seo, on-page-seo, website-audit, technical-seo, accessibility, structured-data - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-09-20) - Price: free ### Overview The On-Page SEO Auditor is an on-page SEO auditor: a standalone desktop bot that visits every URL you hand it and measures each page across nine audit datasets - indexability and HTTP response, on-page elements, content readability, a full links census, media and DOM performance, structured data, an AXE accessibility audit, and two triage views that separate critical issues from healthy pages. Each dataset lands in its own CSV output named after the dataset, beside the launcher, so a run over fifty URLs produces fifty comparable rows per file and nothing is trapped inside a vendor dashboard. ## What does the On-Page SEO Auditor do? You supply a list of URLs in the required `urls_to_audit` project setting, one per line, and the bot works through it with up to four concurrent workers. It starts from a blank page rather than pre-navigating, so every fetch happens inside the audited command flow where the response metadata - status, final URL, redirect chain, headers, protocol, timing - can be captured per page. To keep runs fast it blocks image, media, font, stylesheet, websocket, manifest, texttrack, eventsource and other network requests; the audit reads the DOM and HTTP layer, not pixels. Thresholds are settings variables you can adjust before a run: `title_pixel_limit` (580, the SERP title cutoff in Arial 20px), `desc_pixel_limit` (990, the meta description cutoff in Arial 14px), `low_content_words` (300, below which a page is flagged thin) and `max_outbound_links` (100, above which outbound linking looks like a link farm). Because everything executes on your machine, the URL list you are auditing - which for an agency is effectively a client roster - never passes through anyone else's servers. ## Who uses an on-page seo auditor? The in-house SEO lead asks "did the new template break titles, canonicals or headings across our top pages?" and needs a repeatable pre-publish QA pass. The agency account manager asks "what is wrong with these thirty client URLs, in a file I can attach to a report?" and needs audit artifacts produced locally rather than uploaded to a third-party scanner. The developer asks "is this page indexable, does it ship one H1, and are the security headers in place before launch?" The content lead asks "which pages are thin, which read like a legal contract, and where did placeholder text survive to production?" All four want a dated, columnar artifact they produced themselves - not a score inside someone else's web app. ## How does a website audit bot run work, step by step? Every command in the run is linked to its reference page. For each URL, a [try_catch](https://learn.rtila.com/reference/commands/try_catch) block wraps the [goto](https://learn.rtila.com/reference/commands/goto) navigation with a 60-second timeout; if navigation fails, a [comment](https://learn.rtila.com/reference/commands/comment) logs the failure and the run continues with the next target instead of dying. [wait_for_load_state](https://learn.rtila.com/reference/commands/wait_for_load_state) then settles the page to network-idle, and two [scroll](https://learn.rtila.com/reference/commands/scroll) passes - to the bottom, then back to the top - trigger lazy-loaded content so late-rendered headings, images and links are counted. [detect_tech_stack](https://learn.rtila.com/reference/commands/detect_tech_stack) fingerprints the platform and stores it for the Structured_Data file. Six [extract_data](https://learn.rtila.com/reference/commands/extract_data) steps then run in order over the datasets Response_Indexability, OnPage_Elements, Content_Readability, Links_Census, Media_Performance and Structured_Data, each de-duplicating on Page_URL. A second [try_catch](https://learn.rtila.com/reference/commands/try_catch) wraps [audit_accessibility](https://learn.rtila.com/reference/commands/audit_accessibility), storing the AXE report so the Accessibility_AXE extraction can grade it - and if the engine cannot run, the file says so explicitly. Finally two filtered extractions produce the triage views: Critical_Issues keeps only pages with a 4xx-or-worse status, a title or H1 count other than one, duplicate descriptions or canonicals, a missing viewport, any noindex signal, more than a quarter of images missing alt text, or over 1500 DOM nodes; Healthy_Pages keeps only 2xx pages with exactly one title, one H1, at most one canonical, a title between 10 and 60 characters, and a response under 3000 ms. ## What does the audit output look like? The sample row is from Response_Indexability, the first dataset, whose columns frame every page's verdict: | Page_URL | Status_Code | Status_Class | Is_Success | Response_Time_ms | HTTP_Protocol | Meta_Robots | Canonical_URL | Redirect_Count | |---|---|---|---|---|---|---|---|---| | https://example.com/ | 200 | 2xx Success | true | 412 | h2 | index, follow | https://example.com/ | 0 | **Page_URL** is the URL you submitted and the de-duplication key across all nine files, so rows from different datasets join cleanly in a spreadsheet. **Status_Code** and **Status_Class** record the HTTP response and bucket it from 1xx through 5xx, with boolean companions like Is_Success and Is_Redirect for fast filtering. **Response_Time_ms** is the measured response time, the lab-side timing signal in this dataset. **HTTP_Protocol** shows whether the page was served over h2 or h3, which matters for performance triage. **Meta_Robots** is the raw robots meta content, cross-checked against the X-Robots-Tag header column elsewhere in the file. **Canonical_URL** is the declared canonical, accompanied by verdict columns that name concrete problems - missing, multiple, relative, cross-host, or pointing elsewhere. **Redirect_Count** and the redirect-chain column expose hop sequences that quietly waste crawl budget. Empty values are honest empties: a page with no meta robots tag records nothing there rather than inventing a default. ## What do the other eight datasets cover? OnPage_Elements reports the title with character, word and pixel widths plus a SERP-truncation flag, the meta description with the same treatment, heading counts from H1 to H6, a full heading outline with order violations and duplicates, an hreflang audit (self-reference, x-default, relative URLs, duplicate languages), and a social-meta census scoring Open Graph and Twitter card completeness. Content_Readability carries the six readability formulas, text-to-HTML ratio, reading time, top words, stuffing verdict, placeholder-text flags, the copyright year, and the duplicate-detection fingerprints. Links_Census counts internal, external and unique links, rel values including nofollow and sponsored, missing rel-noopener on new-tab links, generic and duplicate anchor text, mailto and tel links, insecure http targets on https pages, and deep internal paths. Media_Performance audits every image for alt coverage, missing width and height as a layout-shift risk, lazy loading, srcset coverage and modern-format share, then censuses scripts and stylesheets for render-blocking weight, third-party domains, embeds, and DOM size and depth with a TTFB verdict and a 0-100 performance score. Structured_Data parses JSON-LD, flags invalid blocks with their parse errors, lists schema types, and names rich-result eligibility. Accessibility_AXE grades the AXE run and lists violations by severity with their WCAG tags. ## Can it serve as a technical seo checker? Yes - indexability is where this bot earns that label. Per page it answers whether the URL is indexable and, if not, exactly which blocker is responsible: an HTTP error, a noindex from meta robots or the X-Robots-Tag header, a meta refresh, or a non-HTML content type. It records the redirect chain hop by hop, flags host, protocol, www, trailing-slash and case changes between requested and final URL, verifies canonical resolution, checks charset declaration against the header, confirms a mobile viewport that does not block scaling, and grades seven security headers - HSTS with max-age and preload, CSP including unsafe-inline and unsafe-eval, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy and COOP - into a letter grade, while flagging server-version and powered-by leaks. What it does not do is field performance: there is no CrUX or real-user data here, only lab measurements from your own machine. ## How does it compare to subscription audit services? Hosted audit products crawl your site, store the results in their cloud, and meter what you may look at by plan or by URL. This bot inverts all three: you decide the URL list, the data lands on your disk as plain CSVs, there is no per-run or per-URL charge, and the configuration file itself is yours to keep, version and re-run identically next quarter. The trade is real - you get no discovery crawl, no historical dashboard, no scheduled alerts and no pretty PDF. You get an auditable artifact and a repeatable measurement instead, which is what a diff-based workflow (this month's file versus last month's) actually needs. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the executable; SmartScreen may ask for confirmation on first launch, and "More info" then "Run anyway" proceeds if you trust the published checksum. On **macOS**, Gatekeeper may quarantine the download; right-click and "Open", or clear the flag with `xattr -d com.apple.quarantine /path/to/OnPageSEOAuditor`, then launch. On **Linux**, the launcher is a shell script: `chmod +x ./on-page-seo-auditor` then `./on-page-seo-auditor`. Before the first run, paste your URL list into the `urls_to_audit` setting (it is required - the run will not start without it) and adjust the four threshold variables only if your SERP pixel budget or thin-content definition differs from the defaults. A run is visible-browser by design (headless is off in the configuration), so you can watch pages load; for scheduled audits, point your operating system's scheduler - Task Scheduler, cron or launchd - at the launcher and accept that the browser window will appear. ## What do I check when a run returns zero rows? In order. If every file is empty: was `urls_to_audit` populated with full absolute URLs, and did navigation fail for all of them? Failed fetches are caught and written to the run log as comments, so the log tells you which URLs never loaded - a dead URL produces no row anywhere, by design. If only Critical_Issues is empty: that is the success case, no page tripped a hard check. If only Healthy_Pages is empty: your pages fail at least one strict criterion (most often title length outside 10-60 characters or a slow response over 3000 ms) - check the underlying datasets for which one. If Accessibility_AXE rows say the engine did not run: a content security policy, a blocking rule or an unreachable engine source prevented it, and the verdict column states that plainly rather than reporting a fabricated pass. The contract throughout is that failure is loud - zero rows means detection or navigation failed, never that problems were silently swallowed. ## What are the bot's limits? It audits the URLs you submit; it does not crawl, discover or follow links, and it does not parse sitemaps. It measures lab signals, not field data - no Core Web Vitals from real users, no rankings, no backlinks, no search-console-style impressions. It blocks stylesheets and images on the wire, so it cannot judge visual rendering, paint order or how the page actually looks. It does not fix anything: every dataset is a measurement, and remediation is your job in your codebase. It has no alerting, no history and no built-in scheduler - re-run the launcher yourself and diff the CSVs. The AXE step depends on its engine being loadable in the page context, and the configuration acknowledges that with explicit error handling. And it is a snapshotter: four workers over a big list generate sustained load, so it is not a tool to point at a fragile production server during peak hours. ## What is responsible on-page seo analysis? Audit only sites you own or have explicit permission to test - pointing any scanner at someone else's property without consent is their infrastructure decision to make, not yours. Keep lists proportionate: four concurrent workers is gentle by crawler standards but not free, and hammering a small site is indistinguishable from a bad day of traffic. Remember what the content census captures: emails and phone numbers displayed on audited pages are recorded in Content_Readability columns, and for sole traders those are personal data under GDPR, UK GDPR and equivalents - store the outputs accordingly, honour removal requests, and never turn an audit file into an outreach list. Sensitive-comment detection (stray tokens or debug notes in HTML comments) exists to help you fix leaks on your own pages, not to hunt for them on other people's. The acceptable-use policy applies in full before your first run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | urls_to_audit | list | yes | https://rtila.com/ https://example.com/ | Declared in project settings. | | title_pixel_limit | text | no | 580 | Declared in project settings. | | desc_pixel_limit | text | no | 990 | Declared in project settings. | | low_content_words | text | no | 300 | Declared in project settings. | | max_outbound_links | text | no | 100 | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **Response_Indexability** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Page_URL | page_url | no | | | Final_URL | text | no | variable=last_final_url | | Status_Code | text | no | variable=last_http_status | | Status_Text | text | no | variable=last_http_status_text | | Navigation_OK | text | no | variable=last_navigation_ok | | Navigation_Error | text | no | variable=last_navigation_error | | Response_Time_ms | text | no | variable=last_response_time_ms | | HTTP_Protocol | text | no | variable=last_protocol | | Server_Header | text | no | variable=last_server | | Content_Type | text | no | variable=last_content_type | | Charset_Header | text | no | variable=last_charset | | Content_Length | text | no | variable=last_content_length | | Cache_Control | text | no | variable=last_cache_control | | ETag | text | no | variable=last_etag | | Last_Modified | text | no | variable=last_last_modified | | X_Robots_Tag | text | no | variable=last_x_robots_tag | | Redirect_Count | text | no | variable=last_redirect_count | | Set_Cookie_Names | text | no | variable=last_set_cookie_names | | Redirect_Chain | text | no | variable=last_redirect_chain | | All_Response_Headers | text | no | variable=last_response_headers | | Status_Class | text | no | variable=last_http_status | | Is_Success | text | no | variable=last_http_status | | Is_Redirect | text | no | variable=last_http_status | | Is_Client_Error | text | no | variable=last_http_status | | Is_Server_Error | text | no | variable=last_http_status | | URL_Length | page_url | no | | | URL_Depth | page_url | no | | | URL_Protocol | page_url | no | | | URL_Host | page_url | no | | | URL_Path | page_url | no | | | URL_Query_String | page_url | no | | | URL_Fragment | page_url | no | | | URL_Param_Count | page_url | no | | | URL_Param_Names | page_url | no | | | URL_Has_Parameters | page_url | no | | | URL_Has_Fragment | page_url | no | | | URL_Has_Session_ID | page_url | no | | | URL_Tracking_Params | page_url | no | | | URL_File_Extension | page_url | no | | | URL_Is_Non_HTML | page_url | no | | | URL_Word_Count | page_url | no | | | URL_Uses_Underscores | page_url | no | | | URL_Has_Uppercase | page_url | no | | | URL_Is_Secure | page_url | no | | | URL_Is_Pretty | page_url | no | | | Meta_Robots | attribute | no | css=meta[name='robots'] | | Meta_Robots_Count | count | no | css=meta[name='robots'] | | Meta_Robots_All | text | no | css=meta[name='robots'] | | Canonical_URL | attribute | no | css=link[rel='canonical'] | | Canonical_Count | count | no | css=link[rel='canonical'] | | Meta_Refresh | attribute | no | css=meta[http-equiv='refresh' i] | | Base_Href | attribute | no | css=base[href] | | HTML_Lang | attribute | no | css=html | | Charset_Meta | attribute | no | css=meta[charset] | | Viewport_Content | attribute | no | css=meta[name='viewport'] | | Indexability | list | no | css=self | | Security | list | no | css=self | **OnPage_Elements** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Page_URL | page_url | no | | | Title | text | no | css=head title | | Title_Count | count | no | css=head title | | Meta_Description | attribute | no | css=meta[name='description'] | | Meta_Description_Count | count | no | css=meta[name='description'] | | Meta_Keywords | attribute | no | css=meta[name='keywords'] | | Meta_Author | attribute | no | css=meta[name='author'] | | Meta_Robots | text | no | css=meta[name='robots'] | | Meta_Google_Site_Verification | attribute | no | css=meta[name='google-site-verification'] | | Generator | attribute | no | css=meta[name='generator'] | | Favicon | attribute | no | css=link[rel='icon'],link[rel='shortcut icon'],link[rel='apple-touch-icon'] | | Favicon_Count | count | no | css=link[rel='icon'],link[rel='shortcut icon'],link[rel='apple-touch-icon'] | | Theme_Color | attribute | no | css=meta[name='theme-color'] | | Manifest | attribute | no | css=link[rel='manifest'] | | AMP_Link | attribute | no | css=link[rel='amphtml'] | | Is_AMP_Page | count | no | css=html[amp],html[⚡] | | Rel_Next | attribute | no | css=link[rel='next'] | | Rel_Prev | attribute | no | css=link[rel='prev'] | | Total_Meta_Tags | count | no | css=head meta:not([name^='rtila-']) | | Total_Link_Tags | count | no | css=head link | | Title_Chars | text | no | css=head title | | Title_Words | text | no | css=head title | | Title_Pixels | text | no | css=head title | | Title_SERP_Truncated | text | no | css=head title | | Title_Verdict | text | no | css=head title | | Description_Chars | attribute | no | css=meta[name='description'] | | Description_Words | attribute | no | css=meta[name='description'] | | Description_Pixels | attribute | no | css=meta[name='description'] | | Description_SERP_Truncated | attribute | no | css=meta[name='description'] | | Description_Verdict | attribute | no | css=meta[name='description'] | | H1_Count | count | no | css=h1 | | H1_Text | text | no | css=h1 | | H2_Count | count | no | css=h2 | | H3_Count | count | no | css=h3 | | H4_Count | count | no | css=h4 | | H5_Count | count | no | css=h5 | | H6_Count | count | no | css=h6 | | Heading_Count_Total | count | no | css=h1,h2,h3,h4,h5,h6 | | Hreflang_Count | count | no | css=link[rel='alternate'][hreflang] | | Hreflang_Raw | attribute | no | css=link[rel='alternate'][hreflang] | | Heading_Structure | list | no | css=h1,h2,h3,h4,h5,h6 | | Hreflang_Audit | list | no | css=link[rel='alternate'][hreflang] | | Social_Meta | list | no | css=self | **Content_Readability** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Page_URL | page_url | no | | | Paragraph_Count | count | no | css=body p | | List_Count | count | no | css=body ul,ol | | Table_Count | count | no | css=body table | | Blockquote_Count | count | no | css=body blockquote | | Main_Landmark | count | no | css=main,[role='main'],article | | Content_Metrics | list | no | css=self | **Links_Census** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Page_URL | page_url | no | | | Total_Anchors | count | no | css=a | | Anchors_NoHref | count | no | css=a:not([href]) | | Link_Audit | list | no | css=a | **Media_Performance** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Page_URL | page_url | no | | | DOM_Node_Count | count | no | css=* | | Img_Count | count | no | css=img | | Script_Count | count | no | css=script | | Stylesheet_Count | count | no | css=link[rel='stylesheet'] | | Iframe_Count | count | no | css=iframe | | Form_Count | count | no | css=form | | Input_Count | count | no | css=input,textarea,select | | Button_Count | count | no | css=button | | SVG_Count | count | no | css=svg | | Noscript_Count | count | no | css=noscript | | Inline_Style_Attrs | count | no | css=[style] | | Image_Audit | list | no | css=img | | Resource_Audit | list | no | css=script,link[rel='stylesheet'] | | Embed_Audit | list | no | css=iframe,video,audio,embed,object | | DOM_Performance | list | no | css=self | **Structured_Data** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Page_URL | page_url | no | | | JsonLD_Count | count | no | css=script[type='application/ld+json'] | | Microdata_Count | count | no | css=[itemtype] | | RDFa_Count | count | no | css=[typeof] | | Tech_Stack | text | no | variable=tech_stack | | Tech_Stack_Count | text | no | variable=tech_stack | | Structured_Data_Audit | list | no | css=script[type='application/ld+json'] | | Microdata_RDFa | list | no | css=self | **Accessibility_AXE** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Page_URL | page_url | no | | | AXE_Violations | text | no | variable=axe_report | | AXE_Passes | text | no | variable=axe_report | | AXE_Incomplete | text | no | variable=axe_report | | AXE_Inapplicable | text | no | variable=axe_report | | AXE_Summary | list | no | css=self | | DOM_Accessibility | list | no | css=[id] | | Form_Accessibility | list | no | css=self | **Critical_Issues** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | URL | page_url | no | | | Status_Code | text | no | variable=last_http_status | | Title | text | no | css=head title | | Title_Count | count | no | css=head title | | Description_Count | count | no | css=meta[name='description'] | | H1_Count | count | no | css=h1 | | Canonical_Count | count | no | css=link[rel='canonical'] | | Meta_Robots | text | no | css=meta[name='robots'] | | X_Robots_Tag | text | no | variable=last_x_robots_tag | | Viewport_Count | count | no | css=meta[name='viewport'] | | Img_Total | count | no | css=img | | Img_Missing_Alt | count | no | css=img:not([alt]) | | JsonLD_Count | count | no | css=script[type='application/ld+json'] | | DOM_Node_Count | count | no | css=* | **Healthy_Pages** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | URL | page_url | no | | | Status_Code | text | no | variable=last_http_status | | Title | text | no | css=head title | | Title_Chars | text | no | css=head title | | H1_Count | count | no | css=h1 | | Canonical_Count | count | no | css=link[rel='canonical'] | | Response_Time_ms | text | no | variable=last_response_time_ms | Export formats: csv. ### Downloads - Windows x86_64: `BOT_on-page-seo-auditor.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/on-page-seo-auditor/windows - macOS Apple Silicon: `BOT_on-page-seo-auditor.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/on-page-seo-auditor/macos - macOS Intel: `BOT_on-page-seo-auditor.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/on-page-seo-auditor/macos_intel - Linux x86_64: `BOT_on-page-seo-auditor.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/on-page-seo-auditor/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=on-page-seo-auditor` Configuration document: `https://marketplace.rtila.com/api/bots/on-page-seo-auditor/config.json` - Encoding: plain - Digest: fcba8379586e34cd2b9564a664ae92917d9529ba32604229dabe23f9c0d6e430 - Bytes: 146424 ```json { "name": "On-Page SEO Auditor", "settings": { "urls": [ "${urls_to_audit}" ], "headless": false, "max_concurrent_workers": 4, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": true, "variables": [ { "name": "urls_to_audit", "value": "https://rtila.com/\nhttps://example.com/", "required": true, "is_list": true }, { "name": "title_pixel_limit", "value": "580", "required": false, "is_list": false }, { "name": "desc_pixel_limit", "value": "990", "required": false, "is_list": false }, { "name": "low_content_words", "value": "300", "required": false, "is_list": false }, { "name": "max_outbound_links", "value": "100", "required": false, "is_list": false } ] }, "datasets": { "Response_Indexability": { "item_selector": "css=html", "properties": [ { "name": "Page_URL", "type": "page_url", "selector": "" }, { "name": "Final_URL", "type": "text", "selector": "variable=last_final_url" }, { "name": "Status_Code", "type": "text", "selector": "variable=last_http_status" }, { "name": "Status_Text", "type": "text", "selector": "variable=last_http_status_text" }, { "name": "Navigation_OK", "type": "text", "selector": "variable=last_navigation_ok" }, { "name": "Navigation_Error", "type": "text", "selector": "variable=last_navigation_error" }, { "name": "Response_Time_ms", "type": "text", "selector": "variable=last_response_time_ms" }, { "name": "HTTP_Protocol", "type": "text", "selector": "variable=last_protocol" }, { "name": "Server_Header", "type": "text", "selector": "variable=last_server" }, { "name": "Content_Type", "type": "text", "selector": "variable=last_content_type" }, { "name": "Charset_Header", "type": "text", "selector": "variable=last_charset" }, { "name": "Content_Length", "type": "text", "selector": "variable=last_content_length" }, { "name": "Cache_Control", "type": "text", "selector": "variable=last_cache_control" }, { "name": "ETag", "type": "text", "selector": "variable=last_etag" }, { "name": "Last_Modified", "type": "text", "selector": "variable=last_last_modified" }, { "name": "X_Robots_Tag", "type": "text", "selector": "variable=last_x_robots_tag" }, { "name": "Redirect_Count", "type": "text", "selector": "variable=last_redirect_count" }, { "name": "Set_Cookie_Names", "type": "text", "selector": "variable=last_set_cookie_names", "transformations": [ { "type": "script", "code": "if(!value)return '';if(Array.isArray(value))return value.join(', ');if(typeof value==='object')return Object.keys(value).join(', ');return String(value);", "default_value": "" } ] }, { "name": "Redirect_Chain", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "if(!Array.isArray(value)||!value.length)return '';return value.map(function(h){return (h&&h.status?h.status:'?')+' '+(h&&h.url?h.url:'')+(h&&h.location?' -> '+h.location:'');}).join(' | ');", "default_value": "" } ] }, { "name": "All_Response_Headers", "type": "text", "selector": "variable=last_response_headers", "transformations": [ { "type": "script", "code": "if(!value)return '';try{return JSON.stringify(value);}catch(e){return String(value);}", "default_value": "" } ] }, { "name": "Status_Class", "type": "text", "selector": "variable=last_http_status", "transformations": [ { "type": "script", "code": "var n=Number(value);if(!isFinite(n)||n===0)return '0xx No Response';if(n<200)return '1xx Informational';if(n<300)return '2xx Success';if(n<400)return '3xx Redirect';if(n<500)return '4xx Client Error';return '5xx Server Error';", "default_value": "0xx No Response" } ] }, { "name": "Is_Success", "type": "text", "selector": "variable=last_http_status", "transformations": [ { "type": "script", "code": "var n=Number(value);return String(isFinite(n)&&n>=200&&n<300);", "default_value": "false" } ] }, { "name": "Is_Redirect", "type": "text", "selector": "variable=last_http_status", "transformations": [ { "type": "script", "code": "var n=Number(value);return String(isFinite(n)&&n>=300&&n<400);", "default_value": "false" } ] }, { "name": "Is_Client_Error", "type": "text", "selector": "variable=last_http_status", "transformations": [ { "type": "script", "code": "var n=Number(value);return String(isFinite(n)&&n>=400&&n<500);", "default_value": "false" } ] }, { "name": "Is_Server_Error", "type": "text", "selector": "variable=last_http_status", "transformations": [ { "type": "script", "code": "var n=Number(value);return String(isFinite(n)&&n>=500);", "default_value": "false" } ] }, { "name": "URL_Length", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "return String(String(value).length);", "default_value": "0" } ] }, { "name": "URL_Depth", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return String(new URL(String(value)).pathname.split('/').filter(function(s){return s.length>0;}).length);}catch(e){return '0';}", "default_value": "0" } ] }, { "name": "URL_Protocol", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return new URL(String(value)).protocol.replace(':','');}catch(e){return '';}", "default_value": "" } ] }, { "name": "URL_Host", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return new URL(String(value)).hostname;}catch(e){return '';}", "default_value": "" } ] }, { "name": "URL_Path", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return new URL(String(value)).pathname;}catch(e){return '';}", "default_value": "" } ] }, { "name": "URL_Query_String", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return new URL(String(value)).search.replace(/^\\?/,'');}catch(e){return '';}", "default_value": "" } ] }, { "name": "URL_Fragment", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return new URL(String(value)).hash.replace(/^#/,'');}catch(e){return '';}", "default_value": "" } ] }, { "name": "URL_Param_Count", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return String(Array.from(new URL(String(value)).searchParams.keys()).length);}catch(e){return '0';}", "default_value": "0" } ] }, { "name": "URL_Param_Names", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return Array.from(new URL(String(value)).searchParams.keys()).join(', ');}catch(e){return '';}", "default_value": "" } ] }, { "name": "URL_Has_Parameters", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return String(new URL(String(value)).search.length>1);}catch(e){return 'false';}", "default_value": "false" } ] }, { "name": "URL_Has_Fragment", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return String(new URL(String(value)).hash.length>1);}catch(e){return 'false';}", "default_value": "false" } ] }, { "name": "URL_Has_Session_ID", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "return String(/(?:^|[?&;/])(phpsessid|jsessionid|aspsessionid[a-z]*|sid|sessionid|session_id|cfid|cftoken)(?:=|;|$)/i.test(String(value)));", "default_value": "false" } ] }, { "name": "URL_Tracking_Params", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "var m=String(value).match(/[?&](utm_[a-z]+|gclid|fbclid|msclkid|dclid|gbraid|wbraid|twclid|igshid|mc_[a-z]+|yclid|_ga|_hs[a-z]+|vero_[a-z]+|ref|source|medium|campaign)(?==|&|$)/gi);if(!m)return '';var out=[];for(var i=0;i1;}).length);}catch(e){return '0';}", "default_value": "0" } ] }, { "name": "URL_Uses_Underscores", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return String(new URL(String(value)).pathname.indexOf('_')!==-1);}catch(e){return 'false';}", "default_value": "false" } ] }, { "name": "URL_Has_Uppercase", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{var p=new URL(String(value)).pathname;return String(p!==p.toLowerCase());}catch(e){return 'false';}", "default_value": "false" } ] }, { "name": "URL_Is_Secure", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{return String(new URL(String(value)).protocol==='https:');}catch(e){return 'false';}", "default_value": "false" } ] }, { "name": "URL_Is_Pretty", "type": "page_url", "selector": "", "transformations": [ { "type": "script", "code": "try{var u=new URL(String(value));return String(u.search.length<=1 && !/\\.[a-z0-9]{2,5}$/i.test(u.pathname) && u.pathname===u.pathname.toLowerCase());}catch(e){return 'false';}", "default_value": "false" } ] }, { "name": "Meta_Robots", "type": "attribute", "selector": "css=meta[name='robots']", "attribute": "content" }, { "name": "Meta_Robots_Count", "type": "count", "selector": "css=meta[name='robots']" }, { "name": "Meta_Robots_All", "type": "text", "selector": "css=meta[name='robots']" }, { "name": "Canonical_URL", "type": "attribute", "selector": "css=link[rel='canonical']", "attribute": "href" }, { "name": "Canonical_Count", "type": "count", "selector": "css=link[rel='canonical']" }, { "name": "Meta_Refresh", "type": "attribute", "selector": "css=meta[http-equiv='refresh' i]", "attribute": "content" }, { "name": "Base_Href", "type": "attribute", "selector": "css=base[href]", "attribute": "href" }, { "name": "HTML_Lang", "type": "attribute", "selector": "css=html", "attribute": "lang" }, { "name": "Charset_Meta", "type": "attribute", "selector": "css=meta[charset]", "attribute": "charset" }, { "name": "Viewport_Content", "type": "attribute", "selector": "css=meta[name='viewport']", "attribute": "content" }, { "name": "Indexability", "type": "list", "selector": "css=self", "transformations": [ { "type": "script", "code": "var it=(Array.isArray(value)&&value.length)?value[0]:{};function s(x){return (x===null||x===undefined)?\"\":String(x);}function lo(x){return s(x).toLowerCase();}function n(x){var v=parseInt(x,10);return isFinite(v)?v:0;}var pageUrl=s(it.page_url),finalUrl=s(it.final_url)||pageUrl;var statusNum=(it.status===null||it.status===undefined||it.status===\"\")?0:Number(it.status);if(!isFinite(statusNum))statusNum=0;var pu=null,fu=null;try{pu=new URL(pageUrl);}catch(e){}try{fu=new URL(finalUrl);}catch(e){}var metaRaw=s(it.metarobots),xRaw=s(it.xrobots);var combined=(metaRaw+\", \"+xRaw).toLowerCase();var has=function(d){return combined.indexOf(d)!==-1;};var directives=[];combined.split(\",\").forEach(function(d){d=d.trim();if(d)directives.push(d);});function dv(name){for(var i=0;i=400)blockers.push(\"HTTP \"+statusNum);if(noindexSources.length)blockers.push(\"noindex (\"+noindexSources.join(\" + \")+\")\");if(s(it.metarefresh).trim()!==\"\")blockers.push(\"meta refresh\");if(!isHtml)blockers.push(\"Non-HTML content type: \"+ct);var indexable=blockers.length===0;var canonRaw=s(it.canonical).trim(),canonCount=n(it.canonical_n);var canonAbs=/^https?:\\/\\//i.test(canonRaw),canonUrl=\"\",cu=null;if(canonRaw){try{cu=new URL(canonRaw,pageUrl);canonUrl=cu.href;}catch(e){canonUrl=canonRaw;}}var canonSelf=(cu&&pu)?(cu.origin===pu.origin&&cu.pathname===pu.pathname&&cu.search===pu.search):false;var canonFinal=(cu&&fu)?(cu.origin===fu.origin&&cu.pathname===fu.pathname&&cu.search===fu.search):false;var canonIssues=[];if(!canonRaw)canonIssues.push(\"Missing\");if(canonCount>1)canonIssues.push(\"Multiple (\"+canonCount+\")\");if(canonRaw&&!canonAbs)canonIssues.push(\"Relative\");if(canonRaw&&cu&&pu&&cu.protocol!==pu.protocol)canonIssues.push(\"Scheme mismatch\");if(canonRaw&&cu&&pu&&cu.hostname!==pu.hostname)canonIssues.push(\"Cross-host\");if(canonRaw&&cu&&cu.search.indexOf(\"=\")!==-1)canonIssues.push(\"Has parameters\");if(canonRaw&&!canonSelf&&!canonFinal)canonIssues.push(\"Points elsewhere\");var mr=s(it.metarefresh).trim(),mrDelay=null,mrTarget=\"\";if(mr){var m1=mr.match(/^\\s*(\\d+)\\s*;\\s*url\\s*=\\s*(.+)$/i);if(m1){mrDelay=parseInt(m1[1],10);mrTarget=m1[2].trim().replace(/^['\"]|['\"]$/g,\"\");}else{var m2=mr.match(/^\\s*(\\d+)/);if(m2)mrDelay=parseInt(m2[1],10);}}var hostChanged=(pu&&fu)?(pu.hostname!==fu.hostname):false;var protoUp=(pu&&fu)?(pu.protocol!==fu.protocol):false;var wwwChanged=(pu&&fu)?(pu.hostname.replace(/^www\\./,\"\")!==fu.hostname.replace(/^www\\./,\"\")):false;var slashChanged=(pu&&fu)?((pu.pathname.slice(-1)===\"/\")!==(fu.pathname.slice(-1)===\"/\")):false;var caseChanged=(pu&&fu)?(pu.pathname!==fu.pathname&&pu.pathname.toLowerCase()===fu.pathname.toLowerCase()):false;var pathChanged=(pu&&fu)?(pu.pathname!==fu.pathname):false;var paramsChanged=(pu&&fu)?(pu.search!==fu.search):false;var metaCharset=lo(it.charset_meta),hdrCharset=lo(it.charset_hdr);var charsetMatch=(!metaCharset||!hdrCharset)?null:(metaCharset===hdrCharset);var vp=s(it.viewport).trim();var vpPresent=vp!==\"\";var vpBlocks=vpPresent&&(vp.indexOf(\"user-scalable=no\")!==-1||/maximum-scale\\s*=\\s*1(\\.0)?\\b/.test(vp));var cookies=Array.isArray(it.cookies)?it.cookies:[];var cookieStr=cookies.join(\",\").toLowerCase();return {indexable:String(indexable),indexability_verdict:indexable?\"Indexable\":\"Non-Indexable\",indexability_blockers:blockers.join(\"; \"),noindex:String(noindex),noindex_sources:noindexSources.join(\", \"),nofollow:String(nofollow),noarchive:String(has(\"noarchive\")),nosnippet:String(has(\"nosnippet\")),noimageindex:String(has(\"noimageindex\")),max_snippet:dv(\"max-snippet\"),max_image_preview:dv(\"max-image-preview\"),max_video_preview:dv(\"max-video-preview\"),unavailable_after:dv(\"unavailable_after\"),robots_directives_all:directives.join(\", \"),canonical_is_self:String(canonSelf),canonical_is_absolute:String(canonAbs),canonical_resolved:canonUrl,canonical_issues:canonIssues.join(\"; \"),canonical_verdict:canonIssues.length===0?\"OK\":canonIssues.join(\"; \"),host_changed:String(hostChanged),protocol_upgraded:String(protoUp),www_changed:String(wwwChanged),trailing_slash_changed:String(slashChanged),case_changed:String(caseChanged),path_changed:String(pathChanged),params_changed:String(paramsChanged),meta_refresh_present:String(mr!==\"\"),meta_refresh_delay:mrDelay===null?\"\":String(mrDelay),meta_refresh_target:mrTarget,content_type_is_html:String(isHtml),is_pdf:String(isPdf),is_document_resource:String(isDoc),charset_matches_header:charsetMatch===null?\"\":String(charsetMatch),charset_declared:String(metaCharset!==\"\"||hdrCharset!==\"\"),viewport_present:String(vpPresent),viewport_blocks_scaling:String(vpBlocks),is_mobile_friendly:String(vpPresent&&!vpBlocks),html_lang_present:String(s(it.html_lang).trim()!==\"\"),cookie_count:String(cookies.length),has_analytics_cookie:String(/_ga|ga_|_gid|_utm|_fbp|_fbc|_hj/.test(cookieStr)),has_session_cookie:String(/sess|jsessionid|phpsessid|sid|cfid|cftoken|asp\\.net_sessionid/.test(cookieStr))};" } ], "properties": [ { "name": "page_url", "type": "page_url", "selector": "" }, { "name": "final_url", "type": "text", "selector": "variable=last_final_url" }, { "name": "status", "type": "text", "selector": "variable=last_http_status" }, { "name": "xrobots", "type": "text", "selector": "variable=last_x_robots_tag" }, { "name": "content_type", "type": "text", "selector": "variable=last_content_type" }, { "name": "charset_hdr", "type": "text", "selector": "variable=last_charset" }, { "name": "protocol", "type": "text", "selector": "variable=last_protocol" }, { "name": "metarobots", "type": "text", "selector": "css=meta[name='robots']" }, { "name": "canonical", "type": "attribute", "selector": "css=link[rel='canonical']", "attribute": "href" }, { "name": "canonical_n", "type": "count", "selector": "css=link[rel='canonical']" }, { "name": "metarefresh", "type": "attribute", "selector": "css=meta[http-equiv='refresh' i]", "attribute": "content" }, { "name": "charset_meta", "type": "attribute", "selector": "css=meta[charset]", "attribute": "charset" }, { "name": "viewport", "type": "attribute", "selector": "css=meta[name='viewport']", "attribute": "content" }, { "name": "html_lang", "type": "attribute", "selector": "css=html", "attribute": "lang" }, { "name": "cookies", "type": "text", "selector": "variable=last_set_cookie_names" }, { "name": "indexable", "type": "index" }, { "name": "indexability_verdict", "type": "index" }, { "name": "indexability_blockers", "type": "index" }, { "name": "noindex", "type": "index" }, { "name": "noindex_sources", "type": "index" }, { "name": "nofollow", "type": "index" }, { "name": "noarchive", "type": "index" }, { "name": "nosnippet", "type": "index" }, { "name": "noimageindex", "type": "index" }, { "name": "max_snippet", "type": "index" }, { "name": "max_image_preview", "type": "index" }, { "name": "max_video_preview", "type": "index" }, { "name": "unavailable_after", "type": "index" }, { "name": "robots_directives_all", "type": "index" }, { "name": "canonical_is_self", "type": "index" }, { "name": "canonical_is_absolute", "type": "index" }, { "name": "canonical_resolved", "type": "index" }, { "name": "canonical_issues", "type": "index" }, { "name": "canonical_verdict", "type": "index" }, { "name": "host_changed", "type": "index" }, { "name": "protocol_upgraded", "type": "index" }, { "name": "www_changed", "type": "index" }, { "name": "trailing_slash_changed", "type": "index" }, { "name": "case_changed", "type": "index" }, { "name": "path_changed", "type": "index" }, { "name": "params_changed", "type": "index" }, { "name": "meta_refresh_present", "type": "index" }, { "name": "meta_refresh_delay", "type": "index" }, { "name": "meta_refresh_target", "type": "index" }, { "name": "content_type_is_html", "type": "index" }, { "name": "is_pdf", "type": "index" }, { "name": "is_document_resource", "type": "index" }, { "name": "charset_matches_header", "type": "index" }, { "name": "charset_declared", "type": "index" }, { "name": "viewport_present", "type": "index" }, { "name": "viewport_blocks_scaling", "type": "index" }, { "name": "is_mobile_friendly", "type": "index" }, { "name": "html_lang_present", "type": "index" }, { "name": "cookie_count", "type": "index" }, { "name": "has_analytics_cookie", "type": "index" }, { "name": "has_session_cookie", "type": "index" } ] }, { "name": "Security", "type": "list", "selector": "css=self", "transformations": [ { "type": "script", "code": "var it=(Array.isArray(value)&&value.length)?value[0]:{};var h=(it.headers&&typeof it.headers===\"object\")?it.headers:{};function g(n){var v=h[n];return (typeof v===\"string\")?v:\"\";}function lo(x){return String(x).toLowerCase();}var hsts=g(\"strict-transport-security\");var hm=hsts.match(/max-age\\s*=\\s*(\\d+)/i);var hstsDays=hm?Math.round(parseInt(hm[1],10)/86400):0;var csp=g(\"content-security-policy\")||g(\"content-security-policy-report-only\");var xfo=g(\"x-frame-options\");var xcto=g(\"x-content-type-options\");var rp=g(\"referrer-policy\");var pp=g(\"permissions-policy\")||g(\"feature-policy\");var xxp=g(\"x-xss-protection\");var coop=g(\"cross-origin-opener-policy\");var coep=g(\"cross-origin-embedder-policy\");var corp=g(\"cross-origin-resource-policy\");var srv=g(\"server\");var leak=/\\d+\\.\\d+/.test(srv)?srv:\"\";var powered=g(\"x-powered-by\")||g(\"x-aspnet-version\")||g(\"x-aspnetmvc-version\");var present=[],missing=[];if(hsts)present.push(\"HSTS\");else missing.push(\"Strict-Transport-Security\");if(csp)present.push(\"CSP\");else missing.push(\"Content-Security-Policy\");if(xfo)present.push(\"X-Frame-Options\");else missing.push(\"X-Frame-Options\");if(lo(xcto)===\"nosniff\")present.push(\"X-Content-Type-Options\");else missing.push(\"X-Content-Type-Options\");if(rp)present.push(\"Referrer-Policy\");else missing.push(\"Referrer-Policy\");if(pp)present.push(\"Permissions-Policy\");else missing.push(\"Permissions-Policy\");if(coop)present.push(\"COOP\");else missing.push(\"Cross-Origin-Opener-Policy\");var score=Math.round((present.length/7)*100);var grade=score>=90?\"A\":score>=75?\"B\":score>=55?\"C\":score>=35?\"D\":\"F\";var pageUrl=String(it.page_url||\"\");var isHttps=/^https:/i.test(pageUrl);return {hsts:hsts,hsts_max_age_days:String(hstsDays),hsts_includesubdomains:String(hsts.toLowerCase().indexOf(\"includesubdomains\")!==-1),hsts_preload:String(hsts.toLowerCase().indexOf(\"preload\")!==-1),csp:csp,csp_unsafe_inline:String(csp.indexOf(\"unsafe-inline\")!==-1),csp_unsafe_eval:String(csp.indexOf(\"unsafe-eval\")!==-1),x_frame_options:xfo,x_content_type_options:xcto,referrer_policy:rp,permissions_policy:pp,x_xss_protection:xxp,coop:coop,coep:coep,corp:corp,server_version_leak:leak,powered_by_leak:powered,headers_present:present.join(\", \"),headers_missing:missing.join(\", \"),security_score:String(score),security_grade:grade,mixed_content_risk:String(isHttps?\"Check outlinks for http:// resources\":\"Page served over HTTP\"),header_count:String(Object.keys(h).length)};" } ], "properties": [ { "name": "headers", "type": "text", "selector": "variable=last_response_headers" }, { "name": "page_url", "type": "page_url", "selector": "" }, { "name": "hsts", "type": "index" }, { "name": "hsts_max_age_days", "type": "index" }, { "name": "hsts_includesubdomains", "type": "index" }, { "name": "hsts_preload", "type": "index" }, { "name": "csp", "type": "index" }, { "name": "csp_unsafe_inline", "type": "index" }, { "name": "csp_unsafe_eval", "type": "index" }, { "name": "x_frame_options", "type": "index" }, { "name": "x_content_type_options", "type": "index" }, { "name": "referrer_policy", "type": "index" }, { "name": "permissions_policy", "type": "index" }, { "name": "x_xss_protection", "type": "index" }, { "name": "coop", "type": "index" }, { "name": "coep", "type": "index" }, { "name": "corp", "type": "index" }, { "name": "server_version_leak", "type": "index" }, { "name": "powered_by_leak", "type": "index" }, { "name": "headers_present", "type": "index" }, { "name": "headers_missing", "type": "index" }, { "name": "security_score", "type": "index" }, { "name": "security_grade", "type": "index" }, { "name": "mixed_content_risk", "type": "index" }, { "name": "header_count", "type": "index" } ] } ], "required_fields": [ "Page_URL" ] }, "OnPage_Elements": { "item_selector": "css=html", "properties": [ { "name": "Page_URL", "type": "page_url", "selector": "" }, { "name": "Title", "type": "text", "selector": "css=head title", "transformations": [ { "type": "trim" } ] }, { "name": "Title_Count", "type": "count", "selector": "css=head title" }, { "name": "Meta_Description", "type": "attribute", "selector": "css=meta[name='description']", "attribute": "content" }, { "name": "Meta_Description_Count", "type": "count", "selector": "css=meta[name='description']" }, { "name": "Meta_Keywords", "type": "attribute", "selector": "css=meta[name='keywords']", "attribute": "content" }, { "name": "Meta_Author", "type": "attribute", "selector": "css=meta[name='author']", "attribute": "content" }, { "name": "Meta_Robots", "type": "text", "selector": "css=meta[name='robots']" }, { "name": "Meta_Google_Site_Verification", "type": "attribute", "selector": "css=meta[name='google-site-verification']", "attribute": "content" }, { "name": "Generator", "type": "attribute", "selector": "css=meta[name='generator']", "attribute": "content" }, { "name": "Favicon", "type": "attribute", "selector": "css=link[rel='icon'],link[rel='shortcut icon'],link[rel='apple-touch-icon']", "attribute": "href" }, { "name": "Favicon_Count", "type": "count", "selector": "css=link[rel='icon'],link[rel='shortcut icon'],link[rel='apple-touch-icon']" }, { "name": "Theme_Color", "type": "attribute", "selector": "css=meta[name='theme-color']", "attribute": "content" }, { "name": "Manifest", "type": "attribute", "selector": "css=link[rel='manifest']", "attribute": "href" }, { "name": "AMP_Link", "type": "attribute", "selector": "css=link[rel='amphtml']", "attribute": "href" }, { "name": "Is_AMP_Page", "type": "count", "selector": "css=html[amp],html[⚡]" }, { "name": "Rel_Next", "type": "attribute", "selector": "css=link[rel='next']", "attribute": "href" }, { "name": "Rel_Prev", "type": "attribute", "selector": "css=link[rel='prev']", "attribute": "href" }, { "name": "Total_Meta_Tags", "type": "count", "selector": "css=head meta:not([name^='rtila-'])" }, { "name": "Total_Link_Tags", "type": "count", "selector": "css=head link" }, { "name": "Title_Chars", "type": "text", "selector": "css=head title", "transformations": [ { "type": "script", "code": "return String(String(value==null?'':value).trim().length);", "default_value": "0" } ] }, { "name": "Title_Words", "type": "text", "selector": "css=head title", "transformations": [ { "type": "script", "code": "var t=String(value==null?'':value).trim();return String(t?t.split(/\\s+/).length:0);", "default_value": "0" } ] }, { "name": "Title_Pixels", "type": "text", "selector": "css=head title", "transformations": [ { "type": "script", "code": "var W={\" \":278,\"!\":278,\"\\\"\":355,\"#\":556,\"$\":556,\"%\":889,\"&\":667,\"'\":191,\"(\":333,\")\":333,\"*\":389,\"+\":584,\",\":278,\"-\":333,\".\":278,\"/\":278,\":\":278,\";\":278,\"<\":584,\"=\":584,\">\":584,\"?\":556,\"@\":1015,\"[\":278,\"\\\\\":278,\"]\":278,\"^\":469,\"_\":556,\"`\":333,\"{\":334,\"|\":260,\"}\":334,\"~\":584,\"A\":667,\"B\":667,\"C\":722,\"D\":722,\"E\":667,\"F\":611,\"G\":778,\"H\":722,\"I\":278,\"J\":500,\"K\":667,\"L\":556,\"M\":833,\"N\":722,\"O\":778,\"P\":667,\"Q\":778,\"R\":722,\"S\":667,\"T\":611,\"U\":722,\"V\":667,\"W\":944,\"X\":667,\"Y\":667,\"Z\":611,\"a\":556,\"b\":556,\"c\":500,\"d\":556,\"e\":556,\"f\":278,\"g\":556,\"h\":556,\"i\":222,\"j\":222,\"k\":500,\"l\":222,\"m\":833,\"n\":556,\"o\":556,\"p\":556,\"q\":556,\"r\":333,\"s\":500,\"t\":278,\"u\":556,\"v\":500,\"w\":722,\"x\":500,\"y\":500,\"z\":500}; for(var d=0;d<=9;d++){W[String(d)]=556;} var t=String(value==null?\"\":value).trim(); var u=0; for(var i=0;i\":584,\"?\":556,\"@\":1015,\"[\":278,\"\\\\\":278,\"]\":278,\"^\":469,\"_\":556,\"`\":333,\"{\":334,\"|\":260,\"}\":334,\"~\":584,\"A\":667,\"B\":667,\"C\":722,\"D\":722,\"E\":667,\"F\":611,\"G\":778,\"H\":722,\"I\":278,\"J\":500,\"K\":667,\"L\":556,\"M\":833,\"N\":722,\"O\":778,\"P\":667,\"Q\":778,\"R\":722,\"S\":667,\"T\":611,\"U\":722,\"V\":667,\"W\":944,\"X\":667,\"Y\":667,\"Z\":611,\"a\":556,\"b\":556,\"c\":500,\"d\":556,\"e\":556,\"f\":278,\"g\":556,\"h\":556,\"i\":222,\"j\":222,\"k\":500,\"l\":222,\"m\":833,\"n\":556,\"o\":556,\"p\":556,\"q\":556,\"r\":333,\"s\":500,\"t\":278,\"u\":556,\"v\":500,\"w\":722,\"x\":500,\"y\":500,\"z\":500}; for(var d=0;d<=9;d++){W[String(d)]=556;} var t=String(value==null?\"\":value).trim(); var u=0; for(var i=0;i 580);", "default_value": "false" } ] }, { "name": "Title_Verdict", "type": "text", "selector": "css=head title", "transformations": [ { "type": "script", "code": "var t=String(value==null?'':value).trim();if(!t)return 'MISSING';var L=t.length;if(L<10)return 'TOO SHORT';if(L>60)return 'TOO LONG';return 'OK';", "default_value": "MISSING" } ] }, { "name": "Description_Chars", "type": "attribute", "selector": "css=meta[name='description']", "attribute": "content", "transformations": [ { "type": "script", "code": "return String(String(value==null?'':value).trim().length);", "default_value": "0" } ] }, { "name": "Description_Words", "type": "attribute", "selector": "css=meta[name='description']", "attribute": "content", "transformations": [ { "type": "script", "code": "var t=String(value==null?'':value).trim();return String(t?t.split(/\\s+/).length:0);", "default_value": "0" } ] }, { "name": "Description_Pixels", "type": "attribute", "selector": "css=meta[name='description']", "attribute": "content", "transformations": [ { "type": "script", "code": "var W={\" \":278,\"!\":278,\"\\\"\":355,\"#\":556,\"$\":556,\"%\":889,\"&\":667,\"'\":191,\"(\":333,\")\":333,\"*\":389,\"+\":584,\",\":278,\"-\":333,\".\":278,\"/\":278,\":\":278,\";\":278,\"<\":584,\"=\":584,\">\":584,\"?\":556,\"@\":1015,\"[\":278,\"\\\\\":278,\"]\":278,\"^\":469,\"_\":556,\"`\":333,\"{\":334,\"|\":260,\"}\":334,\"~\":584,\"A\":667,\"B\":667,\"C\":722,\"D\":722,\"E\":667,\"F\":611,\"G\":778,\"H\":722,\"I\":278,\"J\":500,\"K\":667,\"L\":556,\"M\":833,\"N\":722,\"O\":778,\"P\":667,\"Q\":778,\"R\":722,\"S\":667,\"T\":611,\"U\":722,\"V\":667,\"W\":944,\"X\":667,\"Y\":667,\"Z\":611,\"a\":556,\"b\":556,\"c\":500,\"d\":556,\"e\":556,\"f\":278,\"g\":556,\"h\":556,\"i\":222,\"j\":222,\"k\":500,\"l\":222,\"m\":833,\"n\":556,\"o\":556,\"p\":556,\"q\":556,\"r\":333,\"s\":500,\"t\":278,\"u\":556,\"v\":500,\"w\":722,\"x\":500,\"y\":500,\"z\":500}; for(var d=0;d<=9;d++){W[String(d)]=556;} var t=String(value==null?\"\":value).trim(); var u=0; for(var i=0;i\":584,\"?\":556,\"@\":1015,\"[\":278,\"\\\\\":278,\"]\":278,\"^\":469,\"_\":556,\"`\":333,\"{\":334,\"|\":260,\"}\":334,\"~\":584,\"A\":667,\"B\":667,\"C\":722,\"D\":722,\"E\":667,\"F\":611,\"G\":778,\"H\":722,\"I\":278,\"J\":500,\"K\":667,\"L\":556,\"M\":833,\"N\":722,\"O\":778,\"P\":667,\"Q\":778,\"R\":722,\"S\":667,\"T\":611,\"U\":722,\"V\":667,\"W\":944,\"X\":667,\"Y\":667,\"Z\":611,\"a\":556,\"b\":556,\"c\":500,\"d\":556,\"e\":556,\"f\":278,\"g\":556,\"h\":556,\"i\":222,\"j\":222,\"k\":500,\"l\":222,\"m\":833,\"n\":556,\"o\":556,\"p\":556,\"q\":556,\"r\":333,\"s\":500,\"t\":278,\"u\":556,\"v\":500,\"w\":722,\"x\":500,\"y\":500,\"z\":500}; for(var d=0;d<=9;d++){W[String(d)]=556;} var t=String(value==null?\"\":value).trim(); var u=0; for(var i=0;i 990);", "default_value": "false" } ] }, { "name": "Description_Verdict", "type": "attribute", "selector": "css=meta[name='description']", "attribute": "content", "transformations": [ { "type": "script", "code": "var t=String(value==null?'':value).trim();if(!t)return 'MISSING';var L=t.length;if(L<50)return 'TOO SHORT';if(L>160)return 'TOO LONG';return 'OK';", "default_value": "MISSING" } ] }, { "name": "H1_Count", "type": "count", "selector": "css=h1" }, { "name": "H1_Text", "type": "text", "selector": "css=h1", "transformations": [ { "type": "trim" } ] }, { "name": "H2_Count", "type": "count", "selector": "css=h2" }, { "name": "H3_Count", "type": "count", "selector": "css=h3" }, { "name": "H4_Count", "type": "count", "selector": "css=h4" }, { "name": "H5_Count", "type": "count", "selector": "css=h5" }, { "name": "H6_Count", "type": "count", "selector": "css=h6" }, { "name": "Heading_Count_Total", "type": "count", "selector": "css=h1,h2,h3,h4,h5,h6" }, { "name": "Hreflang_Count", "type": "count", "selector": "css=link[rel='alternate'][hreflang]" }, { "name": "Hreflang_Raw", "type": "attribute", "selector": "css=link[rel='alternate'][hreflang]", "attribute": "hreflang" }, { "name": "Heading_Structure", "type": "list", "selector": "css=h1,h2,h3,h4,h5,h6", "transformations": [ { "type": "script", "code": "var rows=Array.isArray(value)?value:[];var counts={1:0,2:0,3:0,4:0,5:0,6:0};var outline=[],orderIssues=[],emptyCount=0,longCount=0,seen={},dupes=[];var h1First=\"\",headingWords=0,firstLevel=0,maxLevel=0;for(var i=0;i6)continue;counts[lvl]++;if(!firstLevel)firstLevel=lvl;if(lvl>maxLevel)maxLevel=lvl;var txt=String(rows[i].text||\"\").replace(/\\s+/g,\" \").trim();if(txt===\"\")emptyCount++;if(txt.length>120)longCount++;headingWords+=txt?txt.split(/\\s+/).length:0;outline.push(tag.toUpperCase()+\": \"+(txt.substring(0,90)||(txt?\"\":\"(empty)\")));if(seen[txt]&&txt!==\"\")dupes.push(txt.substring(0,60));else seen[txt]=1;if(i>0){var prev=parseInt(String(rows[i-1].tag||\"h0\").toLowerCase().replace(\"h\",\"\"),10);if(isFinite(prev)&&lvl-prev>1)orderIssues.push(\"H\"+prev+\" -> H\"+lvl+\" at #\"+(i+1));}}var h1Texts=[];for(var j=0;j1)verdict.push(\"Multiple H1 (\"+counts[1]+\")\");if(orderIssues.length)verdict.push(orderIssues.length+\" order violation(s)\");if(emptyCount)verdict.push(emptyCount+\" empty heading(s)\");if(dupes.length)verdict.push(dupes.length+\" duplicate heading(s)\");return {outline:outline.join(\" | \").substring(0,8000),h1_count:String(counts[1]),h1_missing:String(counts[1]===0),h1_multiple:String(counts[1]>1),h1_first_text:h1First,h1_chars:String(h1First.length),h2_texts:\"\",order_violations:String(orderIssues.length),order_violation_list:orderIssues.join(\"; \"),empty_headings:String(emptyCount),duplicate_headings:String(dupes.length),long_headings:String(longCount),heading_word_total:String(headingWords),first_heading_level:firstLevel?String(firstLevel):\"\",max_heading_level:String(maxLevel),headings_verdict:verdict.length?verdict.join(\"; \"):\"OK\"};" } ], "properties": [ { "name": "tag", "type": "property", "selector": "css=self" }, { "name": "text", "type": "text", "selector": "css=self" }, { "name": "outline", "type": "index" }, { "name": "h1_count", "type": "index" }, { "name": "h1_missing", "type": "index" }, { "name": "h1_multiple", "type": "index" }, { "name": "h1_first_text", "type": "index" }, { "name": "h1_chars", "type": "index" }, { "name": "h2_texts", "type": "index" }, { "name": "order_violations", "type": "index" }, { "name": "order_violation_list", "type": "index" }, { "name": "empty_headings", "type": "index" }, { "name": "duplicate_headings", "type": "index" }, { "name": "long_headings", "type": "index" }, { "name": "heading_word_total", "type": "index" }, { "name": "first_heading_level", "type": "index" }, { "name": "max_heading_level", "type": "index" }, { "name": "headings_verdict", "type": "index" } ] }, { "name": "Hreflang_Audit", "type": "list", "selector": "css=link[rel='alternate'][hreflang]", "transformations": [ { "type": "script", "code": "var rows=Array.isArray(value)?value:[];var pageUrl=rows.length?String(rows[0].page_url||\"\"):\";var pu=null;try{pu=new URL(pageUrl);}catch(e){}var vals=[],langs=[],regions=[],relative=0,dupes={},dupeList=[],selfRef=false,hasDefault=false;for(var i=0;i0)issues.push(relative+\" relative hreflang URL(s)\");if(dupeList.length)issues.push(\"Duplicate hreflang values: \"+dupeList.join(\", \"));}return {count:String(rows.length),values:vals.join(\" | \").substring(0,4000),languages:langs.join(\", \"),regions:regions.join(\", \"),has_x_default:String(hasDefault),self_referencing:String(selfRef),relative_urls:String(relative),duplicate_langs:dupeList.join(\", \"),issues:issues.join(\"; \")};" } ], "properties": [ { "name": "lang", "type": "attribute", "selector": "css=self", "attribute": "hreflang" }, { "name": "href", "type": "attribute", "selector": "css=self", "attribute": "href" }, { "name": "page_url", "type": "page_url", "selector": "" }, { "name": "count", "type": "index" }, { "name": "values", "type": "index" }, { "name": "languages", "type": "index" }, { "name": "regions", "type": "index" }, { "name": "has_x_default", "type": "index" }, { "name": "self_referencing", "type": "index" }, { "name": "relative_urls", "type": "index" }, { "name": "duplicate_langs", "type": "index" }, { "name": "issues", "type": "index" } ] }, { "name": "Social_Meta", "type": "list", "selector": "css=self", "transformations": [ { "type": "script", "code": "var it=(Array.isArray(value)&&value.length)?value[0]:{};var head=String(it.head_html||\"\");var pageUrl=String(it.page_url||\"\");function grab(re){var m=head.match(re);return m?m[1].replace(/&/g,\"&\").trim():\"\";}var ogTitle=grab(/property=[\"']og:title[\"'][^>]*content=[\"']([^\"']*)[\"']/i)||grab(/content=[\"']([^\"']*)[\"'][^>]*property=[\"']og:title[\"']/i);var ogDesc=grab(/property=[\"']og:description[\"'][^>]*content=[\"']([^\"']*)[\"']/i)||grab(/content=[\"']([^\"']*)[\"'][^>]*property=[\"']og:description[\"']/i);var ogImage=grab(/property=[\"']og:image[\"'][^>]*content=[\"']([^\"']*)[\"']/i)||grab(/content=[\"']([^\"']*)[\"'][^>]*property=[\"']og:image[\"']/i);var ogImageSecure=grab(/property=[\"']og:image:secure_url[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogW=grab(/property=[\"']og:image:width[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogH=grab(/property=[\"']og:image:height[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogAlt=grab(/property=[\"']og:image:alt[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogUrl=grab(/property=[\"']og:url[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogType=grab(/property=[\"']og:type[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogSite=grab(/property=[\"']og:site_name[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogLocale=grab(/property=[\"']og:locale[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var twCard=grab(/name=[\"']twitter:card[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var twTitle=grab(/name=[\"']twitter:title[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var twDesc=grab(/name=[\"']twitter:description[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var twImage=grab(/name=[\"']twitter:image[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var twSite=grab(/name=[\"']twitter:site[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var twCreator=grab(/name=[\"']twitter:creator[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var artPub=grab(/property=[\"']article:published_time[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var artMod=grab(/property=[\"']article:modified_time[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var artAuthor=grab(/property=[\"']article:author[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var artSection=grab(/property=[\"']article:section[\"'][^>]*content=[\"']([^\"']*)[\"']/i);var ogCount=(head.match(/property=[\"']og:/gi)||[]).length;var twCount=(head.match(/name=[\"']twitter:/gi)||[]).length;var artTags=[];var re=/property=[\"']article:tag[\"'][^>]*content=[\"']([^\"']*)[\"']/gi;var mm;while((mm=re.exec(head))!==null){artTags.push(mm[1]);}var ogMissing=[];if(!ogTitle)ogMissing.push(\"og:title\");if(!ogDesc)ogMissing.push(\"og:description\");if(!ogImage)ogMissing.push(\"og:image\");if(!ogUrl)ogMissing.push(\"og:url\");if(!ogType)ogMissing.push(\"og:type\");var twMissing=[];if(!twCard)twMissing.push(\"twitter:card\");if(!twTitle)twMissing.push(\"twitter:title\");if(!twDesc)twMissing.push(\"twitter:description\");if(!twImage)twMissing.push(\"twitter:image\");var ogUrlMatch=false;if(ogUrl&&pageUrl){try{var a=new URL(ogUrl),b=new URL(pageUrl);ogUrlMatch=(a.origin===b.origin&&a.pathname===b.pathname);}catch(e){}}var ogImgAbs=/^https?:\\/\\//i.test(ogImage);var score=0,total=9;if(ogTitle)score++;if(ogDesc)score++;if(ogImage)score++;if(ogUrl)score++;if(ogType)score++;if(twCard)score++;if(twTitle)score++;if(twDesc)score++;if(twImage)score++;var pct=Math.round((score/total)*100);var verdict=pct>=90?\"Excellent\":pct>=70?\"Good\":pct>=40?\"Incomplete\":\"Missing\";return {og_title:ogTitle,og_description:ogDesc,og_image:ogImage,og_image_secure:ogImageSecure,og_image_width:ogW,og_image_height:ogH,og_image_alt:ogAlt,og_url:ogUrl,og_type:ogType,og_site_name:ogSite,og_locale:ogLocale,og_count:String(ogCount),og_missing:ogMissing.join(\", \"),twitter_card:twCard,twitter_title:twTitle,twitter_description:twDesc,twitter_image:twImage,twitter_site:twSite,twitter_creator:twCreator,twitter_count:String(twCount),twitter_missing:twMissing.join(\", \"),article_published:artPub,article_modified:artMod,article_author:artAuthor,article_section:artSection,article_tags:artTags.join(\", \"),og_url_matches_page:String(ogUrlMatch),og_image_is_absolute:String(ogImgAbs),social_score:String(pct),social_verdict:verdict};" } ], "properties": [ { "name": "head_html", "type": "html", "selector": "css=head" }, { "name": "page_url", "type": "page_url", "selector": "" }, { "name": "og_title", "type": "index" }, { "name": "og_description", "type": "index" }, { "name": "og_image", "type": "index" }, { "name": "og_image_secure", "type": "index" }, { "name": "og_image_width", "type": "index" }, { "name": "og_image_height", "type": "index" }, { "name": "og_image_alt", "type": "index" }, { "name": "og_url", "type": "index" }, { "name": "og_type", "type": "index" }, { "name": "og_site_name", "type": "index" }, { "name": "og_locale", "type": "index" }, { "name": "og_count", "type": "index" }, { "name": "og_missing", "type": "index" }, { "name": "twitter_card", "type": "index" }, { "name": "twitter_title", "type": "index" }, { "name": "twitter_description", "type": "index" }, { "name": "twitter_image", "type": "index" }, { "name": "twitter_site", "type": "index" }, { "name": "twitter_creator", "type": "index" }, { "name": "twitter_count", "type": "index" }, { "name": "twitter_missing", "type": "index" }, { "name": "article_published", "type": "index" }, { "name": "article_modified", "type": "index" }, { "name": "article_author", "type": "index" }, { "name": "article_section", "type": "index" }, { "name": "article_tags", "type": "index" }, { "name": "og_url_matches_page", "type": "index" }, { "name": "og_image_is_absolute", "type": "index" }, { "name": "social_score", "type": "index" }, { "name": "social_verdict", "type": "index" } ] } ], "required_fields": [ "Page_URL" ] }, "Content_Readability": { "item_selector": "css=html", "properties": [ { "name": "Page_URL", "type": "page_url", "selector": "" }, { "name": "Paragraph_Count", "type": "count", "selector": "css=body p" }, { "name": "List_Count", "type": "count", "selector": "css=body ul,ol" }, { "name": "Table_Count", "type": "count", "selector": "css=body table" }, { "name": "Blockquote_Count", "type": "count", "selector": "css=body blockquote" }, { "name": "Main_Landmark", "type": "count", "selector": "css=main,[role='main'],article" }, { "name": "Content_Metrics", "type": "list", "selector": "css=self", "transformations": [ { "type": "script", "code": "var it=(Array.isArray(value)&&value.length)?value[0]:{};var text=String(it.text||\"\").replace(/[\\u200B-\\u200D\\uFEFF]/g,\" \").replace(/\\s+/g,\" \").trim();var html=String(it.html||\"\");var title=String(it.title||\"\").replace(/\\s+/g,\" \").trim();var h1=String(it.h1||\"\").replace(/\\s+/g,\" \").trim();var desc=String(it.desc||\"\").replace(/\\s+/g,\" \").trim();var STOP={the:1,be:1,to:1,of:1,and:1,a:1,in:1,that:1,have:1,i:1,it:1,for:1,not:1,on:1,with:1,he:1,as:1,you:1,do:1,at:1,this:1,but:1,his:1,by:1,from:1,they:1,we:1,say:1,her:1,she:1,or:1,an:1,will:1,my:1,one:1,all:1,would:1,there:1,their:1,what:1,so:1,up:1,out:1,if:1,about:1,who:1,get:1,which:1,go:1,me:1,when:1,make:1,can:1,like:1,time:1,no:1,just:1,him:1,know:1,take:1,people:1,into:1,year:1,your:1,good:1,some:1,could:1,them:1,see:1,other:1,than:1,then:1,now:1,look:1,only:1,come:1,its:1,over:1,think:1,also:1,back:1,after:1,use:1,two:1,how:1,our:1,work:1,first:1,well:1,way:1,even:1,new:1,want:1,because:1,any:1,these:1,give:1,day:1,most:1,us:1,is:1,are:1,was:1,were:1,been:1,has:1,had:1,did:1,does:1};var words=text?text.split(\" \").filter(function(w){return w.length>0;}):[];var wc=words.length;var sentences=text.split(/[.!?]+(?:\\s|$)/).filter(function(s){return s.trim().length>1;});var sc=sentences.length||1;function syl(w){w=w.toLowerCase().replace(/[^a-z]/g,\"\");if(!w)return 0;if(w.length<=3)return 1;w=w.replace(/(?:[^laeiouy]es|ed|[^laeiouy]e)$/,\"\").replace(/^y/,\"\");var m=w.match(/[aeiouy]{1,2}/g);return m?m.length:1;}var sylTotal=0,longWords=0,chars=0;var uniq={},topMap={};for(var i=0;i=3)longWords++;uniq[w]=1;if(w.length>2&&!STOP[w])topMap[w]=(topMap[w]||0)+1;}var uw=Object.keys(uniq).length;var asl=wc/sc;var asw=wc?sylTotal/wc:0;function r2(x){return Math.round(x*100)/100;}var fre=wc?r2(206.835-(1.015*asl)-(84.6*asw)):0;var fkg=wc?r2((0.39*asl)+(11.8*asw)-15.59):0;var fog=wc?r2(0.4*(asl+(wc?(longWords/wc)*100:0))):0;var smog=sc>=30?r2(1.0430*Math.sqrt((longWords*(30/sc)))+3.1291):0;var ari=wc?r2((4.71*(chars/wc))+(0.5*(wc/sc))-21.43):0;var cl=wc?r2((0.0588*(chars/wc*100)/100)*100-(0.296*(wc/sc))-15.8):0;var readVerdict=fre>=90?\"Very Easy\":fre>=80?\"Easy\":fre>=70?\"Fairly Easy\":fre>=60?\"Plain\":fre>=50?\"Fairly Difficult\":fre>=30?\"Difficult\":wc?\"Very Difficult\":\"No text\";var textChars=text.length,htmlChars=html.length;var ratio=htmlChars?r2((textChars/htmlChars)*100):0;var ratioVerdict=ratio===0?\"No text\":ratio<10?\"Very Low\":ratio<25?\"Low\":ratio<70?\"Healthy\":\"High\";var readMin=Math.max(1,Math.round(wc/225));var pairs=Object.keys(topMap).map(function(k){return [k,topMap[k]];});pairs.sort(function(a,b){return b[1]-a[1];});var top=pairs.slice(0,15).map(function(p){return p[0]+\"(\"+p[1]+\")\";});var stopCount=0;for(var j=0;jlongest)longest=words[k2].length;}var avgWordLen=wc?r2(chars/wc):0;function cyrb128(str){var h1=1779033703,h2=3144134277,h3=1013904242,h4=2773480762;for(var q=0,chr;q>>16),2246822507)^Math.imul(h2^(h2>>>13),3266489909);h2=Math.imul(h2^(h2>>>16),2246822507)^Math.imul(h1^(h1>>>13),3266489909);h3=Math.imul(h3^(h3>>>16),2246822507)^Math.imul(h4^(h4>>>13),3266489909);h4=Math.imul(h4^(h4>>>16),2246822507)^Math.imul(h3^(h3>>>13),3266489909);return [h1>>>0,h2>>>0,h3>>>0,h4>>>0];}var norm=text.toLowerCase().replace(/[^a-z0-9\\s]/g,\" \").replace(/\\s+/g,\" \").trim();var shingles={};var nWords=norm.split(\" \");for(var s2=0;s2+20)sim=sim*2+1;else sim=sim*2;}sim=sim>>>0;}var hashArr=cyrb128(norm);var hashHex=\"\";for(var w2=0;w2<4;w2++){hashHex+=(\"00000000\"+hashArr[w2].toString(16)).slice(-8);}var simHex=(\"0000000000000000\"+sim.toString(16)).slice(-16);var langGuess=\"\";var hl=String(it.html_lang||\"\").trim();if(hl)langGuess=hl;else{var enScore=(text.match(/\\b(the|and|of|to|in|is|for|with)\\b/gi)||[]).length;var esScore=(text.match(/\\b(el|la|de|que|y|en|un|una|los|las)\\b/gi)||[]).length;if(enScore>esScore&&enScore>5)langGuess=\"en\";else if(esScore>5)langGuess=\"es\";else langGuess=\"unknown\";}function norm2(x){return x.toLowerCase().replace(/[^a-z0-9]/g,\"\");}var titleInBody=norm2(title).length>6&&norm.indexOf(norm2(title))!==-1;var h1InBody=norm2(h1).length>3&&norm.indexOf(norm2(h1))!==-1;var descInBody=norm2(desc).length>20&&norm.indexOf(norm2(desc))!==-1;var lorem=/lorem\\s+ipsum/i.test(text);var placeholder=/\\b(coming soon|under construction|todo:|tbd|placeholder text|your text here)\\b/i.test(text);var cy=null;var cym=text.match(/©|\\(c\\)|copyright/gi)?text.match(/(?:©|\\(c\\)|copyright)\\s*(?:\\d{4}\\s*[-–]\\s*)?(\\d{4})/i):null;if(cym)cy=parseInt(cym[1],10);var nowYear=new Date().getFullYear();var cyAge=(cy&&cy>1990&&cy<=nowYear+1)?(nowYear-cy):null;var dateMatches=text.match(/\\b(?:19|20)\\d{2}-\\d{2}-\\d{2}\\b|\\b(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\\.?[\\s]+\\d{1,2},?[\\s]+(?:19|20)\\d{2}\\b|\\b\\d{1,2}\\s+(?:January|February|March|April|May|June|July|August|September|October|November|December)\\s+(?:19|20)\\d{2}\\b/gi)||[];var latest=\"\";for(var d2=0;d2parseInt(latest||\"0\",10))latest=String(yr);}}var emails=[];var em=text.match(/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}/g)||[];for(var e2=0;e2=9&&phones.indexOf(ph[p2])===-1)phones.push(ph[p2]);}var socialRe=/(?:https?:\\/\\/)?(?:www\\.)?(facebook|instagram|twitter|x|linkedin|youtube|pinterest|tiktok|github|reddit|whatsapp|t\\.me|telegram)\\.com\\/[^\\s\"'<>)]+/gi;var socialMatches=html.match(socialRe)||[];var socialDomains={};for(var sm2=0;sm2/g)||[]).length;var sensitive=/ https://shop.example.com/promo --[302]--> https://shop.example.com/sale/summer | true | true | false | false | false | 301 Permanent | OK | **Source_URL** is the address you submitted, normalized by the engine. It is the required field and the de-duplication key, so submitting the same URL twice in one list yields one row. **Final_URL** is where the browser actually landed after all hops. Diffing it against your intended destination is the core migration check. **Status_Code** and **Status_Text** are the code and phrase of the last hop - 301 Moved Permanently, 200 OK, 404 Not Found and so on. **Redirect_Count** is the number of hops between source and destination. Zero means the URL answered directly. **Navigation_OK** is true when the browser completed the navigation. False rows also appear in the Errors dataset with the reason. **Error** carries the engine's failure text - DNS resolution failures, timeouts, connection resets. Empty on healthy rows, and never invented: a failed URL shows a real error string or lands in Errors, never a fabricated success. **Content_Type**, **Server** and **Protocol** describe the final response: what came back, from what software, over which HTTP version. Useful for spotting a redirect that lands on an unexpected PDF, or a destination still served by the old host. **Response_Time_ms** is the full navigation time in milliseconds, chain included. Redirects are a latency tax; this column quantifies it. **X_Robots_Tag** and **Cache_Control** are the corresponding response headers from the last hop. An X-Robots-Tag of noindex on a destination page, or a long cache on a temporary redirect, are exactly the kind of quiet problems an audit exists to catch. Empty is normal when the header is absent. **Chain_Status_Codes** is the hop-by-hop status sequence, arrow-joined. **Chain_Details** numbers each hop with its target. **Chain_Visual** draws the full source-to-hop-to-destination map, one hop per line. **Has_301**, **Has_302**, **Has_303**, **Has_307** and **Has_308** are per-row booleans: did any hop in the chain use that code? Filter a whole report on Has_302 to find every temporary redirect in seconds. **Redirect_Type** classifies the row - 301 Permanent, 308 Permanent, 302 Temporary, 303 See Other, 307 Temporary, or No HTTP Redirect when the URL answered directly. When several codes appear in one chain, permanence wins the label, and the Has_ columns carry the full truth. **Chain_Length_Verdict** grades the hop count: No Redirect at zero, OK at one or two, Long Chain up to five, Redirect Loop Risk beyond that. The Errors dataset is a three-column triage view - Source_URL, Status_Code, Error - and Destination_URLs is the de-duplicated set of landing pages, which answers "how many of my old URLs collapse onto the same new page?" ## How does it work as a redirect chain analyzer? A redirect chain analyzer has to answer three questions per URL: what path did the request take, was every hop the right kind of redirect, and is the path getting worse over time? The chain columns answer the first two directly - Chain_Visual is the path, Has_302 and Redirect_Type are the kind - and the third is a workflow: save each run's export with a date in the filename, then diff Chain_Length_Verdict counts week over week. A migration that starts accreting intermediate hops (old page to newer page to newest page) shows up as OK rows drifting to Long Chain, long before users feel it. ## Can it serve as a 301 redirect audit tool after a migration? Yes, and that is its most common job. Export the old URL list from the previous CMS, the legacy sitemap, or server access logs; paste it into the list field; run. Then work three filters in order: rows where Redirect_Type is anything other than 301 Permanent or 308 Permanent - those are moves being served as temporary; rows where Final_URL is not the intended new address - those are broken or stale rules; and rows in Errors - those are old URLs now failing outright, which may be acceptable (deleted content should 404 or 410) or a regression. What this bot will not do is crawl your site to discover URLs, grade on-page SEO, or alert you when something changes; it checks the list you give it, when you give it. ## How does it compare to spot checks and subscription audits? Checking redirects through browser developer tools works for one URL at a time and produces no artifact - there is nothing to hand a client or re-run next month. Subscription crawling platforms will audit redirects as part of a full-site crawl, but they meter the crawl, keep the data in their cloud, and you audit the URLs they discover rather than the exact list you care about. This bot inverts that: you supply the precise list, the checks run from your own machine at no per-run cost, and the dated export is yours. The trade is real - no dashboards, no history, no scheduling - but for a targeted migration or legacy-URL audit, a precise list beats a sampled crawl. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the executable; SmartScreen may ask for confirmation on first launch, and "More info" then "Run anyway" proceeds if you trust the published checksum. On **macOS**, Gatekeeper may quarantine the download; right-click and "Open", or clear the flag with `xattr -d com.apple.quarantine /path/to/RedirectChecker`, then launch. On **Linux**, the launcher is a shell script: `chmod +x ./redirect-checker` then `./redirect-checker`. On first run the launcher fetches its configuration, opens a visible browser window starting on a blank page, and presents the `urls_to_check` list for editing - one URL per line, with the six public test addresses pre-loaded so you can verify the whole pipeline before pointing it at anything real. The test list deliberately includes a domain that does not exist, so a healthy self-test produces report rows and at least one Errors row. No proxies, solver keys or AI providers are required. To run audits on a schedule, trigger the launcher from your operating system's scheduler; the bot itself does not self-schedule. ## What do I check when a run returns zero rows? In order: is the URL list actually populated - an empty list means the loop never executes and every dataset stays empty? If rows appear only in Errors, read the Error column: DNS text means the host does not resolve, timeout text means the server never answered within the navigation limit. If Redirect_Report rows exist but Navigation_OK is false, the browser gave up mid-chain - the classic signature of a genuine redirect loop. The contract is that failure is loud: this bot never fabricates a status code or a destination, so a missing row is always a visible failure, never a silent success. ## What are the limits of the bot? It checks only the URLs you supply - it does not crawl, discover, or follow links found on destination pages. It navigates with scripts and sub-resources blocked, so a redirect implemented purely in client-side script or a single-page-app router is not observed; server-side HTTP redirects and meta-refresh behavior at the document level are its domain. It performs one browser navigation per URL - no HEAD variants, no alternate methods, no cookie or auth states. It de-duplicates on Source_URL, so the same URL listed twice yields one row. It has no scheduler, no alerting and no history; those are your spreadsheet's job. And it follows redirects like a browser, which means very long chains can end in a hard navigation failure rather than a tidy verdict row - by design, and recorded in Errors when it happens. ## What is responsible redirect checking? Every URL in your list receives a real request from a real browser session on your machine. That is harmless at audit scale against servers you own, and it is the reason for the one firm rule: only check URLs you own or have explicit permission to test. Never point the list at third-party infrastructure to probe its behavior, keep run frequency proportionate to what the target server would expect from a handful of visitors, and remember that exported reports may contain internal or pre-launch URLs - store them with the same care you give any internal document. The acceptable-use policy applies in full before your first run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | urls_to_check | list | yes | https://httpbin.org/redirect/1 https://httpbin.org/redirect/3 https://httpbin.org/status/301 https://httpbin.org/status/302 https://www.google.com https://this-domain-does-not-exist-9x7q.com | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **Redirect_Report** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Source_URL | text | no | variable=normalized_url | | Final_URL | text | no | variable=last_final_url | | Status_Code | text | no | variable=last_http_status | | Status_Text | text | no | variable=last_http_status_text | | Redirect_Count | text | no | variable=last_redirect_count | | Navigation_OK | text | no | variable=last_navigation_ok | | Error | text | no | variable=last_navigation_error | | Content_Type | text | no | variable=last_content_type | | Server | text | no | variable=last_server | | Protocol | text | no | variable=last_protocol | | Response_Time_ms | text | no | variable=last_response_time_ms | | X_Robots_Tag | text | no | variable=last_x_robots_tag | | Cache_Control | text | no | variable=last_cache_control | | Chain_Status_Codes | text | no | variable=last_redirect_chain | | Chain_Details | text | no | variable=last_redirect_chain | | Chain_Visual | text | no | variable=last_redirect_chain | | Has_301 | text | no | variable=last_redirect_chain | | Has_302 | text | no | variable=last_redirect_chain | | Has_303 | text | no | variable=last_redirect_chain | | Has_307 | text | no | variable=last_redirect_chain | | Has_308 | text | no | variable=last_redirect_chain | | Redirect_Type | text | no | variable=last_redirect_chain | | Chain_Length_Verdict | text | no | variable=last_redirect_count | **Errors** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Source_URL | text | no | variable=normalized_url | | Status_Code | text | no | variable=last_http_status | | Error | text | no | variable=last_navigation_error | **Destination_URLs** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Destination_URL | text | no | variable=last_final_url | Export formats: csv, json, xlsx. ### Downloads - Windows x86_64: `BOT_redirect-checker.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/redirect-checker/windows - macOS Apple Silicon: `BOT_redirect-checker.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/redirect-checker/macos - macOS Intel: `BOT_redirect-checker.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/redirect-checker/macos_intel - Linux x86_64: `BOT_redirect-checker.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/redirect-checker/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=redirect-checker` Configuration document: `https://marketplace.rtila.com/api/bots/redirect-checker/config.json` - Encoding: plain - Digest: e1406a349457b08595d37c635806675ee467e834b66b20ad77e627bf740b0a3f - Bytes: 9295 ```json { "name": "Redirect Checker", "settings": { "urls": [ "about:blank" ], "headless": false, "max_concurrent_workers": 3, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": true, "variables": [ { "name": "urls_to_check", "value": "https://httpbin.org/redirect/1\nhttps://httpbin.org/redirect/3\nhttps://httpbin.org/status/301\nhttps://httpbin.org/status/302\nhttps://www.google.com\nhttps://this-domain-does-not-exist-9x7q.com", "required": true, "is_list": true } ] }, "datasets": { "Redirect_Report": { "item_selector": "css=html", "properties": [ { "name": "Source_URL", "type": "text", "selector": "variable=normalized_url" }, { "name": "Final_URL", "type": "text", "selector": "variable=last_final_url" }, { "name": "Status_Code", "type": "text", "selector": "variable=last_http_status" }, { "name": "Status_Text", "type": "text", "selector": "variable=last_http_status_text" }, { "name": "Redirect_Count", "type": "text", "selector": "variable=last_redirect_count" }, { "name": "Navigation_OK", "type": "text", "selector": "variable=last_navigation_ok" }, { "name": "Error", "type": "text", "selector": "variable=last_navigation_error" }, { "name": "Content_Type", "type": "text", "selector": "variable=last_content_type" }, { "name": "Server", "type": "text", "selector": "variable=last_server" }, { "name": "Protocol", "type": "text", "selector": "variable=last_protocol" }, { "name": "Response_Time_ms", "type": "text", "selector": "variable=last_response_time_ms" }, { "name": "X_Robots_Tag", "type": "text", "selector": "variable=last_x_robots_tag" }, { "name": "Cache_Control", "type": "text", "selector": "variable=last_cache_control" }, { "name": "Chain_Status_Codes", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "if(!Array.isArray(value)||value.length===0)return '';return value.map(function(h){return h.status||'?';}).join(' -> ');", "default_value": "" } ] }, { "name": "Chain_Details", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "if(!Array.isArray(value)||value.length===0)return '';return value.map(function(h,i){return 'Hop '+(i+1)+': '+((h.status)||'?')+' '+(h.location||h.url||'');}).join(' | ');", "default_value": "" } ] }, { "name": "Chain_Visual", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "if(!Array.isArray(value)||value.length===0)return '';var p=[];for(var i=0;i '+(h.location||''));}return p.join('\\n');", "default_value": "" } ] }, { "name": "Has_301", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "return String(Array.isArray(value)&&value.some(function(h){return h.status===301;}));", "default_value": "false" } ] }, { "name": "Has_302", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "return String(Array.isArray(value)&&value.some(function(h){return h.status===302;}));", "default_value": "false" } ] }, { "name": "Has_303", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "return String(Array.isArray(value)&&value.some(function(h){return h.status===303;}));", "default_value": "false" } ] }, { "name": "Has_307", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "return String(Array.isArray(value)&&value.some(function(h){return h.status===307;}));", "default_value": "false" } ] }, { "name": "Has_308", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "return String(Array.isArray(value)&&value.some(function(h){return h.status===308;}));", "default_value": "false" } ] }, { "name": "Redirect_Type", "type": "text", "selector": "variable=last_redirect_chain", "transformations": [ { "type": "script", "code": "var c=Array.isArray(value)?value:[];if(c.length===0)return 'No HTTP Redirect';var s=c.map(function(h){return h.status;});if(s.indexOf(301)!==-1)return '301 Permanent';if(s.indexOf(308)!==-1)return '308 Permanent';if(s.indexOf(302)!==-1)return '302 Temporary';if(s.indexOf(303)!==-1)return '303 See Other';if(s.indexOf(307)!==-1)return '307 Temporary';return 'Redirect ('+s.join(',')+')';", "default_value": "Unknown" } ] }, { "name": "Chain_Length_Verdict", "type": "text", "selector": "variable=last_redirect_count", "transformations": [ { "type": "script", "code": "var n=Number(value);if(!isFinite(n)||n===0)return 'No Redirect';if(n<=2)return 'OK';if(n<=5)return 'Long Chain';return 'Redirect Loop Risk';", "default_value": "Unknown" } ] } ], "required_fields": [ "Source_URL" ] }, "Errors": { "item_selector": "css=html", "properties": [ { "name": "Source_URL", "type": "text", "selector": "variable=normalized_url" }, { "name": "Status_Code", "type": "text", "selector": "variable=last_http_status" }, { "name": "Error", "type": "text", "selector": "variable=last_navigation_error" } ], "required_fields": [ "Source_URL" ] }, "Destination_URLs": { "item_selector": "css=html", "properties": [ { "name": "Destination_URL", "type": "text", "selector": "variable=last_final_url" } ], "required_fields": [ "Destination_URL" ] } }, "commands": [ { "command": "for_each", "params": { "source_type": "variable", "variable": "urls_to_check", "as": "target_url", "do": [ { "command": "try_catch", "params": { "try": [ { "command": "goto", "params": { "url": "${target_url}" } } ], "catch": [ { "command": "comment", "params": { "text": "Navigation failed: ${last_error}" } } ] } }, { "command": "extract_data", "params": { "dataset": "Redirect_Report" } }, { "command": "extract_data", "params": { "dataset": "Errors", "filter_condition": "item.Navigation_OK === false || item.Status_Code === null || item.Status_Code === ''" } }, { "command": "extract_data", "params": { "dataset": "Destination_URLs", "filter_condition": "item.Destination_URL && item.Destination_URL !== ''" } } ] } } ], "authors": [] } ``` ### FAQ **How do I check a list of URLs for redirects in bulk?** Paste your URLs into the list field, one per line, and start the run. The bot walks the list with up to three parallel workers, navigates each URL, and records the full redirect chain into the Redirect_Report dataset. A six-URL test list is pre-loaded so your very first run doubles as a self-test. **What is a redirect chain and why does its length matter?** A chain is the sequence of hops between the URL you submit and the URL the browser finally lands on. Each hop costs latency and can leak ranking signals if intermediate URLs are temporary. The Chain_Status_Codes, Chain_Details and Chain_Visual columns show every hop, and Chain_Length_Verdict grades it: OK at two hops or fewer, Long Chain up to five, Redirect Loop Risk beyond that. **How do I find 302 redirects that should be 301?** Filter the Redirect_Report output on the Has_302 column or on Redirect_Type showing 302 Temporary. A permanent move served as a temporary redirect tells crawlers the old URL may return, which is the most common migration mistake this bot surfaces. **What is the difference between 301 302 303 307 and 308 status codes?** 301 is a permanent move that may rewrite the request method; 308 is permanent but preserves the method; 302 is temporary and may rewrite the method; 307 is temporary and preserves the method; 303 tells the client to fetch another resource with a GET. For SEO auditing the practical split is permanent (301, 308) versus temporary (302, 307), and the Redirect_Type column classifies each URL accordingly. **How do I audit redirects after a website migration?** Export your old URL list from the previous CMS, sitemap or server logs, paste it into the checker, and run it against the live site. Then review three things: URLs whose Final_URL is not the intended new page, rows where Redirect_Type says temporary, and rows where Chain_Length_Verdict flags Long Chain or Redirect Loop Risk. Re-run the same list weekly to catch regressions. **Why does the checker block images scripts and stylesheets?** The bot only needs the navigation itself, not the rendered page. Blocking images, media, fonts, styles, scripts and background requests makes each check dramatically faster and lighter on bandwidth. The trade-off is documented in the limits section: a redirect that only fires from client-side script will not be observed. **Can it detect redirect loops?** Yes, indirectly and honestly. A browser gives up on a loop and the navigation fails, so the URL lands in the Errors dataset with Navigation_OK false and the engine error text. Chains that are merely very long are graded Redirect Loop Risk by the Chain_Length_Verdict column before they become hard failures. **What do I do with a URL that returns no redirect?** Nothing is wrong - Redirect_Type reads No HTTP Redirect and Chain_Length_Verdict reads No Redirect, with Status_Code showing the direct response, usually 200. In a migration audit these rows deserve attention: an old URL still serving content directly instead of forwarding is a duplicate-content risk. **How many URLs can one run check?** There is no built-in cap; the list length governs the run. Three workers process URLs in parallel, so a few hundred typical URLs finish in minutes. Very large lists mainly need patience and proportionate request rates against the target server. **Does the bot follow redirects all the way to the final destination?** Yes. Navigation behaves exactly like a real browser session: the engine follows every hop, then records the final URL, the final status, the response headers of the last hop, and the full chain of intermediate statuses and locations. **Can I check redirects without a paid SEO tool?** That is the point of this bot. It is a free desktop launcher, your URL list never leaves your machine except as requests to the URLs themselves, and the report exports to CSV, JSON or XLSX for review in any spreadsheet. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## SEO Rank Tracker Check where a domain ranks in web search results for a list of keywords and export the positions to CSV. - URL: https://marketplace.rtila.com/bots/seo-rank-tracker - Category: SEO - Tags: rank-tracking, keyword-ranks, seo-audit, serp-data, google-search - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-09-20) - Price: free ### Overview The SEO Rank Tracker is an seo rank tracker: a standalone desktop bot that runs every keyword on your list as a live web search, finds where a chosen domain appears in the organic results, and records the position in the Default dataset, which the launcher saves as a local CSV. ## What does the SEO Rank Tracker do? You give it two things before a run: a domain (the shipped configuration defaults to `amazon.com`) and a list of keywords, one per line. For each keyword the bot opens a visible browser window - headless mode is off in this configuration - loads the search engine results page for that keyword, and scans the organic result cards. Any card whose displayed URL contains your domain becomes a row: the domain, the keyword, the display URL and the position on the page. Rows for everyone else are discarded by the filter, so the output is a clean "where do I rank" table rather than a copy of the whole results page. One honesty note up front: this bot is not affiliated with, endorsed by, or connected to any search engine. It reads the same public results a logged-out visitor sees. The selectors in its configuration match the results container structure at the time of release; search engines redesign their markup regularly, and the diagnostics section below explains what a mismatch looks like. Because everything runs on your own machine, your keyword strategy - which terms you track, which domains you monitor, how often you check - never passes through a third-party service. That is the point of a local-first tracker: the artifact and the workflow both belong to you. ## Who uses an SEO rank tracker? The in-house SEO lead asks "did the redesign move our category pages up or down?" and needs a dated position table before and after. The e-commerce founder asks "do we actually rank for the ten terms we are optimizing?" and needs a quick check without signing up for a subscription. The freelance consultant asks "what is the current position snapshot for this client report?" and needs a CSV to paste into their own template and date-stamp. All three want the same artifact: a small table of keyword, domain, URL and position, produced on demand, comparable across weeks by diffing files. ## How does a Google rank checker run work, step by step? Every command in the run is linked to its reference page. The configuration first seeds the browser session at the search home page declared in the project settings. Then [for_each](https://learn.rtila.com/reference/commands/for_each) iterates the `keywords` list variable, binding each line to the loop variable `keyword`. Inside the loop, [goto](https://learn.rtila.com/reference/commands/goto) loads `https://google.com/search?q=${keyword}&udm=14` - the `udm=14` parameter pins the engine to its web-results view. Next, [extract_data](https://learn.rtila.com/reference/commands/extract_data) reads the Default dataset: the item selector matches each organic result card in the results container, and four properties are captured - Domain (from the `domain` variable), Keyword (from the loop variable), URL (the display URL text, with a script transformation keeping everything before the first space) and Rank (the index of the card among the rendered results). The extraction carries the filter condition `item.URL.includes(item.Domain)`, so only cards whose display URL contains your domain survive into the dataset. After the loop finishes, the consolidated dataset is written out. There is no deduplication key and no scroll or pagination step: one search page per keyword, first page only, one row per appearance of your domain. ## What does a keyword position tracker output look like? The sample row below is the listing's published sample output, so the table and the field notes agree by construction: | Domain | Keyword | URL | Rank | |---|---|---|---| | amazon.com | stainless steel water bottle | www.amazon.com | 3 | **Domain** echoes the domain variable you set for the run, on every row, so files from several runs can be stacked without losing track of whose positions they are. **Keyword** is the exact search string from your list. Because it is stored per row, the CSV is already in tidy form: one keyword-position pair per line, ready for a pivot. **URL** is the display URL of the matching result card, truncated at the first space by the transformation. It tells you which page of the site holds the position - often more useful than the position itself. **Rank** is the 1-based index of the result card among the organic results rendered at run time. The index is computed before the domain filter is applied, so it is the true on-page position, not a renumbering of only your hits. No row for a keyword means no card matched the domain - see the diagnostics section. ## Can it serve as a SERP rank tracker for client reporting? Yes, with honest boundaries. A SERP rank tracker for reporting ultimately answers "which terms moved, on which pages, since when?" - and a dated CSV per run answers the first two directly; "since when" comes from diffing this week's file against last week's in your spreadsheet. What the bot does not do is store history, draw trend charts, or send alerts. Those are features of subscription products, and the comparison below sets out the trade. For a monthly client report built from a snapshot, the CSV-plus-spreadsheet workflow is complete; for continuous monitoring with dashboards, this is deliberately not that tool. ## How do I do bulk rank checking without an API budget? Bulk rank checking normally costs either subscription seats or per-request API fees. This bot costs neither: the `keywords` variable is a list, so one run walks the entire list sequentially, and the only resources consumed are your own bandwidth and a few seconds per keyword. The practical bound is politeness, not price - a list of 50 to 100 terms in one run is reasonable; thousands of terms per day from one residential IP is the kind of volume that triggers challenges and, honestly, deserves to. Split large lists across runs and days. ## How does it compare to subscription trackers and manual checking? Manual checking works for five keywords and produces no record; by the twentieth keyword, personalization and fatigue make the numbers unreliable and nobody keeps the history. Subscription trackers add daily re-checks, location grids, competitor sets and dashboards, and their published plans typically bill by tracked keywords or seats - treat any vendor pricing page as that vendor's own current claim. SERP data APIs sold per thousand requests give raw positions but leave you to build and host the pipeline. This bot sits at the simplest corner of that map: on-demand snapshots, first page only, one domain per run, output as a local CSV, zero marginal cost per run. You trade automation and history for ownership and simplicity. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your operating system from this page. On **Windows**, run the executable; SmartScreen may ask for confirmation on first launch, and "More info" then "Run anyway" proceeds if you trust the published checksum. On **macOS**, Gatekeeper may quarantine the download; right-click and "Open", or clear the flag with `xattr -d com.apple.quarantine /path/to/SEORankTracker`, then launch. On **Linux**, the launcher is a shell script: `chmod +x ./seo-rank-tracker` then `./seo-rank-tracker`. At run time the bot prompts for the two required variables - `domain` and `keywords` (one keyword per line) - and both must be supplied before the loop starts. The browser window stays visible during the run because headless mode is off; this is intentional, since a visible session is less likely to meet challenges. To run it on a schedule, point your operating system scheduler (Task Scheduler, cron, or launchd) at the launcher - the bot has no internal timer. ## What do I check when a run returns zero rows? First separate "no rows for one keyword" from "no rows at all". Zero rows for a single keyword is often the true answer: the domain simply does not appear on the first page for that term, and the filter dropped every card. Zero rows for every keyword means the run itself failed, so check in order: did a consent or cookie interstitial cover the results page (watch the visible browser window)? Did the engine serve a verification challenge - this configuration has no solver, so a challenge ends the collection honestly rather than working around it? Did the engine redesign its results markup so the card selector no longer matches? Is the domain string an exact substring of the display URLs - a short domain will not match rows showing a different host unless it is genuinely contained in them? The contract is that failure is loud: empty output means "not found or not read", never invented positions. ## What are the limits of the bot? First page only - there is no scroll or pagination, so a domain sitting on page two is recorded as absent. One domain per run; a competitor comparison is two runs and a spreadsheet join. Web results only - the `udm=14` parameter excludes images, news, video and local-pack presentations. No history, no diffing, no alerts, no internal scheduling. Positions are what one logged-out browser on your IP saw at run time; another region or device can legitimately see a different order. It does not bypass anti-bot measures and does not sign in anywhere. And the output records display URLs, not the final resolved link behind each card. ## What is responsible rank tracking? Automated querying may conflict with the search engine's terms of service, and every request leaves from your own machine and IP - there is no shared infrastructure to absorb the volume. Keep lists modest and runs occasional, at a pace a patient human could sustain. Track domains you own or have a legitimate reason to monitor; the positions themselves are public facts, but how you use a competitor snapshot is your responsibility under the platform's terms and your local laws. Do not resell raw results pages. The acceptable-use policy applies in full before your first run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | domain | text | yes | amazon.com | Declared in project settings. | | keywords | list | yes | stainless steel water bottle ergonomic office chair noise cancelling headphones | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **Default** (item selector: `xpath=//div[@id='rso']//a[h3]/ancestor::div[contains(@class, 'g')]`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | Domain | text | no | variable=domain | | Keyword | text | no | variable=keyword | | URL | text | no | css=#rso cite | | Rank | index | no | - | Export formats: csv, json, xlsx. ### Downloads - Windows x86_64: `BOT_seo-rank-tracker.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/seo-rank-tracker/windows - macOS Apple Silicon: `BOT_seo-rank-tracker.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/seo-rank-tracker/macos - macOS Intel: `BOT_seo-rank-tracker.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/seo-rank-tracker/macos_intel - Linux x86_64: `BOT_seo-rank-tracker.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/seo-rank-tracker/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=seo-rank-tracker` Configuration document: `https://marketplace.rtila.com/api/bots/seo-rank-tracker/config.json` - Encoding: plain - Digest: 6f92423c15f457fc31353b7ebea71b3636bd406d232e800fe81fa520a033b487 - Bytes: 2064 ```json { "name": "SEO Rank Tracker", "settings": { "urls": [ "https://www.google.com/" ], "headless": false, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": false, "variables": [ { "name": "domain", "value": "amazon.com", "required": true, "is_list": false }, { "name": "keywords", "value": "stainless steel water bottle\nergonomic office chair\nnoise cancelling headphones", "required": true, "is_list": true } ] }, "datasets": { "Default": { "item_selector": "xpath=//div[@id='rso']//a[h3]/ancestor::div[contains(@class, 'g')]", "properties": [ { "name": "Domain", "type": "text", "selector": "variable=domain" }, { "name": "Keyword", "type": "text", "selector": "variable=keyword" }, { "name": "URL", "type": "text", "selector": "css=#rso cite", "transformations": [ { "type": "script", "code": "return value.split(\" \")[0];" } ] }, { "name": "Rank", "type": "index" } ] } }, "commands": [ { "command": "for_each", "params": { "as": "keyword", "source_type": "variable", "variable": "keywords", "do": [ { "command": "goto", "params": { "url": "https://google.com/search?q=${keyword}&udm=14" } }, { "command": "extract_data", "params": { "dataset": "Default", "filter_condition": "item.URL.includes(item.Domain)" } } ] } } ], "authors": [] } ``` ### FAQ **How do I check my Google ranking for multiple keywords?** Paste one domain into the domain field and one keyword per line into the keywords list. The bot loops the list, runs a live search for each term, and records every result card whose display URL contains your domain, with its position on the page. The run ends with a CSV you can open in Excel or Sheets. **How do I track keyword positions in bulk for free?** The keywords variable is a list, so one run walks the entire list sequentially. The launcher is free and there is no per-run or per-keyword fee - the only cost is your own bandwidth and a few seconds per keyword. The practical bound is politeness: lists of 50 to 100 terms per run are reasonable, and huge daily volumes from one IP will trigger challenges. **How does the bot find my domain in the search results?** It reads the display URL of each organic result card, keeps the text before the first space, and applies the filter condition item.URL.includes(item.Domain). Only cards whose display URL contains the domain you configured survive into the dataset, so the output is your positions, not a copy of the whole results page. **What does the Rank column mean?** Rank is the 1-based index of the result card among the organic results rendered at run time. The index is computed before the domain filter is applied, so it is the true on-page position - a 7 means the card was the seventh organic result, not the seventh row in your file. **Why does a keyword return no rows?** Two very different causes. For a single keyword, no rows is usually the honest answer: the domain does not appear on the first page, and the filter dropped every card. For every keyword at once, the run itself failed - a consent interstitial, a verification challenge (no solver is configured), or a results-page redesign the selectors no longer match. Watch the visible browser window to tell them apart. **How do I export keyword rankings to CSV?** The consolidated Default dataset is written to a local CSV file by the launcher after the loop finishes, with one row per appearance of your domain: Domain, Keyword, URL and Rank. Rename the file with the run date and diff successive files in your spreadsheet to see movement over time. **Can I track a competitor domain instead of my own?** The domain field accepts any string, and the positions recorded are public facts visible to any logged-out visitor. But the platform terms and your local laws still govern how you collect and use the snapshot, and the bot offers no permission system of its own - that judgement is yours. **How often should I re-run the rank tracker?** As often as your volume budget allows; weekly is typical for client reporting and daily is aggressive from a single IP. The bot stores no history, so the workflow is: run, date the CSV, and diff against the previous file. Schedule it with your operating system scheduler if you want unattended weekly runs. **Does the bot use a search API?** No. It loads live results pages in a real browser session, exactly what a logged-out visitor sees. There are no API keys, no request credits and no per-query fees. The trade-off: first page only, web results only, and positions reflect your own IP and region at run time. **Is automated rank checking allowed by search engines?** Automated querying may conflict with the search engine terms of service. This bot keeps volumes human-scale and hands no challenge to a solver - it simply stops - but the decision to run it, and how often, is yours to own under the platform terms and your jurisdiction. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch. ## YouTube Metrics and Comments Collector Collect video metrics and top-level comments from public YouTube watch pages into CSV on your own machine. - URL: https://marketplace.rtila.com/bots/youtube-metrics-comments-collector - Category: Social Media - Tags: youtube, comments, video-metrics, social-media, scraping, csv-export - Platforms: Windows, macOS (Apple Silicon), macOS (Intel), Linux - Version: 1.0.0 (updated 2026-09-18) - Price: free ### Overview The YouTube Metrics and Comments Collector is a YouTube comment scraper: a standalone desktop bot that opens a public watch page, reads the video's own embedded metadata, scrolls the comment section to a target you set, and saves two datasets - one `video_metrics` row and one `comments` row per top-level thread - as CSV files on your machine. ## What does the YouTube Metrics and Comments Collector do? You give it three inputs before a run: `video_url` (one public watch page, or a list of them, one URL per line), `comment_count` (the target number of top-level comments, default 50), and `sort_order` (top for the popular ordering, recent for newest first). The bot then navigates to the video, pulls the title, channel, view count, like count, publish date and duration straight from the player JSON embedded in the page - the same data the page itself renders from, which is far more stable than scraping visible text - and scrolls the comments section until your target is met or the feed ends. Everything lands locally. The configuration does not pin a custom export filename, so each dataset is written as a CSV named for its dataset key, `video_metrics` and `comments`, in the launcher's output folder. Because the run happens on your desktop, your research interests - which channels you study, which videos you batch, what you do with the sentiment in the comments - never pass through anyone else's servers. Two honesty notes up front. First, this build collects top-level threads only; it never expands reply chains. Second, it ships without proxy rotation or CAPTCHA solving, so if a verification wall appears the run logs it and stops rather than fighting through - the failure paths are described further down. ## Who uses a YouTube comment scraper or comment export tool? The **content strategist** asks "what did the audience actually say about this launch video?" and needs a dated CSV of newest-first comments to code themes against, not a screenshot gallery. The **academic researcher** asks "can I build a reproducible corpus of comments on these twenty videos for a published study?" and needs batch list mode, exact field provenance and a dataset they collected themselves and can cite. The **creator** asks "how do my like counts and comment volumes compare across my last ten uploads?" and needs the metrics row per video, re-collected on a schedule they control. All three want an artifact they produced and can defend - rows with `video_id`, `page_url` and collection context attached - rather than a dashboard inside someone else's web app. ## How does a YouTube metrics collector run work, step by step? Every command in the run is linked to its reference page. First, three [set_cookie](https://learn.rtila.com/reference/commands/set_cookie) calls fire before any navigation: two consent cookies that preempt the EU cookie interstitial, and one pinning the interface to English. A [set_viewport_size](https://learn.rtila.com/reference/commands/set_viewport_size) call fixes the window at 480x940, and [goto](https://learn.rtila.com/reference/commands/goto) loads the watch page - with image, media and font resources blocked for speed. A [try_catch](https://learn.rtila.com/reference/commands/try_catch) block dismisses any consent dialog that still appears, then [wait_for_selector](https://learn.rtila.com/reference/commands/wait_for_selector) confirms the player shell has rendered. Phase one is metrics: an [execute_script](https://learn.rtila.com/reference/commands/execute_script) step reads the embedded player response and initial-data objects, a bounded search digs out the like count, and eight [set_variable](https://learn.rtila.com/reference/commands/set_variable) calls park the values so the `video_metrics` dataset can capture them via variable selectors; [extract_data](https://learn.rtila.com/reference/commands/extract_data) writes the row. If the embedded JSON is ever unavailable, a METRICS_DEGRADED marker is logged and the run continues to comments. Phase two prepares the scroll: a script strips the related-videos sidebar and footer so the document bottom is the end of the comments section, one [scroll](https://learn.rtila.com/reference/commands/scroll) teleports there to hydrate every lazy sentinel, then a [while](https://learn.rtila.com/reference/commands/while) loop climbs back in gradual 600-pixel steps - gated by [is_visible](https://learn.rtila.com/reference/commands/is_visible) checks for either the first comment thread or the comments-disabled message, capped at 40 tries - because the comment section only mounts when its sentinel is sampled by an intersection observer, never on a jump. Phase three sorts: an [if](https://learn.rtila.com/reference/commands/if) branch opens the sort menu and [click](https://learn.rtila.com/reference/commands/click)s Top comments or Newest first explicitly on every run, with [scroll_into_view](https://learn.rtila.com/reference/commands/scroll_into_view) first and a text-based fallback behind the position-based click. Phase four collects: [infinite_scroll](https://learn.rtila.com/reference/commands/infinite_scroll) drives the page with a two-second delay per step, its `do` block re-stripping the sidebar as insurance, re-extracting the deduplicated `comments` dataset each cycle, and issuing a [break](https://learn.rtila.com/reference/commands/break) the moment the running total reaches `comment_count`. A final comment line logs the totals for the run. ## Getting YouTube comments to CSV: what lands in each column The metrics table mirrors the published sample output exactly: | video_id | title | channel | views | likes | published | duration_seconds | |---|---|---|---|---|---|---| | abcdefghijk | Restoring a 1972 field recorder - full teardown | Example Field Notes | 184322 | 9214 | 2026-08-14 | 942 | **video_id** and **channel_id** are the platform's own stable identifiers - join on them across runs and batches rather than on titles, which creators edit. **title**, **channel**, **views**, **likes**, **published** and **duration_seconds** come from the embedded player JSON; views and likes are captured as displayed strings, so strip any formatting in your spreadsheet rather than expecting pre-parsed numbers. **description** is the open-graph summary, kept for context. **page_url** records the exact watch page the row came from, which is what makes a multi-video batch auditable. The comments dataset carries its own columns: | author | author_url | text | likes | published | replies | |---|---|---|---|---|---| | Example Viewer | https://www.youtube.com/@exampleviewer | Great teardown - the belt replacement tip saved mine. | 42 | 3 weeks ago | 5 replies | **author** and **text** are trimmed verbatim. **author_url** is normalised to an absolute channel link by a transformation script. **likes** is the one normalised field: a script converts display shorthand (1.2K, 3M) into plain integers, defaulting to 0 when no count is shown. **published** is the platform's relative label as displayed - "3 weeks ago" - kept raw so the run is auditable against the page. **replies** is the raw reply-toggle label, not reply content. Deduplication runs on author plus text plus published, which is what lets overlapping scroll batches append safely. ## Can it serve as a video analytics scraper for channel research? Yes, within its shape. A video analytics scraper ultimately answers "how did engagement on these videos move, and what are people saying?" - and a batch run over a channel's recent uploads, re-run monthly, gives you exactly that: views, likes and comment volume per `video_id` over time, plus the newest-first comment corpus to read against the numbers. What it will not do is chart anything, alert you, or re-run itself; schedule the launcher with your OS scheduler and diff the CSVs. If you need live dashboards and automated reporting, that is a product category, not this bot - the trade is set out below. ## How does it compare to manual copying and hosted collection? Hand-copying comments works for twenty threads and collapses at two hundred, produces no like counts or timestamps worth trusting, and cannot be repeated identically next month. This bot produces the same deduplicated CSV on your own machine for nothing per run, and the workflow is a file you keep. Against hosted export websites and metered collection platforms - the ones that charge per comment, per record or per month and keep your exports in their cloud - the difference is structural: here the data lands on your disk, the retention and compliance decisions are explicitly yours, and a bigger comment target costs only your own time and bandwidth. Against the platform's official developer API, the trade is different again: the API is the sanctioned route but runs on a shared daily unit budget that makes large comment pulls a quota-accounting exercise, and it requires a developer project at all. This bot reads the public page like a viewer. The honest costs of local collection: no dashboards, no scheduling, no history, and selector-level maintenance when the page changes. ## How do you set it up on Windows, macOS and Linux? Download the launcher for your platform from this page - a single raw file, no archive and no extraction step anywhere. On **Windows**, run the downloaded .exe directly; SmartScreen may ask for confirmation on first launch, and "More info" then "Run anyway" proceeds once you have verified the published checksum. On **macOS**, two launchers are published - check Apple menu, About This Mac for Apple Silicon or Intel and pick the matching one, since running the wrong build is the most common first-launch mistake. Make it executable with `chmod +x`, clear the Gatekeeper quarantine with `xattr -d com.apple.quarantine /path/to/youtube-metrics-comments-collector`, then launch. On **Linux**, the launcher is a single .sh file: `chmod +x ./youtube-metrics-comments-collector` then `./youtube-metrics-comments-collector` - the executable bit can be lost in transfer, so keep the chmod step. Do not rename the file: the filename carries the configuration id. First run asks for the three inputs (video URL, comment target, sort order); for unattended repeats, schedule the launcher with Task Scheduler on Windows, launchd on macOS or cron on Linux. ## What do I check when a run returns zero rows? In order. Metrics empty: is the URL a public watch page that loads in an ordinary logged-out browser, or an embed, age-restricted or region-locked video? A METRICS_DEGRADED marker in the log means the embedded JSON route failed - the comments half may still be fine. Comments empty with a COMMENTS_UNAVAILABLE marker: the section is disabled, restricted, or never rendered in time - confirm by opening the page yourself. Fewer comments than the target but a non-zero file: the video simply has fewer top-level threads than you asked for, or the scroll cap derived from `comment_count` was reached - raise the target and re-run. Empty likes columns are normal: many threads display no like count, and the transformation defaults them to 0. A run that stops at a verification wall is honest, not broken - this build deliberately ships no solver, so the answer is to slow down, not to fight through. The contract throughout: failure is loud and logged, never fabricated data. ## What are the limits of the bot? It collects what a logged-out viewer can see on one watch page at a time: metrics from the embedded player data and top-level comment threads. It does not expand replies, collect channel pages, playlists, search results or Shorts-specific surfaces, and it does not download video or audio. It has no proxy rotation or CAPTCHA solving in this build, no scheduling of its own, and no alerting - it is a snapshotter, not a monitor. Batch mode processes URLs sequentially with one worker. The comment target is a floor guarantee, not an exact cut: expect overshoot of up to one loading batch. And the metrics route depends on the page's embedded JSON staying present; when it does not, the bot says so in the log rather than guessing. ## What is responsible comment collection? Comments are user-posted content tied to named or pseudonymous accounts - unambiguously personal data under GDPR, UK GDPR and similar regimes. Establish your lawful basis before repurposing anything, honour deletion and opt-out requests immediately, and never use author names or channel links for unsolicited outreach, spam or profiling of individuals. Review the platform's terms of service with open eyes: automated collection may conflict with them, and the decision to run is yours to own. Keep request rates at what a patient human could sustain - the two-second scroll delay and single-worker design are there for exactly that reason - and keep your corpus scoped to what your research or reporting actually needs. The acceptable-use policy applies in full before your first run. ### Input fields | Name | Kind | Required | Default | Description | | --- | --- | --- | --- | --- | | video_url | text | yes | | Declared in project settings. | | comment_count | text | yes | 50 | Declared in project settings. | | sort_order | text | yes | top | Declared in project settings. | | start_urls | url | yes | - | Start URLs the bot navigates to. | ### Output **video_metrics** (item selector: `css=html`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | video_id | text | no | variable=m_video_id | | channel_id | text | no | variable=m_channel_id | | title | text | no | variable=m_title | | channel | text | no | variable=m_channel | | views | text | no | variable=m_views | | likes | text | no | variable=m_likes | | published | text | no | variable=m_published | | duration_seconds | text | no | variable=m_duration | | description | attribute | no | css=meta[property='og:description'] | | page_url | page_url | no | - | **comments** (item selector: `css=ytd-comment-thread-renderer`) | Field | Type | Computed | Selector | | --- | --- | --- | --- | | author | text | no | css=#author-text | | author_url | attribute | no | css=#author-text | | text | text | no | css=#content-text | | likes | text | no | css=#vote-count-middle | | published | text | no | css=.published-time-text, #published-time-text | | replies | text | no | css=#more-replies, #more-text | | page_url | page_url | no | - | Export formats: csv, json. ### Downloads - Windows x86_64: `BOT_youtube-metrics-comments-collector.exe` (67678272 bytes, sha256 4da79ab883cfa86edf236d44f09576eee9c72cf8c0922e1c31f3d18a45f29477) - https://mp.rtila.net/d/youtube-metrics-comments-collector/windows - macOS Apple Silicon: `BOT_youtube-metrics-comments-collector.sh` (60820144 bytes, sha256 3d8e83ec2231d6d283d957bc01e23cf3a59587f14d02449609d1c696ef9289b8) - https://mp.rtila.net/d/youtube-metrics-comments-collector/macos - macOS Intel: `BOT_youtube-metrics-comments-collector.sh` (68022320 bytes, sha256 cf5d493d8a06906728be9fd6ffdc4f365d7fcbfc844a63360f146bcdaaf61162) - https://mp.rtila.net/d/youtube-metrics-comments-collector/macos_intel - Linux x86_64: `BOT_youtube-metrics-comments-collector.sh` (86265752 bytes, sha256 56aa6f40dae43fa52cb4c49de3c2b2296ff5097462338fa4a8337a5a962f0c4f) - https://mp.rtila.net/d/youtube-metrics-comments-collector/linux ### Configuration Config-fetch URL: `https://mp.rtila.net/?id=youtube-metrics-comments-collector` Configuration document: `https://marketplace.rtila.com/api/bots/youtube-metrics-comments-collector/config.json` - Encoding: plain - Digest: a331f5367f9707ecdbf27df35c164756245cbd99c2ec534746235fb4e5562454 - Bytes: 21709 ```json { "name": "YouTube Metrics and Comments Collector", "settings": { "urls": [ "${video_url}" ], "headless": false, "navigation_timeout": 60000, "humanoidEnabled": false, "proxyEnabled": false, "captchaEnabled": false, "matchTimezoneAndLocale": false, "block_resources": false, "get_urls_from_sitemap": false, "respect_robots_txt": false, "skip_initial_navigation": true, "variables": [ { "name": "video_url", "value": "", "required": true, "is_list": false }, { "name": "comment_count", "value": "50", "required": true, "is_list": false }, { "name": "sort_order", "value": "top", "required": true, "is_list": false } ] }, "datasets": { "video_metrics": { "item_selector": "css=html", "properties": [ { "name": "video_id", "type": "text", "selector": "variable=m_video_id" }, { "name": "channel_id", "type": "text", "selector": "variable=m_channel_id" }, { "name": "title", "type": "text", "selector": "variable=m_title" }, { "name": "channel", "type": "text", "selector": "variable=m_channel" }, { "name": "views", "type": "text", "selector": "variable=m_views" }, { "name": "likes", "type": "text", "selector": "variable=m_likes" }, { "name": "published", "type": "text", "selector": "variable=m_published" }, { "name": "duration_seconds", "type": "text", "selector": "variable=m_duration" }, { "name": "description", "type": "attribute", "selector": "css=meta[property='og:description']", "attribute": "content" }, { "name": "page_url", "type": "page_url" } ], "required_fields": [ "title" ] }, "comments": { "item_selector": "css=ytd-comment-thread-renderer", "properties": [ { "name": "author", "type": "text", "selector": "css=#author-text", "transformations": [ { "type": "trim" } ] }, { "name": "author_url", "type": "attribute", "selector": "css=#author-text", "attribute": "href", "transformations": [ { "type": "script", "code": "if(!value) return ''; var v=String(value); return v.indexOf('http')===0 ? v : 'https://www.youtube.com'+(v.charAt(0)==='/'?'':'/')+v;" } ] }, { "name": "text", "type": "text", "selector": "css=#content-text", "transformations": [ { "type": "trim" } ] }, { "name": "likes", "type": "text", "selector": "css=#vote-count-middle", "transformations": [ { "type": "script", "code": "if(!value) return '0'; var m=String(value).match(/([\\d.,]+)\\s*([KM]?)/i); if(!m) return '0'; var n=parseFloat(m[1].replace(/,/g,'')); if(isNaN(n)) return '0'; var s=(m[2]||'').toUpperCase(); if(s==='K')n*=1e3; if(s==='M')n*=1e6; return String(Math.round(n));" } ] }, { "name": "published", "type": "text", "selector": "css=.published-time-text, #published-time-text", "transformations": [ { "type": "trim" } ] }, { "name": "replies", "type": "text", "selector": "css=#more-replies, #more-text", "transformations": [ { "type": "trim" } ] }, { "name": "page_url", "type": "page_url" } ] } }, "commands": [ { "command": "set_cookie", "params": { "name": "SOCS", "value": "CAI", "url": "https://www.youtube.com" } }, { "command": "set_cookie", "params": { "name": "CONSENT", "value": "YES+cb", "url": "https://www.youtube.com" } }, { "command": "set_cookie", "params": { "name": "PREF", "value": "hl=en", "url": "https://www.youtube.com" } }, { "command": "set_viewport_size", "params": { "width": 480, "height": 940 } }, { "command": "goto", "params": { "url": "${last_target_url}" } }, { "command": "try_catch", "params": { "try": [ { "command": "wait_for_selector", "params": { "selector": "css=button[aria-label*='Reject all'], button[aria-label*='Accept all']", "timeout": 4000 } }, { "command": "click", "params": { "selector": "css=button[aria-label*='Reject all'], button[aria-label*='Accept all']", "timeout": 5000 } } ], "catch": [ { "command": "comment", "params": { "text": "No consent interstitial (cookies worked or non-EU)." } } ] } }, { "command": "wait_for_selector", "params": { "selector": "css=ytd-watch-flexy", "timeout": 45000 } }, { "command": "set_viewport_size", "params": { "width": 480, "height": 940 } }, { "command": "comment", "params": { "text": "PHASE 1: metrics from embedded JSON (stable route)." } }, { "command": "try_catch", "params": { "try": [ { "command": "execute_script", "output_variable": "m", "params": { "script": "var p=window.ytInitialPlayerResponse||{};var v=p.videoDetails||{};var mf=(p.microformat||{}).playerMicroformatRenderer||{};var d=window.ytInitialData||{};var likes='';(function w(o,dep){if(likes||!o||typeof o!=='object'||dep>12)return;for(var k in o){if(/likeCount/i.test(k)&&typeof o[k]==='string'&&/[0-9]/.test(o[k])){likes=o[k];return;}w(o[k],dep+1);}})(d,0);return {video_id:String(v.videoId||''),channel_id:String(mf.channelId||''),title:String(v.title||document.title||''),channel:String(v.author||mf.ownerChannelName||''),views:String(v.viewCount||''),likes:String(likes||''),published:String(mf.publishDate||''),duration:String(v.lengthSeconds||'')};" } }, { "command": "set_variable", "params": { "name": "m_video_id", "value": "${m.video_id}" } }, { "command": "set_variable", "params": { "name": "m_channel_id", "value": "${m.channel_id}" } }, { "command": "set_variable", "params": { "name": "m_title", "value": "${m.title}" } }, { "command": "set_variable", "params": { "name": "m_channel", "value": "${m.channel}" } }, { "command": "set_variable", "params": { "name": "m_views", "value": "${m.views}" } }, { "command": "set_variable", "params": { "name": "m_likes", "value": "${m.likes}" } }, { "command": "set_variable", "params": { "name": "m_published", "value": "${m.published}" } }, { "command": "set_variable", "params": { "name": "m_duration", "value": "${m.duration}" } }, { "command": "extract_data", "params": { "dataset": "video_metrics" } } ], "catch": [ { "command": "comment", "params": { "text": "METRICS_DEGRADED: embedded JSON unavailable; comments will still be collected." } } ] } }, { "command": "comment", "params": { "text": "PHASE 2: strip related-videos column + footer so the comments section owns the document bottom. This is what makes infinite_scroll's bottom-teleport land ON the comment continuation spinner instead of overshooting into lazily-loaded related videos." } }, { "command": "execute_script", "params": { "script": "var removed=[];var sec=document.querySelector('ytd-watch-flexy #secondary')||document.querySelector('#secondary');if(sec){sec.remove();removed.push('secondary');}var rel=document.querySelector('ytd-watch-next-secondary-results-renderer');if(rel){rel.remove();removed.push('related');}var foot=document.querySelector('tp-yt-app #footer')||document.querySelector('ytd-app #footer');if(foot){foot.remove();removed.push('footer');}return removed.join(',');" }, "output_variable": "stripped" }, { "command": "comment", "params": { "text": "PHASE 2b: teleport to true document bottom ONCE so every comment-section sentinel intersects (full hydration + interactive sort header + armed continuation spinner). Safe only because #secondary was just stripped — bottom == end of comments." } }, { "command": "scroll", "params": { "selector": "css=html", "direction": "bottom", "delay": 0 } }, { "command": "wait", "params": { "timeout": 1500 } }, { "command": "comment", "params": { "text": "PHASE 3: force-mount lazy comment section. Gradual 600px steps (never teleport) so YouTube's IntersectionObserver samples the sentinel; dual visibility gates exit on first thread OR on 'comments off' message." } }, { "command": "set_variable", "params": { "name": "threads_visible", "value": "false" } }, { "command": "set_variable", "params": { "name": "comments_off", "value": "false" } }, { "command": "set_variable", "params": { "name": "mount_tries", "value": "0" } }, { "command": "while", "params": { "condition": { "type": "script", "expression": "String(variables.threads_visible) !== 'true' && String(variables.comments_off) !== 'true' && Number(variables.mount_tries) < 40" }, "do": [ { "command": "scroll", "params": { "direction": "by_pixels", "amount": 600, "delay": 0 } }, { "command": "wait", "params": { "timeout": 450 } }, { "command": "is_visible", "params": { "selector": "css=ytd-comment-thread-renderer" }, "output_variable": "threads_visible" }, { "command": "is_visible", "params": { "selector": "css=ytd-comments #message, ytd-comments ytd-message-renderer" }, "output_variable": "comments_off" }, { "command": "set_variable", "params": { "name": "mount_tries", "value": "${Number(mount_tries) + 1}" } } ] } }, { "command": "set_variable", "params": { "name": "comments_unavailable", "value": "false" } }, { "command": "try_catch", "params": { "try": [ { "command": "wait_for_selector", "params": { "selector": "css=ytd-comment-thread-renderer", "timeout": 10000 } } ], "catch": [ { "command": "set_variable", "params": { "name": "comments_unavailable", "value": "true" } } ] } }, { "command": "if", "params": { "condition": { "type": "script", "expression": "String(variables.comments_unavailable) !== 'true'" }, "then": [ { "command": "comment", "params": { "text": "PHASE 4: explicit sort selection (both branches click — never trust the channel's default ordering). Position-based nth click first (language-neutral), exact-text fallback second (guaranteed English via PREF hl=en cookie)." } }, { "command": "try_catch", "params": { "try": [ { "command": "scroll_into_view", "params": { "selector": "css=ytd-comments-header-renderer", "timeout": 4000 } }, { "command": "if", "params": { "condition": { "type": "script", "expression": "String(variables.sort_order || '').toLowerCase().indexOf('recent') > -1 || String(variables.sort_order || '').toLowerCase().indexOf('new') > -1" }, "then": [ { "command": "click", "params": { "selector": "css=ytd-comments-header-renderer tp-yt-paper-dropdown-menu-trigger #icon-button, ytd-comments-header-renderer tp-yt-paper-dropdown-menu-trigger, ytd-comments-header-renderer button[aria-label*='Sort'], #sort-menu #icon-button", "timeout": 8000 } }, { "command": "wait", "params": { "timeout": 1200 } }, { "command": "try_catch", "params": { "try": [ { "command": "click", "params": { "selector": "css=ytd-comments-header-renderer tp-yt-paper-listbox a, ytd-comments-header-renderer tp-yt-paper-listbox tp-yt-paper-item >> nth=1", "timeout": 5000 } } ], "catch": [ { "command": "click", "params": { "selector": "text=Newest first", "timeout": 5000 } } ] } }, { "command": "wait", "params": { "timeout": 2500 } }, { "command": "wait_for_selector", "params": { "selector": "css=ytd-comment-thread-renderer", "timeout": 20000 } } ], "else": [ { "command": "click", "params": { "selector": "css=ytd-comments-header-renderer tp-yt-paper-dropdown-menu-trigger #icon-button, ytd-comments-header-renderer tp-yt-paper-dropdown-menu-trigger, ytd-comments-header-renderer button[aria-label*='Sort'], #sort-menu #icon-button", "timeout": 8000 } }, { "command": "wait", "params": { "timeout": 1200 } }, { "command": "try_catch", "params": { "try": [ { "command": "click", "params": { "selector": "css=ytd-comments-header-renderer tp-yt-paper-listbox a, ytd-comments-header-renderer tp-yt-paper-listbox tp-yt-paper-item >> nth=0", "timeout": 5000 } } ], "catch": [ { "command": "click", "params": { "selector": "text=Top comments", "timeout": 5000 } } ] } }, { "command": "wait", "params": { "timeout": 2500 } }, { "command": "wait_for_selector", "params": { "selector": "css=ytd-comment-thread-renderer", "timeout": 20000 } } ] } } ], "catch": [ { "command": "comment", "params": { "text": "SORT_DEGRADED: menu interaction failed; continuing with default ordering." } } ] } }, { "command": "comment", "params": { "text": "PHASE 5: collect. infinite_scroll on css=html (the standards-mode scrollingElement; body.scrollHeight is unreliable on YouTube). do-block: re-strip insurance, deduped extract, exact-N break. Current engine checkpoints variables+memory every cycle, so crash-resume never duplicates rows." } }, { "command": "set_variable", "params": { "name": "total", "value": "0" } }, { "command": "infinite_scroll", "params": { "selector": "css=html", "max_scrolls": "${Math.max(12, Math.ceil(Number(comment_count) / 10) + 8)}", "scroll_delay": 2000, "do": [ { "command": "execute_script", "params": { "script": "var s=document.querySelector('ytd-watch-flexy #secondary')||document.querySelector('#secondary');if(s)s.remove();var f=document.querySelector('tp-yt-app #footer')||document.querySelector('ytd-app #footer');if(f)f.remove();return true;" } }, { "command": "extract_data", "params": { "dataset": "comments" }, "output_variable": "fresh" }, { "command": "set_variable", "params": { "name": "total", "value": "${Number(total) + (Array.isArray(fresh) ? fresh.length : 0)}" } }, { "command": "if", "params": { "condition": { "type": "script", "expression": "Number(variables.total) >= Number(variables.comment_count)" }, "then": [ { "command": "break", "params": {} } ], "else": [] } } ] } }, { "command": "comment", "params": { "text": "Collection finished: total=${total} requested=${comment_count} sort=${sort_order} mount_tries=${mount_tries} stripped=${stripped}" } } ], "else": [ { "command": "comment", "params": { "text": "COMMENTS_UNAVAILABLE: comment section never rendered (disabled, restricted, bot-walled, or blocked). Metrics were still collected." } } ] } } ], "authors": [] } ``` ### FAQ **How do I export YouTube comments to CSV?** Set the video_url input to a public watch page, choose a comment target and a sort order, and run the launcher. The bot scrolls the comment section, extracts each thread into the comments dataset, and saves the result as a CSV in the launcher output folder on your machine - one row per top-level comment, with author, text, normalised like count, published label and the page URL. No cloud account and no conversion step are involved. **How many comments can one run collect?** As many as you ask for, bounded by what the video actually has. The comment_count input defaults to 50 top-level comments; the scroll cap is derived from it automatically (at least 12 scroll steps, plus roughly one step per ten requested comments). Raise the target for deep comment sections; the run stops as soon as the collected total reaches it. **Why does the final comment count exceed the target?** YouTube loads comments in batches of roughly twenty threads. The collector checks the total after each extracted batch and stops once the target is met, so the final file can overshoot by up to one batch - requesting 50 may save up to about 64 rows. This is deliberate: cutting mid-batch would silently drop comments the page had already rendered. **Does it collect comment replies?** No. The bot collects top-level comment threads only and never expands reply chains. The replies column records the raw label shown under a thread (the view-replies toggle text) where one exists, so you can see that replies exist and roughly how many, but reply content itself is out of scope for this build. **Can I scrape multiple videos at once?** Yes. The video_url input supports list mode: enable it and paste one watch-page URL per line, and the engine processes each URL in turn with the same comment target and sort order. Every output row carries its own page_url, video_id and channel_id, so a multi-video batch stays separable after export. **What happens when comments are disabled?** The run stays useful and fails loud. The mounting loop watches for the comments-disabled message; if it appears, or if no comment thread renders in time, the bot logs a COMMENTS_UNAVAILABLE marker and skips collection. The video metrics row is still saved, so you always get the statistics half of the dataset. **How do I sort comments by newest first?** Set the sort_order input to recent (the aliases new and newest are also accepted); the default top collects the popular ordering. The bot never trusts the channel default ordering - it opens the sort menu and clicks the matching entry on every run, with a position-based click first and an exact-English-text fallback second, because the English-locale cookie is pinned before navigation. **Why does the bot set cookies before loading the video?** Three cookies are set first: two consent cookies that preempt the EU cookie interstitial, and one that pins the interface language to English. This keeps runs unattended (no consent dialog to dismiss mid-scroll) and makes the English text fallbacks in the sort menu reliable. If an interstitial still appears, a try/catch block clicks the reject-all or accept-all button and carries on. **How are duplicate comments handled?** The comments dataset de-duplicates on the combination of author, text and published label. Because collection overlaps batches while scrolling - each cycle re-extracts everything currently rendered - deduplication is what makes appending safe: a comment seen in three cycles lands in the CSV exactly once. **Is scraping YouTube comments allowed?** That is your call to own, not the bot's. Automated collection may conflict with the platform's terms of service, and comment text plus author names are personal data under GDPR, UK GDPR and similar regimes. Review the terms, establish a lawful basis before repurposing anything, keep collection to what a patient human viewer could read, and never use the output for spam or profiling. **Does the bot need an internet connection to run?** Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts. **Can I rename the downloaded file?** No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch.