Skip to main content
RTILA Marketplace

Link Crawler - project configuration

Crawl internal links on a website and collect meta titles, status codes and headers into one CSV audit file.

SEO Beginner Stable v1.1.0 Updated Updated 2026-09-14 9 min read
Download

  • Windows x86_64

    BOT_link-crawler.exe

    98.9 MB

    Download

  • macOS Apple Silicon chmod +x

    BOT_link-crawler.sh

    87.5 MB

    Download

  • macOS Intel chmod +x

    BOT_link-crawler.sh

    99.4 MB

    Download

  • Linux x86_64 chmod +x

    BOT_link-crawler.sh

    122.5 MB

    Download

Do not rename the downloaded file - the filename carries the bot's configuration id.

This tab presents the complete project configuration exactly as published - the same JSON the launcher fetches at launch - ready to read, copy, or import into the desktop product. Back to the overview tab.

Project configuration

Full project JSON for link-crawler

Canonical byte length: 2,358 - the length of the canonical form the digest covers.

Canonical digest: dbb0b3bf63d1504615d183b9db980ede7ab1fdeb950807136cfa9c3d83065084

About the delivered configuration encoding

At launch the backend delivers this configuration to the launcher as plain JSON. The block below is the complete project definition the launcher receives.

What you are reading has machine-readable equivalents, and the JSON at those routes is the same document as the JSON rendered on this page: the listing overview, the per-listing detail document and the configuration document. The digest and byte count shown in the panel header are computed over that document's canonical JSON form - recursively sorted keys, two-space indentation, no trailing newline - which is the form the provisioning pipeline signs and the reconcile control re-digests; it differs from the highlighted block above in key order and by that trailing newline, so hashing the visible text will not reproduce the digest.

Download this JSON 
{
  "name": "Link Crawler",
  "settings": {
    "urls": [
      "${target_url}"
    ],
    "headless": false,
    "humanoidEnabled": false,
    "proxyEnabled": false,
    "captchaEnabled": false,
    "matchTimezoneAndLocale": false,
    "block_resources": false,
    "get_urls_from_sitemap": false,
    "respect_robots_txt": false,
    "skip_initial_navigation": false,
    "variables": [
      {
        "name": "target_url",
        "value": "https://rtila.com",
        "required": true,
        "is_list": false
      }
    ]
  },
  "datasets": {
    "page_metadata": {
      "item_selector": "css=html",
      "properties": [
        {
          "name": "meta_title",
          "type": "text",
          "selector": "css=head title",
          "transformations": [
            {
              "type": "trim"
            }
          ]
        },
        {
          "name": "url",
          "type": "text",
          "selector": "variable=last_navigation_url"
        },
        {
          "name": "status",
          "type": "text",
          "selector": "variable=last_http_status"
        },
        {
          "name": "content_type",
          "type": "text",
          "selector": "variable=last_content_type"
        },
        {
          "name": "charset",
          "type": "text",
          "selector": "variable=last_charset"
        },
        {
          "name": "content_length",
          "type": "text",
          "selector": "variable=last_content_length"
        },
        {
          "name": "server",
          "type": "text",
          "selector": "variable=last_server"
        },
        {
          "name": "cache_control",
          "type": "text",
          "selector": "variable=last_cache_control"
        },
        {
          "name": "x_robots_tag",
          "type": "text",
          "selector": "variable=last_x_robots_tag"
        }
      ],
      "required_fields": [
        "meta_title"
      ]
    }
  },
  "commands": [
    {
      "command": "crawl_links",
      "params": {
        "concurrency": 7,
        "delay": 1000,
        "link_type": "internal",
        "max_depth": 3,
        "max_urls": 999,
        "do": [
          {
            "command": "extract_data",
            "params": {
              "dataset": "page_metadata"
            }
          }
        ]
      }
    }
  ],
  "authors": []
}

Related listings

SEO beginner Free

Outbound Link Checker

Audit page links in bulk: internal vs external counts, link schemes and navigation health for every URL.

seo link-audit outbound-links excel sheets

1-10 minutes depending on URL count and page size Uses 1 commands Exports csv, json, xlsx v1.0.0

SEO beginner Free

Redirect Checker

Check a URL list for redirects: status codes, chains, final destinations and timing in one report.

seo-audit redirects http-status excel sheets

1-10 minutes depending on list size and response times Uses 1 commands Exports csv, json, xlsx v1.0.0

SEO beginner experimental Free

SEO Rank Tracker

Check where a domain ranks in web search results for a list of keywords and export the positions to CSV.

rank-tracking keyword-ranks seo-audit excel sheets

1-10 minutes depending on keyword count Uses 1 commands Exports csv, json, xlsx v1.0.0

Frequently asked questions

How do I crawl all internal links on a website?
Enter the start URL when the launcher asks for target_url. The crawl_links command discovers internal links on every page it visits and follows them up to depth 3, capped at 999 URLs per run, extracting one metadata row per page as it goes. External links are recorded as out of scope and not followed, so one run gives you the internal graph of a single site.
How do I find missing meta titles on my site?
Run a crawl and filter the meta_title column for blanks in any spreadsheet. Because meta_title is a required field in the page_metadata dataset, rows with a missing title are logged during the run and still kept in the output, so a blank cell means the page genuinely served no title element at load time rather than the row being dropped.
What is max depth in a link crawl?
Depth counts link hops from the start URL: depth 0 is the page you entered, depth 1 is everything it links to, and so on. This bot stops at depth 3, which is deliberate - pages buried deeper than three clicks from the homepage are usually hard for search engines and users to reach too. The 999 URL cap applies on top of the depth limit, whichever is hit first ends the crawl.
How do I check x robots tag headers across a site?
Every visited page records its x_robots_tag response header into the x_robots_tag column. After the crawl, filter that column for noindex, nofollow or noarchive values to see exactly which URLs are being excluded from indexing by server level directives - the kind of instruction that overrides on page meta tags and is easy to leave behind after a migration.
Can the crawler find broken pages by http status?
Yes. The status column records the HTTP response code for every visited URL, so filtering for 404, 410, 500 and other non 200 values gives you a broken page report in one pass. Soft 404s - pages that return 200 but show an error - are not detected; check suspicious 200 rows by their meta title and content length.
How many urls can one crawl handle?
Up to 999 URLs per run, which is the max_urls bound in the crawl_links configuration. With seven concurrent fetches and a one second delay between requests, a full 999 URL crawl typically finishes in a few minutes on a responsive host. Larger sites can be audited in sections by starting separate runs from different entry URLs.
Does the crawler visit external links?
No. The crawl is configured with link_type internal, so it follows only links that stay on the same site. External links are neither visited nor added to the row count. That keeps the run a site audit rather than a web crawl, and keeps request volume directed at one host you are responsible for treating politely.
How do I export crawl results to CSV?
The final extract_data step consolidates the page_metadata dataset and the standalone launcher writes it out as a CSV file you can open directly in Excel or Sheets - no conversion step. The same data can be exported to JSON or XLSX from the launcher export options if your workflow prefers those formats.
Why are duplicate urls removed from the crawl?
The page_metadata dataset de-duplicates on the url field, and the crawler tracks visited URLs as it goes, so each address appears exactly once even when dozens of pages link to it. Query parameter variants of the same page are treated as distinct URLs, which is intentional - spotting parameter duplication is one of the useful outcomes of an audit.
Does the link crawler work on JavaScript sites?
Partially. Pages are processed at the load state in a real browser, so titles and headers rendered by that point are captured. Sites that render their content or navigation only after load may produce rows with an empty meta_title or a shallow link graph, because links injected later are never discovered. A visible browser window opens during the run so you can see exactly what the crawler sees.
Does the bot need an internet connection to run?
Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts.
Can I rename the downloaded file?
No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch.