How do I crawl all internal links on a website?
Enter the start URL when the launcher asks for target_url. The crawl_links command discovers internal links on every page it visits and follows them up to depth 3, capped at 999 URLs per run, extracting one metadata row per page as it goes. External links are recorded as out of scope and not followed, so one run gives you the internal graph of a single site.
How do I find missing meta titles on my site?
Run a crawl and filter the meta_title column for blanks in any spreadsheet. Because meta_title is a required field in the page_metadata dataset, rows with a missing title are logged during the run and still kept in the output, so a blank cell means the page genuinely served no title element at load time rather than the row being dropped.
What is max depth in a link crawl?
Depth counts link hops from the start URL: depth 0 is the page you entered, depth 1 is everything it links to, and so on. This bot stops at depth 3, which is deliberate - pages buried deeper than three clicks from the homepage are usually hard for search engines and users to reach too. The 999 URL cap applies on top of the depth limit, whichever is hit first ends the crawl.
How do I check x robots tag headers across a site?
Every visited page records its x_robots_tag response header into the x_robots_tag column. After the crawl, filter that column for noindex, nofollow or noarchive values to see exactly which URLs are being excluded from indexing by server level directives - the kind of instruction that overrides on page meta tags and is easy to leave behind after a migration.
Can the crawler find broken pages by http status?
Yes. The status column records the HTTP response code for every visited URL, so filtering for 404, 410, 500 and other non 200 values gives you a broken page report in one pass. Soft 404s - pages that return 200 but show an error - are not detected; check suspicious 200 rows by their meta title and content length.
How many urls can one crawl handle?
Up to 999 URLs per run, which is the max_urls bound in the crawl_links configuration. With seven concurrent fetches and a one second delay between requests, a full 999 URL crawl typically finishes in a few minutes on a responsive host. Larger sites can be audited in sections by starting separate runs from different entry URLs.
Does the crawler visit external links?
No. The crawl is configured with link_type internal, so it follows only links that stay on the same site. External links are neither visited nor added to the row count. That keeps the run a site audit rather than a web crawl, and keeps request volume directed at one host you are responsible for treating politely.
How do I export crawl results to CSV?
The final extract_data step consolidates the page_metadata dataset and the standalone launcher writes it out as a CSV file you can open directly in Excel or Sheets - no conversion step. The same data can be exported to JSON or XLSX from the launcher export options if your workflow prefers those formats.
Why are duplicate urls removed from the crawl?
The page_metadata dataset de-duplicates on the url field, and the crawler tracks visited URLs as it goes, so each address appears exactly once even when dozens of pages link to it. Query parameter variants of the same page are treated as distinct URLs, which is intentional - spotting parameter duplication is one of the useful outcomes of an audit.
Does the link crawler work on JavaScript sites?
Partially. Pages are processed at the load state in a real browser, so titles and headers rendered by that point are captured. Sites that render their content or navigation only after load may produce rows with an empty meta_title or a shallow link graph, because links injected later are never discovered. A visible browser window opens during the run so you can see exactly what the crawler sees.
Does the bot need an internet connection to run?
Yes on first run: the launcher fetches its configuration by id from the backend at launch, so an internet connection is required at least the first time it starts.
Can I rename the downloaded file?
No. The downloaded file must keep its exact name - the filename carries the per-bot configuration id, and renaming it breaks the configuration fetch at launch.
