Overview
The Link Crawler is a link crawler: a standalone desktop bot that follows every internal link on a target website, records what each page actually serves - meta title, HTTP status, content type, charset, size and selected response headers - into the page_metadata dataset, and exports the whole audit as a CSV file. You supply one start URL; the bot walks the internal graph from there.
What does the Link Crawler do?
You point it at a homepage or any entry page. The crawl discovers internal links on that page, fetches them, discovers the links on those pages, and keeps going to a maximum of three hops from the start, stopping at 999 URLs per run. Every page it visits contributes exactly one row: the title element from the head of the document, plus HTTP level facts read from the response itself - the status code, content type, charset, byte size, server header, cache control header, and the x robots tag header when one is sent. Because it runs on your own machine, the audit of a site - which URLs exist, which ones are broken, which ones are quietly marked noindex - never passes through a third party dashboard. One honest note up front: this is a technical metadata collector, not a content analyzer. It records what pages return; it does not read, score or rewrite what they say.
Who uses a link crawler?
The technical SEO consultant asks “are there noindexed pages that should rank, and titles missing where they should not be?” and needs the status and x robots tag columns side by side. The site owner mid migration asks “did every old URL survive the move, and are the redirects actually returning clean status codes?” and needs a dated before and after crawl. The web developer asks “which pages are serving the wrong content type or an unexpected charset?” and needs the header columns without opening a browser tab per URL. All of them want an artifact they produced - a CSV with the full URL on every row - rather than a screenshot of someone else audit UI with a project limit on it.
How does an internal link crawler run work, step by step?
Every command in the run is linked to its reference page. Before the crawl starts, the launcher collects the required project variable target_url - the start address, defaulted in the configuration to a documentation site so a first test run is never aimed anywhere unexpected - and that value feeds the start URL queue as ${target_url}. The single crawl_links command then does the walking: link_type internal, max_depth 3, max_urls 999, concurrency 7, and a delay of 1000 milliseconds between requests - seven polite parallel fetches, not a firehose. Only HTML style addresses are followed; PDFs, images and other file links are left alone. Inside the crawl do block, an extract_data step runs on every visited page against the page_metadata dataset, whose item selector is css=html - one record per whole document. The title comes from css=head title with a trim transformation; every HTTP fact comes from variable= selectors reading the response of the page just fetched (variable=last_http_status, variable=last_content_type, and so on). Rows de-duplicate on the url field, and meta_title is the dataset required field, so titleless pages are logged loudly and kept rather than silently dropped. The final export writes the consolidated dataset to CSV. The browser window stays visible during the run - headless mode is off in this configuration - so you can watch the crawl move through the site in real time.
What does the website crawl report look like?
The sample row below is the published sample output for this bot, so the table and the field notes agree by construction:
| meta_title | url | status | content_type | charset | content_length | server | cache_control | x_robots_tag |
|---|---|---|---|---|---|---|---|---|
| RTILA X - Local First AI Web Automation Software | https://rtila.com/features | 200 | text/html; charset=UTF-8 | UTF-8 | 48213 | nginx | public, max-age=3600 |
meta_title is the text of the title element exactly as served, trimmed of surrounding whitespace. Empty cells are the interesting ones: they mark pages that shipped no title at load time.
url is the full address of the visited page, read from the navigation result rather than reconstructed from the link that pointed at it - so redirects are recorded as the destination that actually answered. It is also the de-duplication key.
status is the HTTP response code. Filter this column for anything outside the 200s and 300s to get the broken page list; 301 and 302 rows let you verify redirect chains after a migration.
content_type is the served MIME type with any parameter suffix. HTML pages that arrive as application/octet-stream or plain text are configuration bugs this column surfaces instantly.
charset is the character encoding from the response. Mismatched charset values across a site are a common source of mojibake in titles and body text.
content_length is the response size in bytes as reported by the server. Wildly small values on pages that should be substantial often indicate error shells or stub pages.
server is the server header when the host chooses to send one. Many hosts suppress it - empty is normal, not a failure.
cache_control is the caching directive header. Comparing this across templates shows whether your caching policy is applied consistently or page by page by accident.
x_robots_tag is the indexation directive header, captured verbatim when present. This is the column that catches the silent killers: a stray noindex left on production after a staging migration. Empty means no directive was sent at header level, which is the expected state for most pages.
How does it work as a seo audit crawler?
A seo audit crawler ultimately answers “what exists, what answers, and what is allowed to be indexed?” - and this dataset answers all three in one pass. Existence: the URL column is the discovered internal inventory, and any page you expected to find that is absent was not reachable within three hops, which is itself a finding. Health: the status column separates live pages from broken ones. Indexation: the x robots tag column exposes server level exclusion directives. The audit is comparative over time - crawl weekly or before and after a release, then diff the files by URL to see exactly what changed. What it will not do is score page content, analyze backlinks, measure rank, or track Core Web Vitals; those are different product categories, and this bot makes no claim to them.
How do you use it as a meta title checker?
Open the CSV and work the meta_title column three ways. First, filter for blanks - every empty cell is a page serving no title. Second, sort or pivot on the title values to find duplicates; two dozen category pages sharing one template title is invisible while browsing and obvious in a sorted column. Third, add a length formula in your spreadsheet to flag titles outside the range search results display cleanly. The bot deliberately ships the title verbatim rather than “fixed” - trimming whitespace is the only transformation applied - so the checker tells you what the server said, and the editorial judgment stays yours.
How does this website crawl tool compare to hosted audit alternatives?
Hosted crawling services are generally reported to price by project or by URL allowance, keep your crawl history in their infrastructure, and cap exports by plan tier. This bot inverts all three: the crawl runs on your machine, the CSV lands on your disk, the 999 URL bound per run is a politeness setting you can re-run as often as you like, and the workflow configuration is a file you own. The trade is equally honest - there is no dashboard, no stored history, no scheduled re-crawl and no trend charts. You get dated artifacts and a spreadsheet; the analysis layer is whatever you build on top, which for most audits means a pivot table and ten minutes.
How do you set it up on Windows, macOS and Linux?
Download the launcher for your operating system from this page. On Windows, run the executable; SmartScreen may ask for confirmation on first launch, and “More info” then “Run anyway” proceeds if you trust the published checksum. On macOS, Gatekeeper may quarantine the download; right click and “Open”, or clear the flag with xattr -d com.apple.quarantine /path/to/LinkCrawler, then launch. On Linux, the launcher is a shell script: chmod +x ./link-crawler then ./link-crawler. At first run the launcher asks for the target_url value - enter the full start address including the scheme. A visible browser window opens while the crawl runs; leave it alone until the export completes. To re-run an audit weekly, schedule the launcher with your operating system scheduler (Task Scheduler, cron, or launchd) and diff the CSVs.
What do I check when a crawl returns zero rows?
In order: is the start URL a live page in an ordinary browser, or a signed in, consent walled or region locked view that a fresh session cannot reach? Did the page load but serve no HTML document the extractor could read - a pure JavaScript shell that renders after load can produce an empty title, though the row itself should still appear? Did the run end immediately because the start address redirected off site, taking the internal only crawl out of scope? The contract is that failure is loud: zero rows means the crawl could not visit pages, never that results were fabricated or quietly swallowed. For a shallow crawl - far fewer rows than the site should have - the likeliest cause is navigation that only exists after script driven rendering; check one of the missing sections manually in the visible browser window before concluding anything.
What are the limits of the bot?
It follows internal links only, to depth 3, up to 999 URLs per run - deliberate bounds that keep a single run polite and finishable. It records nine metadata fields per page and nothing else: no word counts, no heading structure, no image alt text, no inlink counts, and no depth column (a page presence in the file means it was reachable within the bounds; its hop distance is not stored). It processes pages at the load state, so late rendered content and late injected links are invisible to it. It cannot enter logged in areas, and it does not bypass paywalls or consent walls. It is a snapshotter, not a monitor: nothing re runs itself and nothing alerts you - schedule it and diff the files.
What is responsible site crawling?
Crawl only websites you own or have explicit permission to audit; pointing any crawler at a third party site you do not control raises terms of service and, in some jurisdictions, legal questions that are yours to answer before the run, not after. Keep the built in pacing - seven concurrent fetches with a one second delay - and lower it further on small shared hosts that can feel seven parallel requests. Respect x robots tag and robots directives as statements of the publisher intent, and treat the output as what it is: technical metadata about page responses, not a license to republish page content. The acceptable use policy applies in full before your first run.
