Crawl the sites that block crawlers.

• 13 result tabs — Internal, External, Response Codes, Page Titles, Meta Description, Headings (H1–H6), Directives, Images, Broken Assets, Links, Redirects, Sitemaps (orphan pages) and Dates.
• Live progress over SSE — the dashboard updates as pages land, without polling.
• Reads the site properly — robots.txt, sitemaps including nested <sitemapindex>, redirect chains, and status checks on external links and assets.
• Dates from JSON-LD — datePublished and dateModified from schema.org markup, matched against sitemap <lastmod> per URL.
• Multi-user — member and admin roles, per-user crawls and API keys, and guards that stop you disabling the last active admin.
• Reproducible crawls — every setting is stored on the crawl itself, never read from the environment, so Redo crawl reruns it with exactly what it ran with the first time.
• Fails loudly, not quietly — if the remote browser drops mid-crawl the run fails with an explanation instead of finishing with a silent hole in the results.
• Export anything — CSV or Excel per tab with your filters applied, plus a multi-sheet master workbook. Webhooks fire crawl.completed on finish.

# Pull the published image instead of building
docker pull kamenarov/krawlify-app:0.1.0

# Postgres, Redis, migrations, web and worker
IMAGE=kamenarov/krawlify-app:0.1.0 docker compose up -d
# → http://localhost:3000 — register, then start a crawl

• Source
• kamenarov/krawlify-app

• Its own tabs only. They are grouped under a purple 🕷 Krawlify label, and the debugger attaches to nothing else.
• Cleans up after itself. The crawler's tabs close five seconds after the last client disconnects.
• Three values stored, total — the relay URL, a 256-bit token this browser generated, and whether you last pressed Connect or Disconnect. Nothing in chrome.storage.sync.
• No telemetry, no analytics, no content scripts. The only connection it makes is to the relay address in its settings.
• Cookies are opt-in — an optional permission you grant on the options page and can revoke at any time.

Permissions: debugger, tabs, tabGroups, storage, alarms. Optional: cookies, <all_urls>.

• Chrome Web Store
• Source

The extension and the relay ship from one repository — they are two halves of the same thing.

# ExtensionInstallForcelist — force-install from the Web Store
oonpeiliblmkimphjhhfaghpjbjmogon;https://clients2.google.com/service/update2/crx

• Multi-tenant by token. One relay serves many browsers; the Bearer token in the header is what picks yours. The relay has no token of its own and keeps no list.
• Holds nothing. Tab ownership lives in memory and is discarded when the connection ends. No crawled page content is written to disk.
• Loopback by default — 127.0.0.1 unless you opt out, with --tls-cert, --trust-proxy, --allow-host and --allow-origin for anything wider.
• Ordinary DevTools endpoints — /json/version, /json/list, /json/new, /status, /healthz, plus the browser and page WebSockets.
• Deliberately boring failures. No token and an unknown token look the same from outside.

# published image, amd64 + arm64
docker run -p 127.0.0.1:9333:9333 \
  kamenarov/krawlify-relay:0.1.0

# or from a clone
npm start                # → 127.0.0.1:9333

# liveness, unauthenticated
curl http://127.0.0.1:9333/healthz

# control channel: ws /control
status · tabs.list · tabs.create · tabs.close
tabs.activate · tabs.navigate · cookies.get*

# * requires --allow-cookies

• Source
• kamenarov/krawlify-relay

• Only the URLs you give it. No discovery, no link following, no sitemap parsing — which is exactly what you want when auditing a named set of pages.
• Two ways to get a browser. Relay mode crawls inside the Chrome you already have open, in the 🕷 Krawlify tab group. Launch mode (--launch) starts your installed Chrome on your own profile instead — no extension, no relay, nothing to set up.
• Faces the bot check in a visible tab for up to 60 seconds, then crawls with the session that cleared it. --wait-full holds it open for captchas and logins you solve by hand.
• One workbook out — a Summary sheet plus internal, response-codes, titles, meta, headings, directives, images, broken-assets, redirects, links, external and dates.
• Opinionated thresholds — titles 30–60 characters (561 px), meta descriptions 70–160 characters (985 px), alt text at most 100 characters.
• Fails early and specifically. Three preflight checks — dependencies, browser credentials, URL list — each with its own exit code, and --check runs all three without crawling anything.

# install once, from npm
npm install -g krawlify-agent

# register it as a Claude Code skill
krawlify-agent install-skill            # ~/.claude/skills
krawlify-agent install-skill --project  # ./.claude/skills

# no relay, no extension — just your Chrome
npx krawlify-agent --urls urls.txt --launch

# check the setup without crawling
krawlify-agent --check

# .env.local — relay mode only
KRAWLIFY_TOKEN=<the token from the extension>
KRAWLIFY_RELAY=http://127.0.0.1:9333

Output lands in your working directory as krawlify-report-<host>-<YYYYMMDD-HHmm>.xlsx.

• Source
• krawlify-agent