Security research use only
crawlx maps the attack surface of dynamic web applications you are authorized to test. LLM-guided planning and an accessibility-tree browser plane find the pages, forms and endpoints; an independent safety kernel decides what is allowed to run.
git clone https://github.com/noxred-research/crawlx.git
cd crawlx
cargo build --release -p crawlx-cliCapabilities
$ crawlx crawl http://127.0.0.1:34635/s1/ \
--mode static --seed 20261019 \
--allow-private-networks \
--authorization-record authz.local.json
discovered 90
admitted 30
deduplicated 55
policy_blocked 3
robots_blocked 2
succeeded 28
failed 2Pre-1.0 source release. Production qualification is in progress — do not expose a checkout as a public service.
01AI/LLM-native
Language models and learned rankers help crawlx explore single-page applications. They rank candidate actions; they never authorize one. Model confidence is not safety confidence, and nothing in the model layer can approve an action.
An optional planner proposes a short plan in a closed vocabulary: at most 20 steps, candidate indexes and value references only — no selectors, no URLs. Page text reaches a prompt only as tagged untrusted data, and the safety kernel still adjudicates every step.
Calibrated probabilities over a crawler-built candidate list, from an external typed-decision service or an optional ONNX ranker. A confidence gate abstains when unsure; a circuit breaker stops a failing engine.
Dynamic apps are read through the accessibility tree, with versioned element identity, reversible state clustering and a learned enablement graph.
Pages become Markdown with metadata and links. Opt-in word-window chunk ranges are available on the scrape API for downstream LLM pipelines.
crawlx mcp serves the Firecrawl MCP tools over stdio, as a thin client
of a crawlx API.
The Firecrawl-compatible API at /compat/firecrawl/v2 serves
extract and agent, for tool interoperability. They need an
OpenAI-compatible backend you configure; without one the endpoint answers 503.
02Security research
Most crawlers stop at HTTP, and most browser automation clicks anything. crawlx observes what an application exposes, in read-only mode by default, and records what it did.
A URL plane for documents and a pinned-Chromium browser plane for script-heavy apps, in one run. Observe-only mode records states and accessibility-tree snapshots without interacting.
Form extraction, JavaScript endpoint mining (off|regex|ast|both),
sitemaps, feeds and known files, with seeded, reproducible form input.
--detect-secrets flags exposed credentials in crawled content. Secrets
are redacted from logs, traffic metadata and exports.
03Controls
Defaults refuse. Capabilities widen only through an explicit, recorded grant, and a kernel independent of the crawl logic enforces it.
Scope by host, registrable domain or domain, robots.txt per RFC 9309,
politeness limits per origin, and a deny-first egress policy that refuses loopback,
private, link-local and cloud-metadata addresses on every redirect hop and browser
request.
Every fetch and browser action is classified. The default
read_only profile cannot change server state;
test_environment and authorized_security_test widen it
only with an authorization record.
Private-network access, elevated profiles and the robots override need an operator-created record stating what you are authorized to do, with a create, verify, revoke and offline-sign lifecycle.
Every action is journaled before it is performed and every safety verdict is recorded. An independent auditor checks the record from outside the worker and pauses violating runs.
ZDR mode never creates a durable response cache. Payloads sit in a volatile buffer with a fixed lifetime.
Retained content and sensitive artifacts are sealed under per-artifact data keys, wrapped by AWS KMS or an operator key file, and published only after a recheck against the run-wide secret registry.
04Security research use only
crawlx exists for authorized security research and testing. You are responsible for how you use it.
05Architecture
A 21-crate Rust workspace with enforced dependency layering. The model layer cannot depend on the safety kernel, so it cannot authorize anything.
Interfaces
crawlx CLIGuidance (advisory)
Safety kernel admits every fetch and every browser action
URL plane
Browser plane
Storage and audit
Read the architecture document ↗ for details.
06Quick start
You need the pinned Rust toolchain. Dynamic and hybrid crawls also need the pinned Chrome for Testing build; the helper script downloads it and verifies its SHA-256. Use a target you are authorized to test.
Step 1 · Build
git clone https://github.com/noxred-research/crawlx.git
cd crawlx
cargo build --release -p crawlx-cliStep 2 · Observe a web application
export CRAWLX_CHROMIUM_PATH=$(ci/fetch-chromium.sh)
crawlx crawl https://app.example.org \
--mode dynamic --observe-only --output ./outStep 3 · Add secret and form discovery
crawlx crawl https://app.example.org --mode hybrid \
--detect-secrets --form-extraction \
--javascript-mode both --output ./out07Tool interoperability
The /compat/firecrawl/v2 facade accepts fc- keys, so
Firecrawl SDKs, its CLI, LangChain’s loader and MCP clients can drive a crawlx you
operate. The integration notes list base URLs per client and what does not work.
crawlx serve --firecrawl-keys \
--api-key acme=fc-<32 hex>
crawlx mcp --api-url http://127.0.0.1:808008Open source
crawlx is licensed under the GNU Affero General Public License, version 3 only — not “version 3 or later.” Third-party dependencies, vendored sources and embedded resources keep their own licenses and notices.
Use GitHub private vulnerability reporting. Please do not open public issues for suspected vulnerabilities, and do not include live credentials or real personal data in a report.