Back

Firecrawl Crawl

Research & search81.2K installs

What it does

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.

Install

npx skills add https://github.com/firecrawl/cli --skill firecrawl-crawl

Website

www.skills.sh/firecrawl/cli/firecrawl-crawl

Details

From the skill’s SKILL.md or README, by its authors. View the original

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout to bound polling. Without --wait, crawl returns a job ID for async polling.
  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also

  • firecrawl-scrape — scrape individual pages
  • firecrawl-map — discover URLs before deciding to crawl
  • firecrawl-download — download site to local files (uses map + scrape)
  • firecrawl-build-scrape — building bulk extraction into an app instead of running it here

More Research & search skills

See all

  • Research & search

    Research

    Investigate a question against high-trust primary sources and capture the findings as a Markdown file in the repo. Use when the user wants a topic researched, docs or API facts gathered, or reading legwork delegated to a background agent.

    npx skills add https://github.com/mattpocock/skills --skill research
    783.1K installs

  • Research & search

    Azure AI

    Build and troubleshoot Azure AI Search, Speech, OpenAI, and Document Intelligence apps. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, Speech SDK logs, session ID, result ID, OCR, convert text to speech.

    npx skills add https://github.com/microsoft/azure-skills --skill azure-ai
    651.7K installs

  • Research & search

    Azure Resource Lookup

    List, find, and show Azure resources across subscriptions or resource groups. Handles prompts like \"list the websites in my subscription\", \"list my web apps\", \"show my app services\", \"list virtual machines\", \"list my VMs\", \"show storage accounts\", \"find container…

    npx skills add https://github.com/microsoft/azure-skills --skill azure-resource-lookup
    649.6K installs

  • Research & search

    Caveman Stats

    Show recorded output and cache-read token usage and mode attribution for the current Claude Code session, or locate the host's native usage report. Trigger: /caveman-stats.

    npx skills add https://github.com/juliusbrussee/caveman --skill caveman-stats
    373.3K installs

Skills are shared by members and not reviewed by us. Read a skill’s code before you install it.