feat: Add crawl and map commands

[abimaelmartell]

Summary

Adds two new CLI commands (crawl and map) to the Firecrawl CLI, enabling users to crawl websites and discover URLs directly from the terminal. Both commands support asynchronous job handling with status checking and wait modes.

Changes

New Commands

firecrawl crawl

• Start crawl: firecrawl crawl <url> - Returns job ID immediately for async processing
• Check status: firecrawl crawl <job-id> --status - Check status of existing crawl job (auto-detects UUID format)
• Wait mode: firecrawl crawl <url> --wait - Waits for crawl completion with optional progress display
• Options: Support for limit, max-depth, exclude/include paths, sitemap handling, concurrency control, and more

firecrawl map

• Map URLs: firecrawl map <url> - Discovers and lists URLs on a website
• Options: Limit, search query, sitemap handling, subdomain inclusion, query parameter handling
• Output formats: Human-readable (one URL per line) or JSON format

Features

• Job ID auto-detection: Automatically detects UUID format to distinguish job IDs from URLs
• Progress display: Real-time progress updates when using --wait --progress for crawl operations
• Flexible output: Support for stdout, file output, and pretty-printed JSON
• Comprehensive error handling: Clear error messages for API failures and edge cases

Implementation Details

• Added src/commands/crawl.ts - Crawl command implementation
• Added src/commands/map.ts - Map command implementation
• Added src/types/crawl.ts - Type definitions for crawl command
• Added src/types/map.ts - Type definitions for map command
• Added src/utils/job.ts - Utility functions for job ID detection and URL validation
• Updated src/index.ts - Registered new commands with full option support

Testing

• Added comprehensive test suite following existing patterns:
  • src/__tests__/commands/crawl.test.ts - 19 tests covering all crawl functionality
  • src/__tests__/commands/map.test.ts - 14 tests covering all map functionality
  • src/__tests__/utils/job.test.ts - 8 tests for utility functions
• All 67 tests passing ✅

Usage Examples

Start a crawl and get job ID

firecrawl crawl https://example.com --limit 100

Check crawl status

firecrawl crawl 550e8400-e29b-41d4-a716-446655440000 --status

Start crawl and wait for completion

firecrawl crawl https://example.com --wait --progress

Map URLs on a site

firecrawl map https://example.com --limit 50

Map with JSON output

firecrawl map https://example.com --json --pretty