feat: documentdb-query language — CompletionItemProvider, HoverProvider, acorn validation

[tnaum-ms]

Implements Step 4 — Filter CompletionItemProvider of the shell integration feature (#508). Registers a custom documentdb-query Monaco language with intelligent autocomplete, hover docs, and real-time validation for the Collection View query editors (filter, project, sort).

Plan: docs/plan/04-filter-completion-provider.md

What's included

Custom Language Registration

• documentdb-query language ID — registered once (idempotent), reused across all query editors
• JS Monarch tokenizer — reuses Monaco's built-in JavaScript tokenizer for syntax highlighting without loading the TypeScript worker (~400–600 KB saved)
• Model URI scheme — documentdb://{filter|project|sort}/{sessionId} for per-editor context routing

CompletionItemProvider

• Static operator completions from documentdb-constants — query operators, BSON constructors, system variables, filtered by editor type via meta tag presets (FILTER_COMPLETION_META, PROJECTION_COMPLETION_META)
• Dynamic field completions from SchemaAnalyzer — field names with display type, sparse indicator, and pre-escaped insertText for dotted/special-character field names
• $ prefix range fix — extends the word range backward when the cursor follows $, preventing double-dollar insertion (e.g., $$gt)
• Type-aware operator sorting — when field BSON type is known, operators are sorted by relevance: type-matching first (0_), universal second (1_), non-matching last (2_) — all operators remain visible for discoverability

HoverProvider

• Operator documentation on hover — shows description + link to DocumentDB API docs for any $-prefixed operator or BSON constructor

Validation (acorn)

• Syntax errors (red squiggles) — acorn.parseExpressionAt() detects malformed expressions in real time (300ms debounce)
• Near-miss identifier warnings (yellow squiggles) — acorn-walk traverses the AST; unknown function calls and member-expression objects within Levenshtein distance ≤ 2 of a BSON constructor or known global produce "Did you mean 'X'?" warnings
• Handles both patterns — direct calls (ObjctId("abc")) and member calls (Daate.now(), Maht.min())

Query Parser Replacement

• Replaced toFilterQuery.ts (hand-rolled regex parser supporting only 4 BSON constructors) with @mongodb-js/shell-bson-parser — a battle-tested parser that handles relaxed JavaScript syntax including all BSON constructors, Date.now(), Math.*, regex literals, and nested expressions
• Retained toFilterQuery.ts as a thin wrapper for backward compatibility — delegates to shell-bson-parser internally

Legacy Pipeline Removal

• Removed generateMongoFindJsonSchema() and basicMongoFindFilterSchema.json — the old 10-operator JSON Schema approach, replaced by 308 operators from documentdb-constants
• Removed getAutocompletionSchema tRPC endpoint — replaced by getFieldCompletionData which pushes FieldCompletionData[] to the webview

Wiring

• All three editors (filter, project, sort) correctly wired with per-editor validation cleanup refs
• Field data pushed after query execution via tRPC — session.getKnownFields() → toFieldCompletionItems() → completionStore
• Cleanup on component unmount — clearCompletionContext(sessionId) prevents stale data

Architecture

┌─────────────────────────────────────────────────────────┐
│  QueryEditor.tsx                                        │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐                 │
│  │ filter   │  │ project │  │  sort   │   Monaco        │
│  │ editor   │  │ editor  │  │ editor  │   editors       │
│  └────┬─────┘  └────┬────┘  └───┬─────┘                 │
│       │documentdb://filter/sid  │          │             │
└───────┼─────────────────────────┼──────────┼─────────────┘
        ↓                         ↓          ↓
┌──────────────────────────────────────────────────────────┐
│  registerDocumentDBQueryLanguage(monaco)                  │
│  ├─ CompletionItemProvider  (URI → editorType → meta)    │
│  ├─ HoverProvider           ($operator → docs)           │
│  └─ Validation              (acorn parse + walk)         │
└──────────────────────────────────────────────────────────┘
        ↑                                    ↑
┌───────┴──────┐                  ┌──────────┴──────────┐
│ documentdb-  │                  │  completionStore     │
│ constants    │                  │  (Map<sid, fields>)  │
│ (308 ops)    │                  │  ← tRPC push         │
└──────────────┘                  └─────────────────────┘

Key decisions

Decision	Choice	Rationale
Language ID	documentdb-query (Option E)	JS highlighting without TS worker; clean completion slate; no JS globals noise
Tokenizer	Reuse JS Monarch tokenizer	Full JS syntax highlighting for free; no custom grammar maintenance
Completion routing	Single provider + URI parsing	One registration handles all editor types; URI carries context
Validation engine	acorn + acorn-walk	Lightweight (~28 KB), fast, full ES2024 parse; no additional bundle
Query parser	@mongodb-js/shell-bson-parser	Battle-tested, handles all BSON constructors + relaxed JS syntax
Near-miss scope	BSON constructors + KNOWN_GLOBALS	Catches Daate.now() and ObjctId() patterns
Type-aware sorting	Sort prefix (0_/1_/2_) not exclusion	Fields may be polymorphic; demote non-matching operators, don't hide them

Tests

• 817 tests across 51 suites, all passing
• New test files: documentdbQueryCompletionProvider.test.ts (48 tests), documentdbQueryValidator.test.ts (32 tests)
• Coverage: completion mapping, meta tag routing, field completions, sort prefixes, type-aware ordering, syntax errors, near-miss detection (direct + member calls), Levenshtein distance

Deferred to Step 4.5

• Context-sensitive completions — cursor-position-aware filtering (key vs. value vs. operator position). Infrastructure is ready; requires cursor context detection. See docs/plan/04.5-context-sensitive-completions.md.

[Copilot]

Pull request overview

Introduces a custom documentdb-query Monaco language for the Collection View query editors, shifting autocompletion/hover/validation away from JSON-schema-based Monaco JSON features and enabling relaxed “DocumentDB API expression” syntax (unquoted keys, BSON constructors, etc.) across filter/projection/sort parsing.

Changes:

• Add a documentdb-query Monaco language with JS Monarch tokenization, custom completions, and hover help (backed by @vscode-documentdb/documentdb-constants).
• Add a pure validator using acorn/acorn-walk and wire it into the QueryEditor with debounced Monaco markers.
• Replace JSON-schema autocompletion (schema generation + JSON worker) with field-completion data fetched via tRPC and stored in a completion context.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/webviews/documentdbQuery/registerLanguage.ts	Registers custom Monaco language, tokenizer, completion + hover providers.
src/webviews/documentdbQuery/languageConfig.ts	Defines language ID + URI scheme and helpers for editor context routing.
src/webviews/documentdbQuery/languageConfig.test.ts	Unit tests for URI build/parse utilities and constants.
src/webviews/documentdbQuery/index.ts	Barrel exports for language registration, validator, and completion store.
src/webviews/documentdbQuery/documentdbQueryValidator.ts	Adds acorn-based syntax + near-miss BSON constructor diagnostics.
src/webviews/documentdbQuery/documentdbQueryValidator.test.ts	Unit tests for validator behavior and distance helper.
src/webviews/documentdbQuery/documentdbQueryHoverProvider.ts	Hover content for operators/BSON constructors via constants registry.
src/webviews/documentdbQuery/documentdbQueryHoverProvider.test.ts	Unit tests for hover matching behavior.
src/webviews/documentdbQuery/documentdbQueryCompletionProvider.ts	Maps constants + field data into Monaco completion items.
src/webviews/documentdbQuery/documentdbQueryCompletionProvider.test.ts	Unit tests for completion mapping and store integration.
src/webviews/documentdbQuery/completionStore.ts	In-webview session completion context store for dynamic fields.
src/webviews/documentdbQuery/completionStore.test.ts	Unit tests for completion context store operations.
src/webviews/documentdb/collectionView/components/queryEditor/QueryEditor.tsx	Switches editors to documentdb-query, adds debounced validation + model URIs, and cleanup.
src/webviews/documentdb/collectionView/collectionViewRouter.ts	Replaces schema-returning endpoint with field-completion-data endpoint.
src/webviews/documentdb/collectionView/collectionViewContext.ts	Removes setJsonSchema API from context.
src/webviews/documentdb/collectionView/CollectionView.tsx	Fetches field completion data and stores it for completions (instead of pushing JSON schema).
src/utils/json/data-api/autocomplete/generateMongoFindJsonSchema.ts	Removes JSON schema generator previously used for Monaco JSON autocompletion.
src/utils/json/data-api/autocomplete/basicMongoFindFilterSchema.json	Removes fallback JSON schema used when no field data was available.
src/documentdb/utils/toFilterQuery.ts	Uses @mongodb-js/shell-bson-parser (Loose) for filter parsing and updates error messaging.
src/documentdb/utils/toFilterQuery.test.ts	Updates tests for shell-bson-parser behavior and broader BSON constructor support.
src/documentdb/ClusterSession.ts	Parses projection/sort with shell-bson-parser (Loose) to match updated filter parsing.
src/documentdb/ClustersClient.ts	Parses projection/sort with shell-bson-parser (Loose) and updates user-facing error strings.
package.json	Adds shell-bson-parser + acorn/acorn-walk dependencies.
package-lock.json	Locks new dependencies and updates related transitive versions.
l10n/bundle.l10n.json	Updates localized strings to reflect relaxed syntax and removes unused message.