docs: nightly research report 2026-03-18

Report #12. New findings:
- agents/claude_baseline_agent.py:31 — second hardcoded path (LOCOBENCH_CLAUDE_MD_TEMPLATE)
- scripts/export_official_results.py:45 — DEFAULT_REPO_BLOB_BASE points to stale CodeScaleBench org
- scripts/generate_eval_report.py:1005 — falsy bug repeats (line 147 already catalogued)
- 274 tasks × 2 missing metadata fields (verification_modes, use_case_category) = 548 missing values; no runtime check
- data/contextbench/ parquet files confirm DuckDB is natural next step

Recommended next feature: task metadata auto-repair + DuckDB result analytics store.
Stale MEMORY.md dashboard entry corrected (dashboard does not exist).

Author: sjarmak

reports/nightly/2026-03-18-review.md

@@ -0,0 +1,250 @@
+# Nightly Research Report — 2026-03-18 (Report #12)
+
+## Executive Summary
+
+Six consecutive days with zero code fixes. The backlog has now grown to ~100 open issues across 12 reports, yet the remediation pipeline remains stalled. This report surfaces a new critical path bug in `claude_baseline_agent.py` (LOCOBENCH template path hardcoded to a non-existent machine), a stale GitHub org URL in the export pipeline, a repeated falsy-value bug on an additional line not previously noted, and an architectural finding that the project's analysis layer is fundamentally broken: the missing `verification_modes` and `use_case_category` fields silently invalidate all filtered queries. The recommended next feature is a **DuckDB-based result analytics store** to replace the flat-file scan pattern and enable queryable, cross-run analysis.
+
+---
+
+## 1. Code & Architecture Review
+
+### 1.1 NEW CRITICAL: Second Hardcoded Path in claude_baseline_agent.py
+
+`agents/claude_baseline_agent.py:31` contains a second hardcoded personal path not previously catalogued (the `/tmp` race condition at lines 1134-1141 was noted in report #11, but this path is separate):
+
+```python
+LOCOBENCH_CLAUDE_MD_TEMPLATE = Path(
+    "/home/stephanie_jarmak/CodeScaleBench/benchmarks/locobench_agent/templates/CLAUDE.md"
+)
+```
+
+This path is doubly wrong:
+- Points to `/home/stephanie_jarmak/` (not `/Users/sjarmak/`)
+- References `CodeScaleBench` (not `CodeContextBench`)
+
+The template is used when spawning locobench evaluation tasks. Any locobench run on any machine other than the original dev machine will crash with `FileNotFoundError` at task launch. This is distinct from the `apply_verifier_fixes.py:9` path noted in report #11.
+
+**Fix:** Replace with `Path(__file__).resolve().parent.parent / "benchmarks/locobench_agent/templates/CLAUDE.md"`.
+
+---
+
+### 1.2 NEW HIGH: Stale GitHub Org URL in Export Pipeline
+
+`scripts/export_official_results.py:45` embeds a hardcoded GitHub URL:
+
+```python
+DEFAULT_REPO_BLOB_BASE = "https://github.com/sourcegraph/CodeScaleBench/blob/main"
+```
+
+The repository has been renamed to `CodeContextBench`. Every task link generated in exported HTML reports points to a 404 page (or GitHub redirect at best). Given that the export is the primary artifact shared with external reviewers, stale links undermine credibility.
+
+**Fix:** Replace with `"https://github.com/sourcegraph/CodeContextBench/blob/main"` or derive from `git remote get-url origin`.
+
+---
+
+### 1.3 NEW MEDIUM: Falsy Bug Repeated at Line 1005 of generate_eval_report.py
+
+Report #11 noted the falsy bug at `generate_eval_report.py:147`:
+
+```python
+mcp_mode = hc.get("mcp_mode") or r.config_name
+```
+
+The same pattern reappears at **line 1005** in the same file (a second code path used when building the comparison table). If `mcp_mode` is a valid empty string, both sites would incorrectly fall back to `config_name`. This means fixes to line 147 alone are incomplete — both sites must be patched.
+
+---
+
+### 1.4 Analysis Layer Is Structurally Broken: Missing Metadata in All 274 Tasks
+
+Previous reports noted that `verification_modes` and `use_case_category` are missing from all 274 task definitions. The full impact hasn't been articulated until now:
+
+- **`--use-case-category` filter**: silently returns 0 tasks (no error, no warning)
+- **`--verification-mode` filter**: silently returns 0 tasks
+- **Auto-detection of verifier type**: falls back to heuristics that are known to misclassify
+- **Weighted scoring**: weight lookup fails silently (falls back to 1.0 for all tasks)
+
+These fields are not cosmetic — they gate the correctness of every analysis run. Until all 274 tasks have these fields populated, no filtered query can be trusted. There is currently no script to detect this gap at runtime (the `repo_health.py` check validates schema structure but not field completeness).
+
+**Scale**: 274 tasks × 2 missing fields = 548 missing metadata values.
+
+---
+
+### 1.5 Result Storage Does Not Scale
+
+Benchmark results are stored as flat files (`result.json`, `task_metrics.json` per task per run). The `results/` directory currently contains only `repo_cloc_counts.json` — no run outputs. The `export_official_results.py` pipeline must scan the entire run directory tree to build the HTML export, and the HTML output truncates at 1200 rows (noted in report #9).
+
+Meanwhile, `data/contextbench/` already contains parquet files (test/hard/verified subsets), indicating that the project has partially explored columnar storage. However, there is no queryable interface — no DuckDB schema, no SQL interface, no way to ask "what is the average score by suite across the last 5 runs?" without writing a custom script.
+
+The current architecture requires a full file-system scan for every analysis, which will degrade as the task count grows beyond 283.
+
+---
+
+### 1.6 Dashboard Referenced in MEMORY.md Does Not Exist
+
+`MEMORY.md` states: "Dashboard at `dashboard/app.py` (Streamlit), DB at `data/codecontextbench.db`"
+
+Neither exists:
+- `/Users/sjarmak/CodeContextBench/dashboard/` — directory absent
+- `data/codecontextbench.db` — file absent (data/ contains only parquet files and CSVs)
+
+Report #10 flagged this as "Dashboard Reference in Docs Does Not Exist" but the memory entry was never corrected. The stale MEMORY.md entry is misleading to agents starting new sessions.
+
+**Fix:** Update `MEMORY.md` to remove the dashboard/DB references until those components are built.
+
+---
+
+### 1.7 Cost Report Default Guard Pattern Incorrect
+
+`scripts/cost_report.py:262`:
+
+```python
+pct = ((cost / tasks) / (baseline_cost / config_tasks.get("baseline", 1)) - 1) * 100
+```
+
+If the "baseline" config is absent from the run, `get("baseline", 1)` returns integer `1`, producing a nonsensical cost comparison (dividing by 1 as a proxy for baseline cost). The correct guard is `get("baseline") or 1` which at least documents the intent of the fallback. But more importantly, the absence of a baseline config should probably surface as an error rather than silently compute a bogus percentage. This is related to the `defaultdict(int)` issue noted in report #9's `cost_report` entry but is a different code path.
+
+---
+
+## 2. Feature & UX Improvements
+
+### 2.1 No Way to Query Results Across Runs
+
+Currently, answering "how did scores change between run A and run B for suite csb_sdlc_fix?" requires:
+1. Writing a custom script
+2. Scanning thousands of JSON files
+3. Joining across multiple data structures manually
+
+There is no interactive or repeatable query interface. The HTML export at 1200 rows is the only cross-run view, and it's static.
+
+**Sketch:** A `ccb query` CLI command backed by DuckDB could answer:
+```sql
+SELECT suite, AVG(score), COUNT(*) FROM results
+WHERE run_id IN ('run-a', 'run-b')
+GROUP BY suite ORDER BY avg DESC;
+```
+
+### 2.2 No Diff View Between Config Modes
+
+The core value proposition of the benchmark is measuring MCP vs. baseline performance. Currently, comparing two config modes requires running `generate_eval_report.py` and reading through the HTML. There is no concise "delta report" showing which tasks improved, degraded, or changed behavior between baseline and MCP configs.
+
+**Sketch:** `python3 scripts/compare_configs.py --run-a baseline-run-001 --run-b mcp-run-001 --output delta.md` producing a table sorted by score delta.
+
+### 2.3 MEMORY.md Has Stale Entries
+
+The `MEMORY.md` dashboard/DB entries are actively misleading new agent sessions. Agents reading MEMORY.md at session start will attempt to verify dashboard state or DB queries that cannot succeed.
+
+**Fix:** Prune stale entries immediately; add a note that the dashboard is "planned but not yet built."
+
+---
+
+## 3. Research Recommendations
+
+### 3.1 DuckDB for Benchmark Result Storage
+
+The `data/contextbench/` directory already contains parquet files, confirming that columnar storage has been considered. [DuckDB](https://duckdb.org/) is the natural next step:
+
+- Zero-server setup (embedded library, single `.duckdb` file)
+- Reads parquet natively: `SELECT * FROM 'data/contextbench/*.parquet'`
+- Handles 283+ tasks × N runs without performance degradation
+- Enables ad-hoc SQL queries from CLI, Jupyter, and Python scripts
+- Integrates with existing Python toolchain: `pip install duckdb`
+
+A schema like:
+```sql
+CREATE TABLE results (
+    run_id TEXT, task_id TEXT, suite TEXT, config_mode TEXT,
+    score FLOAT, max_score FLOAT, duration_s FLOAT,
+    mcp_type TEXT, timestamp TIMESTAMP
+);
+```
+would immediately enable the comparison queries the project currently cannot answer.
+
+### 3.2 Task Metadata Inference from Static Analysis
+
+Given that 274 tasks lack `verification_modes` and `use_case_category`, a static analysis pass over `test.sh` files could auto-infer these fields:
+
+- If `test.sh` sources `answer_json_verifier_lib.sh` → `verification_mode: answer_json`
+- If `test.sh` sources `dual_score_lib.sh` → `verification_mode: dual_score`
+- If parent directory is `csb_sdlc_fix` → `use_case_category: fix`
+- If parent directory is `csb_org_crossrepo` → `use_case_category: crossrepo`
+
+This could be implemented as a 100-line `scripts/repair_task_metadata.py` with a `--dry-run` flag. Once run, all 274 tasks would have valid metadata, unblocking filtered queries, weighted scoring, and auto-detection.
+
+### 3.3 Pre-commit with Targeted Ruff Rules (Follow-up from Report #11)
+
+Report #11 recommended adding `pyproject.toml` with Ruff. The specific configuration needed (not yet in place):
+
+```toml
+[tool.ruff.lint]
+select = ["S603", "S604", "SIM115", "BLE001", "PTH", "UP"]
+per-file-ignores = {"scripts/sanitize_secrets.py" = ["S105", "S106"]}
+```
+
+This catches 4 of the 5 bug categories found in reports #10–#12 automatically. The report #11 PRD covers this in detail; it remains unimplemented.
+
+---
+
+## 4. Recommended Next Feature
+
+### Task Metadata Auto-Repair + DuckDB Result Store
+
+The project's analysis layer has two compounding gaps:
+1. All 274 tasks lack `verification_modes` and `use_case_category` → every filtered query returns 0 results
+2. No queryable result store → answering "did MCP help?" requires custom scripts every time
+
+These two gaps together mean the benchmark's core output — the MCP vs. baseline comparison — cannot be reliably queried, filtered, or trended. This is the single highest-leverage improvement because it makes all previous and future benchmark runs actually useful for analysis.
+
+**Feature: `scripts/repair_task_metadata.py` + `scripts/init_result_db.py`**
+
+**Part 1 — Task Metadata Repair (2 hours)**
+
+`scripts/repair_task_metadata.py`:
+- Iterates over all 274 task directories in `benchmarks/`
+- Inspects `tests/test.sh` for `source` statements to infer `verification_mode`
+  - `answer_json_verifier_lib.sh` → `answer_json`
+  - `dual_score_lib.sh` → `dual_score`
+  - Neither → `custom`
+- Infers `use_case_category` from parent suite directory name (e.g., `csb_sdlc_fix` → `fix`)
+- Writes inferred values to each task's `task.json` (atomic write: temp + rename)
+- `--dry-run` mode prints diff without writing
+- `--validate` mode checks all tasks have both fields and exits non-zero if any are missing
+- Adds a `repo_health.py` check calling `--validate` to prevent regression
+
+**Part 2 — DuckDB Result Store (3 hours)**
+
+`scripts/init_result_db.py`:
+- Creates `data/results.duckdb` with normalized schema (runs, tasks, metrics tables)
+- `--ingest-run <run-dir>` scans a run directory and inserts all `result.json` files
+- `--query "SELECT ..."` for ad-hoc SQL from CLI
+- Reads existing parquet files in `data/contextbench/` as initial seed data
+
+`scripts/compare_configs.py`:
+- Queries DuckDB to produce a markdown delta report between two run IDs
+- Sorted by score delta descending
+- Flags regressions (score drop > 0.1) with a warning marker
+
+**Acceptance Criteria:**
+1. All 274 tasks have non-empty `verification_modes` and `use_case_category`
+2. `repo_health.py --quick` fails if any task is missing either field
+3. `python3 scripts/repair_task_metadata.py --validate` exits 0
+4. `python3 scripts/init_result_db.py --ingest-run <dir>` succeeds without error
+5. `python3 scripts/compare_configs.py --run-a X --run-b Y` produces valid markdown output
+6. All new code passes Ruff with rules S603, SIM115, BLE001
+
+**Why this over the alternatives:**
+- The code quality gate (report #11) is infrastructure — important but doesn't produce analysis output
+- Verifier consolidation (report #10) is maintenance — reduces toil but doesn't unlock new capability
+- This feature directly answers "did MCP help?" in a reliable, repeatable way, which is the project's core purpose
+
+---
+
+## Issues Added to CLAUDE.md This Session
+
+- `agents/claude_baseline_agent.py:31`: LOCOBENCH_CLAUDE_MD_TEMPLATE hardcoded to `/home/stephanie_jarmak/CodeScaleBench` (second hardcoded path in this file, distinct from /tmp race at lines 1134-1141)
+- `scripts/export_official_results.py:45`: `DEFAULT_REPO_BLOB_BASE` points to stale GitHub org `CodeScaleBench`
+- `scripts/generate_eval_report.py:1005`: Falsy bug repeats (previously only line 147 was catalogued)
+- `data/contextbench/*.parquet`: Parquet files exist; DuckDB integration is the natural next step
+- MEMORY.md: Dashboard/DB entries are stale (dashboard/app.py and data/codecontextbench.db do not exist)
+- 274 tasks × 2 missing metadata fields = 548 values; no runtime detection in repo_health.py
+
+*Remediation velocity: 6 consecutive days without a code fix (Mar 12 → Mar 18). ~100 open issues across 12 reports.*