feat: add support for line-by-line evaluation

[AAgnihotry]

Summary

Adds line-by-line evaluation capability to both new evaluators (version-based) and legacy evaluators (category/type-based), enabling per-line evaluation of multi-line outputs with partial credit scoring.

Problem

Current evaluators provide binary pass/fail results for multi-line outputs. If any part of the output is incorrect, the entire evaluation fails with a score of 0.0, even if most lines are correct.

Solution

This PR introduces line-by-line evaluation that:

• Splits outputs by a configurable delimiter (default: \n)
• Evaluates each line independently
• Returns both per-line scores and an aggregated score (average)
• Provides partial credit based on the percentage of correct lines
• Works with both new evaluators and legacy evaluators

Key Changes

New Evaluators (Version-based)

• output_evaluator.py: Added lineByLineEvaluator and lineDelimiter config options to OutputEvaluatorConfig
• output_evaluator.py: Implemented _evaluate_line_by_line() method in BaseOutputEvaluator
• test_evaluator_methods.py: Added 4 comprehensive tests for line-by-line evaluation

Legacy Evaluators (Category/Type-based)

• base_legacy_evaluator.py: Added lineByLineEvaluation and lineDelimiter fields to BaseLegacyEvaluator
• base_legacy_evaluator.py: Implemented _evaluate_line_by_line() method for legacy evaluators
• base_legacy_evaluator.py: Added helper methods: _split_into_lines() and _get_actual_output()
• test_legacy_exact_match_evaluator.py: Added 5 comprehensive tests for legacy line-by-line evaluation

Runtime & Bug Fixes

• runtime.py: Fixed aggregation logic to skip line-by-line sub-results (e.g., "ExactMatch (Line 1)")
• output_evaluator.py: Fixed bug where targetOutputKey wasn't properly applied to individual lines

Sample Project

• samples/line_by_line_test/: Added complete sample agent demonstrating the feature with:
  • Simple agent that outputs one item per line
  • 3 evaluators: new line-by-line, regular, and legacy line-by-line
  • 3 test cases showing partial credit scoring
  • Full documentation in README.md

Benefits

• ✅ Partial credit: 2/3 correct lines = 0.67 score instead of 0.0
• ✅ Granular feedback: See which specific lines passed/failed
• ✅ Flexible: Works with any output evaluator (ExactMatch, Contains, JsonSimilarity, LLMJudge)
• ✅ Universal: Works with both new and legacy evaluator formats
• ✅ Backwards compatible: Disabled by default, opt-in via config

Example Usage

New Evaluators (Low-Code JSON config):

{
  "version": "1.0",
  "evaluatorTypeId": "uipath-exact-match",
  "evaluatorConfig": {
    "name": "LineByLineExactMatch",
    "lineByLineEvaluator": true,
    "lineDelimiter": "\n"
  }
}

Legacy Evaluators (Low-Code JSON config):

{
  "category": 0,
  "type": 1,
  "name": "LegacyLineByLineExactMatch",
  "targetOutputKey": "result",
  "lineByLineEvaluation": true,
  "lineDelimiter": "\n"
}

Coded Evaluator:

@evaluator
def line_by_line_evaluator() -> ExactMatchEvaluator:
    return ExactMatchEvaluator(
        evaluatorConfig={
            "lineByLineEvaluator": True,
            "lineDelimiter": "\n"
        }
    )

Test Results

Sample agent output showing all three evaluators:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃  Evaluation               ┃  LineByLineExactMatch ┃  RegularExactMatch  ┃  LegacyLineByLineExactMatch ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│  Test all lines match     │                  1.0  │                1.0  │                         1.0 │
│  Test when one line       │                  0.7  │                0.0  │                         0.7 │  ← Partial credit!
│  doesn't match            │                       │                     │                             │
│  Test with single item    │                  1.0  │                1.0  │                         1.0 │
├───────────────────────────┼───────────────────────┼─────────────────────┼─────────────────────────────┤
│  Average                  │                  0.9  │                0.7  │                         0.9 │
└───────────────────────────┴───────────────────────┴─────────────────────┴─────────────────────────────┘

Testing

• ✅ All existing tests pass (1632 tests)
• ✅ Added 4 new line-by-line tests for new evaluators
• ✅ Added 5 new line-by-line tests for legacy evaluators
• ✅ Sample agent demonstrates full functionality with all 3 evaluator types
• ✅ Type checking passes (mypy)
• ✅ Linting passes (ruff)
• ✅ Build succeeds

Bugs Fixed

1. Runtime aggregation bug: Line sub-results were causing validation errors during score aggregation
2. targetOutputKey bug: Individual lines weren't wrapped in proper structure when targetOutputKey != "*"
3. Type checking bug: Added proper type assertions for LineByLineEvaluationDetails

🤖 Generated with Claude Code

Development Packages

uipath

[project]
dependencies = [
  # Exact version:
  "uipath==2.10.30.dev1014815583",

  # Any version from PR
  "uipath>=2.10.30.dev1014810000,<2.10.30.dev1014820000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[AAgnihotry]

[Chibionos]

need to rearrange the functions they are pilled up on the core evaluator files

[Chibionos]

this should not break medline when they turn on URT for them.
We should also make sure we track this on the sprint board.