Expand conversation.repetition.detected telemetry to cancelled requests

[Copilot]

Fixes microsoft/vscode#287573

The conversation.repetition.detected event was not firing for cancelled requests, missing cases where thought looping leads to user cancellation.

Changes

Added checkRepetitionInDeltas method

• Reconstructs text from response deltas collected before cancellation
• Approximates tokens via whitespace splitting (actual token array unavailable until completion)
• Applies existing anomaly detection: isRepetitive() for token patterns, calculateLineRepetitionStats() for line repetition
• Sends same telemetry events with finishReason: 'canceled'

Updated processCanceledResponse

• Accepts streamRecorder and telemetryProperties parameters
• Invokes repetition check before returning cancelled response

Limitations

Enhanced telemetry for cancelled requests lacks RequestId object fields (completionId, deploymentId, serverExperiments) since only the headerRequestId string is available. Token approximation is less precise than actual tokenization but sufficient for detecting obvious repetition patterns.

// Before: repetition only checked in completed responses
if (!this.isRepetitive(chatCompletion, baseTelemetry?.properties)) {
    completions.push(chatCompletion);
}

// After: also checked in cancelled responses
case FetchResponseKind.Canceled:
    return this.processCanceledResponse(response, ourRequestId, streamRecorder, telemetryProperties);

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Expand coverage of conversation.repetition.detected to cancelled requests</issue_title>
<issue_description>

We currently have the conversation.repetition.detected event that looks for repeating lines. It appears from the codeand my own local testing that we do not fire this event when the request was cancelled. I believe this is a major gap because true thought repetition within a request would probably lead to the user intervening and cancelling before it completes most of the time. We would like to use this metric to track occurrence of thought looping within a single request. We may also be able to use other tools in CAPI to detect this, but if it is a relatively easy change, I think it could be worth looking at this from the client side as well.

FYI @isidorn , @lramos15 </issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Expand coverage of conversation.repetition.detected to cancelled requests vscode#287573

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

Pull request overview

This PR extends the conversation.repetition.detected telemetry event to fire for cancelled requests, addressing a gap where thought looping that leads to user cancellation was not being tracked.

Changes:

• Added checkRepetitionInDeltas method to detect repetition patterns in partial responses from cancelled requests
• Updated processCanceledResponse to accept additional parameters and invoke repetition checking before returning
• Modified the cancelled request handling flow to pass streamRecorder and telemetryProperties for repetition detection

[Copilot]

The filtering condition delta.text != null && delta.text !== '' could be simplified using a more idiomatic approach. Consider extracting this to a named function or using a clearer filter condition like delta.text?.length > 0 which combines null/undefined checking with empty string checking.

Suggested change

	const textContent = deltas.filter(delta => delta.text != null && delta.text !== '').map(delta => delta.text).join('');
	const textContent = deltas.filter(delta => delta.text?.length > 0).map(delta => delta.text).join('');

[Copilot]

While the comment above explains that this is an approximation, it would be helpful to add a brief inline comment noting that this tokenization approach will differ from actual LLM tokenization (which may split on different boundaries). This would help future maintainers understand why results might differ between completed and cancelled requests.

Suggested change

	// tokenization but sufficient for detecting obvious repetition patterns.
	// tokenization but sufficient for detecting obvious repetition patterns.
	// Note: This whitespace-based split does not match the model's real tokenization, so
	// repetition metrics here may differ slightly from those computed for completed responses.