libobs: Monitoring deduplication fix for 'Desktop Audio' on monitor_only

[pkviet]

Description

When an Audio Output Capture source (AOC) like 'Desktop Audio' has monitoring_type == OBS_MONITORING_TYPE_MONITOR_ONLY, deduplication should not be triggered. this is an edge case which may not cover a reasonable use case, but for the sake of completeness, we deal with it.

Motivation and Context

Fix bug reported by @Penwy

How Has This Been Tested?

Bug is fixed on windows 11 25 H2.
Deduplication is not triggered if AOC is monitor_only.
Tested on a sine signal: the level stays nominal.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[PatTheMav]

Just to wrap my head around this:

• I capture desktop audio output
• I have desktop audio output set up as my monitoring device
• The audio output capture source is set to "only monitor"

Wouldn't this lead to a feedback loop? Because we're continuously feeding the captured desktop audio output back into the output (while nothing will be output on stream)?

[pkviet]

Just to wrap my head around this:

• I capture desktop audio output
• I have desktop audio output set up as my monitoring device
• The audio output capture source is set to "only monitor"

Wouldn't this lead to a feedback loop? Because we're continuously feeding the captured desktop audio output back into the output (while nothing will be output on stream)?

there can't be a feedback loop because Desktop Audio and all AOC sources have the OBS_SOURCE_DO_NOT_SELF_MONITOR flag which is enforced here:

• https://github.com/obsproject/obs-studio/blob/master/libobs/audio-monitoring/win32/wasapi-output.c#L384
• https://github.com/obsproject/obs-studio/blob/master/libobs/audio-monitoring/osx/coreaudio-output.c#L176
• https://github.com/obsproject/obs-studio/blob/master/libobs/audio-monitoring/pulse/pulseaudio-output.c#L361

What happens is that an AOC set to 'monitor only' is neither monitored nor on the output; so why the hell, would users set the AOC to that ? i have no clue.
But if an AOC does not output, there is then no reason to silence the monitored sources, hence the check added in this PR.
Tbh it is a silly edge case ... which led to discussions in dev channel as to whether this really needs fixing.
But the fix is trivial.

[PatTheMav]

Fair enough, though the whole automatic muting system has gotten so arcane that it's really hard for anyone to follow who is not actively working on it.

In this case it depends on another (non-documented) automagic internal functionality to work rather then letting failure scenarios simply fail just because OBS is used as a crutch to fill an obvious missing piece in a streaming setup (in some cases: a USB audio device).

[pkviet]

Fair enough, though the whole automatic muting system has gotten so arcane that it's really hard for anyone to follow who is not actively working on it.

In this case it depends on another (non-documented) automagic internal functionality to work rather then letting failure scenarios simply fail just because OBS is used as a crutch to fill an obvious missing piece in a streaming setup (in some cases: a USB audio device).

I agree; if asio became the streaming standard on windows, we would not have such issues about monitoring and desktop capture of a game. Tbh I wasn't even aware of those issues because i mainly use an asio device for direct monitoring ... Should a software become a crutch to people who don't have the right equipment ? that's a policy decision above my level. But there's a case to be made that obs is so popular that we should take into account people who just don't have either material means or technical level to deal with external sound devices. So I can hear arguments on both sides.