feat(macOS): Capture audio on macOS using Tap API#4209
feat(macOS): Capture audio on macOS using Tap API#4209ReenigneArcher merged 57 commits intoLizardByte:masterfrom
Conversation
ThomVanL
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Bundle ReportChanges will increase total bundle size by 19 bytes (0.0%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: sunshine-esmAssets Changed:
Files in
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4209 +/- ##
=========================================
Coverage ? 18.51%
=========================================
Files ? 107
Lines ? 22448
Branches ? 9970
=========================================
Hits ? 4156
Misses ? 15929
Partials ? 2363
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
Thanks for the PR. I have built this on my M1 Pro MBP and have been trying to get it to work, but haven't had much success so far. I am not able to get any sound to be captured and/or sent. So far I am only testing with the simplest use case of audio playing out of my MBP speakers. I edited the code to make the tap and aggregate non-private, so I could try to view the tap/aggregate with Apple's sample app I can see the tap, but not the aggregate. I'm not sure what's wrong. Can you detail your testing process? I do seem to be getting OK log entries and since I'm running from iTerm, all my permissions seem to be in order (iTerm has access to many things). I had to make the following changes to get it to build: plus a fix in our input.cpp that breaks the latest Xcode 16.4, I'm surprised if you didn't run into this one. I am running Xcode 16.4 clang-1700.0.13.5). |
This comment was marked as outdated.
This comment was marked as outdated.
|
Hey @andygrundman, thanks for taking the time to test the PR. Sorry to hear it’s not working correctly. Here’s my setup on an M4 MBP and what I did. clang --version
Apple clang version 17.0.0 (clang-1700.0.13.5)
Target: arm64-apple-darwin24.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/binI made the changes in VS Code and followed the steps in mkdir build
cmake -B build -G Ninja -S .
ninja -C buildFor testing I did the following steps:
So at one point I'm running three streams simultaneously and the audio plays through the devices. My audio_sink = Steam Streaming Speakers
macos_system_wide_audio_tap = true
origin_web_ui_allowed = pc
stream_audio = true
wan_encryption_mode = 2With I’m familiar with the sample app! It might be the case that the aggregate device settings are probably still marked as private. You’ll need to flip those to Sunshine/src/platform/macos/av_audio.mm Line 604 in 6404705 Sunshine/src/platform/macos/av_audio.mm Line 649 in 6404705 Then the tap will show up in Apple's sample app's UI. 
And the aggregate device shows up as well. 
But even with those values flipped to |
|
Thanks for the detailed info. I forgot my log only had Info level, when using Debug my log does look exactly like yours. I feel like I must just have a permission issue, maybe using iTerm as the permission "owner" isn't correct. Do you see both of these items? What process names are they using? I only get a System Audio item when recording a test file in AudioTapSample. Here's basically what my Screen & System Audio Recording settings look like: If this is the issue, I wonder how Sunshine can detect that it doesn't actually have permission, hmm. |
|
You're right, because I ran into a similar issue with VS Code. Granting it "Screen & System Audio Recording" wasn’t enough; I had to explicitly allow "System Audio Recording Only." From what I found online, the VS Code app bundle itself might be causing the problem. I also tried running 
I did not even notice the little privacy notice at the top until just now, thanks for that. Here's what it's like on my end. 
When I launched Edit: just to be clear, my dev loop consists of launching sunshine builds through the VS Code integrated terminal. |
|
It appears that tests are crashing on macOS 15 and 26, at these tests
Also, still one lint issue: https://github.com/LizardByte/Sunshine/actions/runs/22879151110/job/66380516025?pr=4209#step:10:57 |
It looks like we might still need your TCC fix after all… but I will run a few tests first! |
It's somewhat odd because the macOS-26 completes some of the audio tests before crashing out. |
|
Hey @ReenigneArcher I've had some time to really dive into this a bit more and it's, like you mentioned, worth noting that the TCC prompting is not happening during the Homebrew validation on the master branch. I thought that was a little strange, as well. You can see that I've pushed a small (preliminary) change that I'd love to see run through the workflow. Your feedback is very much welcome. I had a closer look at the CI workflow logs (the Homebrew macOS validation) and I noticed other parts of the code also interact with TCC but appear to skip cleanly: On macOS 14: So, I went and implemented something in But about those audio tests... I think the reason these pass on the master branch is a bit subtle. When no audio sink is set in With the CoreAudio tap implementation, however, not setting a sink in As an alternative, we could inject permissions directly into the TCC database, but that feels a little brittle and error-prone. So the question is: should we add a (I'll sort out rebasing my commits, too.) |
|
I also noticed the audio test wasn't really the greatest test. We could probably improve it in a later PR. For now, I think I'm good with this PR as is. Would you want to submit a follow up PR to help improve the tests? My preference would be to actually verify audio works, but I understand sometimes this isn't the easiest to do in a CI environment where we don't have control of the hardware. For the encoder tests, I just make them skip if they don't work, unless software encoding fails in which case it is a failure. Software encoding should always work, whether in a CI environment or not. |
|
Looks like the audio tests are still failing on macos 15 and 26, so maybe it needs a better fix before this is merged. |
Sure thing, I don't mind taking a look.
That's unfortunate... Though I don't mind digging into it some more! |
|
@ThomVanL so you think it's a permissions issue?
Would adding a virtual audio device allow it to pass? Looks like blackhole is available in homebrew with 3 variants. We could make these dependencies in the homebrew package? |
|
@ReenigneArcher Yes, I think it's got to do with the kTCCServiceAudioCapture permission. I have been also considering adding a sink, so it's definitely something we could try. I have been conducting quite a bit of testing on a separate repository, with just the code path that creates the tap and aggregate device, gtest and a workflow; least with enough code to prompt the TCC. But so far I've been unable to add those permissions to the TCC database. It also looks like the permission is not configured in GitHub's macOS runners, based on what I can tell. I will do some more testing this evening. |
|
@ThomVanL the script you linked is where I originally discovered how to adjust the permissions in the runner as well. You can add anything you want in the same way they do because SIP is disabled in the runners. The hard part, when I originally did it, was figuring out what to give the permission to... but that was with macports. I think with homebrew it should be easier as it's supported on the runners out of the box. |
|
@ReenigneArcher I did some more testing on this across a few different approaches, but I wasn’t able to get the TCC settings to apply correctly. I tried configuring it in multiple ways, but no success so far. As a fallback, I replicated the current behavior by forcing the sink to a non-existent value: (I have not yet committed this, by the way) struct AudioTest: PlatformTestSuite, testing::WithParamInterface<std::tuple<std::basic_string_view<char>, config_t>> {
void SetUp() override {
m_config = std::get<1>(GetParam());
m_mail = std::make_shared<safe::mail_raw_t>();
#ifdef __APPLE__
m_saved_sink = config::audio.sink;
if (config::audio.sink.empty()) {
config::audio.sink = "__nonexistent_audio_sink__";
m_overrode_sink = true;
}
#endif
}
void TearDown() override {
#ifdef __APPLE__
if (m_overrode_sink) {
config::audio.sink = m_saved_sink;
}
#endif
}
config_t m_config;
safe::mail_t m_mail;
#ifdef __APPLE__
std::string m_saved_sink;
bool m_overrode_sink {};
#endif
};That said, I’m not particularly a big fan of this approach. I find it to be more along the lines of a quick fix rather than a proper solution.. Though as far as I can tell it mirrors the current behavior in master. I’ll keep digging into the TCC side tomorrow evening to see if there’s a way to get the permissions set correctly in the runner. |
ThomVanL
force-pushed
the
users/thomasvanlaere/feat-macos-ca-taps
branch
from
dd24999 to
50b2105
Compare
|
@ReenigneArcher Even better, I've been able to add the correct TCC permissions, and in my testing I've been able to get OK results from all four I ended up just expanding upon that script from the "actions/runner-images" I linked earlier. It didn't occur to me to add |
|
|
@ThomVanL nice work! Looks like everything is green! |
Description
This PR adds system-wide audio tap support for macOS. The implementation introduces:
Additional changes
cmakefiles for compatibility with Homebrew-based setup.opensslandopusare found automatically. This replaces the need to run manuallncommands, but I’m not sure if this is the best long-term approach. Feedback welcome.cmake/compile_definitions/unix.cmaketo ensureSUNSHINE_ASSETS_DIRresolves correctly.src_assets/macos/assets/Info.plistto prepare for required macOS permission prompts.macos_system_wide_audio_tapconfig option to theaudio_tstruct.Testing
Notes
setupMicrophonefunctionality as it is also a viable option!macos_system_wide_audio_tapsetting.If the AI involvement is a blocker for merging, no worries. I can totally understand! This was also a learning project for me and I had a lot of fun building it.
Screenshot
Web UI – Audio/Video Configuration
New option for enabling system-wide audio recording on macOS. Disables the audio sink option when checked.
macOS Permission Prompt
System permission request when Sunshine first tries to access system audio:
System Settings – Screen & System Audio Recording
macOS privacy settings showing Sunshine/Terminal access to "System Audio Recording Only":
Issues Fixed or Closed
Roadmap Issues
Type of Change
Checklist
AI Usage