From d35085c4132dfd70747ab155c8442857017a23c4 Mon Sep 17 00:00:00 2001 From: khang <49549324+ntdkhang@users.noreply.github.com> Date: Sat, 17 Oct 2026 17:27:00 +0700 Subject: [PATCH 1/2] Add simulator movie recording and on-demand host capture --- CHANGELOG.md | 22 ++ README.md | 62 +++- SimulatorInjection/SCFrameStore.h | 2 +- SimulatorInjection/SCMovieRecorder.h | 18 + SimulatorInjection/SCMovieRecorder.m | 275 ++++++++++++++ SimulatorInjection/SCVideoGeometry.h | 31 ++ SimulatorInjection/SimCamInject.m | 341 +++++++++++++++--- SimulatorInjection/Tests/AnalyzeMovie.m | 85 +++++ SimulatorInjection/Tests/CheckOrientation.m | 40 ++ SimulatorInjection/Tests/ExtractMovieFrames.m | 21 ++ SimulatorInjection/Tests/HostAudioFeed.m | 115 ++++++ .../Tests/NativeDriverProbe.swift | 116 ++++++ SimulatorInjection/Tests/README.md | 144 ++++++++ SimulatorInjection/Tests/RecordingProbe.m | 191 ++++++++++ .../Tests/build-native-driver-probe.sh | 18 + SimulatorInjection/Tests/build-probe.sh | 27 ++ .../Tests/check-fixture-asset.py | 22 ++ .../Tests/publish-orientation-fixture.py | 68 ++++ .../Tests/run-native-driver-probe.sh | 58 +++ .../Tests/run-orientation-probe.sh | 39 ++ SimulatorInjection/Tests/run-probe.sh | 43 +++ SimulatorInjection/Tests/run-webcam-probe.py | 122 +++++++ SimulatorInjection/build.sh | 5 +- simcamctl/FeedCommand.swift | 16 +- 24 files changed, 1832 insertions(+), 49 deletions(-) create mode 100644 SimulatorInjection/SCMovieRecorder.h create mode 100644 SimulatorInjection/SCMovieRecorder.m create mode 100644 SimulatorInjection/SCVideoGeometry.h create mode 100644 SimulatorInjection/Tests/AnalyzeMovie.m create mode 100644 SimulatorInjection/Tests/CheckOrientation.m create mode 100644 SimulatorInjection/Tests/ExtractMovieFrames.m create mode 100644 SimulatorInjection/Tests/HostAudioFeed.m create mode 100644 SimulatorInjection/Tests/NativeDriverProbe.swift create mode 100644 SimulatorInjection/Tests/README.md create mode 100644 SimulatorInjection/Tests/RecordingProbe.m create mode 100644 SimulatorInjection/Tests/build-native-driver-probe.sh create mode 100644 SimulatorInjection/Tests/build-probe.sh create mode 100644 SimulatorInjection/Tests/check-fixture-asset.py create mode 100644 SimulatorInjection/Tests/publish-orientation-fixture.py create mode 100644 SimulatorInjection/Tests/run-native-driver-probe.sh create mode 100644 SimulatorInjection/Tests/run-orientation-probe.sh create mode 100644 SimulatorInjection/Tests/run-probe.sh create mode 100644 SimulatorInjection/Tests/run-webcam-probe.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 576e7b7..e0756a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,17 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [Unreleased] +### Added +- Optional socket-owned host capture: running injected sessions acquire hardware + on demand and release it on stop or process exit. Supervised host feeds also + stop on stdin EOF when their controller exits. +- Simulator injection supports `AVCaptureMovieFileOutput` recording to H.264/AAC + QuickTime files, including microphone audio through `AVAudioEngine`, recording + delegates/state, separate media connections, and rotation/mirroring transforms. +- Isolated public-API recording probes decode finalized files and exercise repeat + takes, cancellation, teardown, permission denial, and missing samples. An optional + integration harness compiles Small Hello's original Media sources unchanged. + ### Changed - **Frame transport rewritten on CoreMediaIO sink streams.** The v1.0.0 design pushed frames into the camera extension over a private `NSXPCListener`. @@ -32,6 +43,17 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm start a CMIO stream from the sandbox). ### Fixed +- Preview layers attached before capture inputs are configured no longer commit + the Simulator's native stub graph, which could cancel SwiftUI recording startup. +- Releasing a preview's injected connection no longer enters native graph cleanup + for an uninitialized capture connection. +- Webcam recording and preview share source-coordinate normalization: the upright + Mac image is the emulated camera's 90° portrait reference, preventing sideways + movies from normal iPhone portrait recording calls. `simcamctl feed --camera` + now supplies the camera source tag. Visual regressions check four rotations, + each mirrored/unmirrored, through preview and finalized movie pixels. +- Injection initializer hooks preserve Objective-C init-family ownership. Sessions + own their outputs and keep the injected graph isolated through recording teardown. - **Test-pattern timer leak.** Each CMIO client that started streaming created a new timer without cancelling the previous one; orphaned timers kept emitting at full rate forever. Only the first client creates the timer now. diff --git a/README.md b/README.md index 568e3ba..abe3f73 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,67 @@ into your app **in the Simulator only**. Your app still imports nothing. Frames travel through `/private/tmp/SimulatorCamera/frame.bin` (`SimulatorInjection/SCFrameStore.h`). Video frames are BGRA; apps that force a 420 YpCbCr format get BGRA anyway. Not supported yet: barcode types other than -QR, depth, multi-cam, movie file output. +QR, depth, multi-cam. + +### Movie recording with sound + +The injection library supports `AVCaptureMovieFileOutput` start/stop recording, +`isRecording`, recording delegates, and separate video/audio connections. It +writes finalized H.264 + mono AAC QuickTime movies from injected video frames +and the Simulator's microphone route. Apps keep using AVFoundation; there is no +app-side writer or library import. Video connection rotation and mirroring are +written into the movie's display transform. + +Mac webcam images are already upright, unlike an iPhone's sensor-coordinate +buffers. Camera-tagged frames therefore use a **90° portrait reference**: a +portrait connection requesting 90° displays the host image upright, while 0°, +180° and 270° still rotate relative to that reference. Mirroring is applied in +display coordinates. Webcam preview and movies use the same mapping, preserving +the source aspect ratio and actual frame dimensions without cropping/resampling +the recorded pixels. Other source types use the existing zero-offset sensor +coordinate convention. Rebuild `simcamctl` as well as the injection library: +`feed --camera` now writes the camera source tag needed for normalization. + +Add a native microphone `AVCaptureDeviceInput`, obtain microphone permission, +and activate a recording-capable `AVAudioSession`. The library preserves the +caller's audio-session settings. Audio comes from an internal `AVAudioEngine`, +because the injected capture session does not run a native capture graph. +Adding `AVCaptureAudioDataOutput` to that session does **not** deliver audio. +Audio from a selected *video file source* is not forwarded. + +The Simulator's host input must carry sound. A silent virtual/loopback input +produces a genuinely silent recording; this library does not substitute a tone +or pretend microphone permissions were granted. No system extension is needed +for the built-in moving test pattern plus microphone recording. + +Wait for `didFinishRecording` before reading the completed file or starting the +next take. Startup cancellation and missing samples fail with an error; +stopping a session or removing its output finalizes an already-started take. +The supported subset uses new local `.mov` destinations and fixed frame +dimensions. Custom codec/output settings, recording limits, pause/resume, and +full device-quality timing/latency guarantees are outside this implementation. +An injected session remains injected after input removal; use a new session to +return to native capture. This avoids committing fake configuration into the +Simulator's stub graph, which otherwise emits spurious runtime errors. + +See [recording verification](SimulatorInjection/Tests/README.md) for isolated +public-API probes, decoded audio/video checks, and an unchanged-caller harness. + +### On-demand host capture + +An optional host controller can provide a Unix socket via `SIMCAM_BRIDGE_SOCKET` +in the injected app's environment. On the first injected session's `startRunning`, +the library connects and sends `CAPTURE\n`. The controller starts its hardware +feeds and replies with one byte, `R`, only when live frames/audio are ready. +The connection remains open until the last session stops or is released; process +exit also closes it automatically. The controller must stop its feeds when the +last client disconnects. Failed startup leaves the session stopped. Controller +loss interrupts capture; managed capture never substitutes a test pattern. + +`simcamctl feed` and `Tests/HostAudioFeed` accept `SIMCAM_STOP_ON_STDIN_EOF=1` +for supervised use: keep their stdin pipes open while owned, and close them to +stop. This also releases hardware when a controller crashes. A zero duration +keeps `HostAudioFeed` running until its owner stops it. ## Picking a source diff --git a/SimulatorInjection/SCFrameStore.h b/SimulatorInjection/SCFrameStore.h index 8934a03..bdd5551 100644 --- a/SimulatorInjection/SCFrameStore.h +++ b/SimulatorInjection/SCFrameStore.h @@ -25,7 +25,7 @@ typedef struct { uint32_t width; uint32_t height; uint32_t bytesPerRow; - uint32_t sourceKind; /* SimCamSourceKind raw value, informational */ + uint32_t sourceKind; /* SimCamSourceKind; 1 = upright Mac webcam image */ uint64_t seq; /* odd = write in progress */ double hostTime; /* seconds, writer's clock, for staleness checks */ uint8_t reserved[24]; diff --git a/SimulatorInjection/SCMovieRecorder.h b/SimulatorInjection/SCMovieRecorder.h new file mode 100644 index 0000000..627e278 --- /dev/null +++ b/SimulatorInjection/SCMovieRecorder.h @@ -0,0 +1,18 @@ +#import + +// All methods except the atomic recording getter run on the injection frame queue. +@interface SCMovieRecorder : NSObject +@property (atomic, readonly, getter=isRecording) BOOL recording; +@property (nonatomic, readonly) BOOL finished; +- (instancetype)initWithOutput:(AVCaptureMovieFileOutput *)output + URL:(NSURL *)url + delegate:(id)delegate + connections:(NSArray *)connections + audio:(BOOL)audio + rotation:(CGFloat)rotation + mirrored:(BOOL)mirrored + queue:(dispatch_queue_t)queue; +- (void)start; +- (void)appendVideo:(CVPixelBufferRef)pixels time:(CMTime)time; +- (void)stop; +@end diff --git a/SimulatorInjection/SCMovieRecorder.m b/SimulatorInjection/SCMovieRecorder.m new file mode 100644 index 0000000..1060f94 --- /dev/null +++ b/SimulatorInjection/SCMovieRecorder.m @@ -0,0 +1,275 @@ +#import "SCMovieRecorder.h" +#import "SCVideoGeometry.h" +#import + +@interface SCMovieRecorder () +@property (atomic, readwrite, getter=isRecording) BOOL recording; +@property (nonatomic, readwrite) BOOL finished; +@end + +@implementation SCMovieRecorder { + AVCaptureMovieFileOutput *_output; // Retained only until the terminal callback is enqueued. + NSURL *_url; + id _delegate; + NSArray *_connections; + BOOL _needsAudio, _started, _stopping, _hasVideo, _hasAudio; + CGFloat _rotation; + CGFloat _sourcePortraitRotation; + BOOL _mirrored; + dispatch_queue_t _queue; + AVAudioEngine *_engine; + AVAssetWriter *_writer; + AVAssetWriterInput *_video, *_audio; + AVAssetWriterInputPixelBufferAdaptor *_adaptor; + NSMutableArray *_pendingAudio; + CMTime _origin, _lastVideo, _lastAudio, _lastAudioEnd; + double _sampleRate; + size_t _width, _height; + id _interruptionObserver; + id _engineObserver; + BOOL _tapInstalled; +#if SIMCAM_RECORDING_TESTS + dispatch_source_t _testAudioTimer; +#endif +} + +static NSError *SCMovieError(NSString *message) { + return [NSError errorWithDomain:AVFoundationErrorDomain code:AVErrorUnknown + userInfo:@{NSLocalizedDescriptionKey: message}]; +} + +static void SCRecordingLog(NSString *message) { + if (getenv("SIMCAM_RECORDING_LOG")) NSLog(@"[SimulatorCamera] movie %@", message); +} + +- (instancetype)initWithOutput:(AVCaptureMovieFileOutput *)output URL:(NSURL *)url + delegate:(id)delegate + connections:(NSArray *)connections audio:(BOOL)audio + rotation:(CGFloat)rotation mirrored:(BOOL)mirrored queue:(dispatch_queue_t)queue { + if ((self = [super init])) { + _output = output; _url = url; _delegate = delegate; _connections = connections; + _needsAudio = audio; _rotation = rotation; _mirrored = mirrored; _queue = queue; + _pendingAudio = [NSMutableArray array]; + _lastVideo = _lastAudio = _lastAudioEnd = kCMTimeInvalid; + } + return self; +} + +- (void)start { + SCRecordingLog(@"starting"); + self.recording = YES; + _origin = CMClockGetTime(CMClockGetHostTimeClock()); + if (!_url.isFileURL || [[NSFileManager defaultManager] fileExistsAtPath:_url.path]) { + [self fail:SCMovieError(@"Recording requires a new local file URL")]; return; + } + __weak SCMovieRecorder *weakSelf = self; + dispatch_queue_t queue = _queue; + _interruptionObserver = [NSNotificationCenter.defaultCenter addObserverForName:AVAudioSessionInterruptionNotification object:nil queue:nil usingBlock:^(NSNotification *note) { + if ([note.userInfo[AVAudioSessionInterruptionTypeKey] unsignedIntegerValue] == AVAudioSessionInterruptionTypeBegan) + dispatch_async(queue, ^{ [weakSelf stop]; }); + }]; + if (_needsAudio) { + if (AVAudioSession.sharedInstance.recordPermission != AVAudioSessionRecordPermissionGranted) { + [self fail:SCMovieError(@"Microphone permission has not been granted")]; return; + } +#if SIMCAM_RECORDING_TESTS + if (getenv("SC_TEST_TONE")) { + // Test-build-only hardware boundary. Uses the production PCM encoder/muxer. + _sampleRate = 48000; + _testAudioTimer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, _queue); + __block uint64_t frame = 0; + CMTime origin = _origin; + dispatch_source_set_timer(_testAudioTimer, dispatch_time(DISPATCH_TIME_NOW, 0), 20 * NSEC_PER_MSEC, NSEC_PER_MSEC); + dispatch_source_set_event_handler(_testAudioTimer, ^{ + // A microphone clock keeps sampling when a dispatch timer coalesces. + // Catch up samples rather than shortening audio when the encoder queue is busy. + double elapsed = CMTimeGetSeconds(CMTimeSubtract(CMClockGetTime(CMClockGetHostTimeClock()), origin)); + uint64_t target = (uint64_t)(elapsed * 48000); + if (target <= frame) return; + NSUInteger count = (NSUInteger)(target - frame); + CMTime pts = CMTimeAdd(origin, CMTimeMake(frame, 48000)); + NSMutableData *data = [NSMutableData dataWithLength:count * sizeof(float)]; + float *samples = data.mutableBytes; + for (NSUInteger i = 0; i < count; i++, frame++) samples[i] = 0.2 * sin(2 * M_PI * 440 * frame / 48000.0); + [weakSelf appendPCM:data time:pts]; + }); + dispatch_resume(_testAudioTimer); + } else { +#endif + // The caller owns AVAudioSession category/activation. Do not restore or replace its lease. + _engine = [AVAudioEngine new]; + SCRecordingLog(@"obtaining microphone input node"); + AVAudioInputNode *input = _engine.inputNode; + AVAudioFormat *format = [input outputFormatForBus:0]; + SCRecordingLog([NSString stringWithFormat:@"microphone format %@", format]); + _sampleRate = format.sampleRate; + if (!format.channelCount || _sampleRate <= 0 || format.commonFormat != AVAudioPCMFormatFloat32) { + [self fail:SCMovieError(@"Microphone has no usable float PCM input format")]; return; + } + _engineObserver = [NSNotificationCenter.defaultCenter addObserverForName:AVAudioEngineConfigurationChangeNotification object:_engine queue:nil usingBlock:^(NSNotification *note) { + dispatch_async(queue, ^{ [weakSelf fail:SCMovieError(@"Microphone route changed during recording")]; }); + }]; + [input installTapOnBus:0 bufferSize:1024 format:format block:^(AVAudioPCMBuffer *buffer, AVAudioTime *when) { + SCMovieRecorder *recorder = weakSelf; + if (!recorder || !when.hostTimeValid || !buffer.frameLength) return; + // Copy the microphone channel before the engine recycles its buffer. + NSMutableData *pcm = [NSMutableData dataWithLength:buffer.frameLength * sizeof(float)]; + float *dst = pcm.mutableBytes; + const float *src = buffer.floatChannelData[0]; + for (AVAudioFrameCount i = 0; i < buffer.frameLength; i++) dst[i] = src[i * buffer.stride]; + CMTime pts = CMClockMakeHostTimeFromSystemUnits(when.hostTime); + dispatch_async(queue, ^{ [recorder appendPCM:pcm time:pts]; }); + }]; + _tapInstalled = YES; + NSError *error = nil; + SCRecordingLog(@"starting audio engine"); + if (![_engine startAndReturnError:&error]) { [self fail:error ?: SCMovieError(@"Microphone failed to start")]; return; } + SCRecordingLog(@"audio engine running"); +#if SIMCAM_RECORDING_TESTS + } +#endif + } + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 3 * NSEC_PER_SEC), _queue, ^{ + SCMovieRecorder *recorder = weakSelf; + if (recorder && !recorder->_started && !recorder->_stopping && !recorder.finished) + [recorder fail:SCMovieError(@"Timed out waiting for video and microphone samples")]; + }); +} + +- (BOOL)prepareWriter:(CVPixelBufferRef)pixels { + NSError *error = nil; + _writer = [[AVAssetWriter alloc] initWithURL:_url fileType:AVFileTypeQuickTimeMovie error:&error]; + if (!_writer) { [self fail:error]; return NO; } + _width = CVPixelBufferGetWidth(pixels); _height = CVPixelBufferGetHeight(pixels); + _video = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:@{ + AVVideoCodecKey: AVVideoCodecTypeH264, AVVideoWidthKey: @(_width), AVVideoHeightKey: @(_height)}]; + _video.expectsMediaDataInRealTime = YES; + _sourcePortraitRotation = SCFramePortraitRotation(pixels); + _video.transform = SCVideoDisplayTransform(CGSizeMake(_width, _height), _rotation, _mirrored, _sourcePortraitRotation); + _adaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:_video sourcePixelBufferAttributes:nil]; + if (![_writer canAddInput:_video]) { [self fail:SCMovieError(@"Video encoder unavailable")]; return NO; } + [_writer addInput:_video]; + if (_needsAudio) { + _audio = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeAudio outputSettings:@{ + AVFormatIDKey: @(kAudioFormatMPEG4AAC), AVSampleRateKey: @(_sampleRate), AVNumberOfChannelsKey: @1, AVEncoderBitRateKey: @96000}]; + _audio.expectsMediaDataInRealTime = YES; + if (![_writer canAddInput:_audio]) { [self fail:SCMovieError(@"Audio encoder unavailable")]; return NO; } + [_writer addInput:_audio]; + } + if (![_writer startWriting]) { [self fail:_writer.error]; return NO; } + [_writer startSessionAtSourceTime:_origin]; + NSArray *pending = [_pendingAudio copy]; [_pendingAudio removeAllObjects]; + for (NSArray *entry in pending) [self appendPCM:entry[0] time:[entry[1] CMTimeValue]]; + return !self.finished; +} + +- (void)appendVideo:(CVPixelBufferRef)pixels time:(CMTime)time { + if (_stopping || self.finished || !self.recording) return; + if (_started && _needsAudio && CMTIME_IS_VALID(_lastAudioEnd) && CMTimeGetSeconds(CMTimeSubtract(time, _lastAudioEnd)) > 1) { + [self fail:SCMovieError(@"Microphone stopped delivering samples")]; return; + } + if (!_writer && ![self prepareWriter:pixels]) return; + if (_width != CVPixelBufferGetWidth(pixels) || _height != CVPixelBufferGetHeight(pixels)) { + [self fail:SCMovieError(@"Video dimensions changed during recording")]; return; + } + if (_sourcePortraitRotation != SCFramePortraitRotation(pixels)) { + [self fail:SCMovieError(@"Video source orientation changed during recording")]; return; + } + if (CMTIME_IS_VALID(_lastVideo) && CMTimeCompare(time, _lastVideo) <= 0) return; + if (!_video.readyForMoreMediaData) return; + if (![_adaptor appendPixelBuffer:pixels withPresentationTime:time]) { [self fail:_writer.error]; return; } + _lastVideo = time; _hasVideo = YES; + [self notifyStartedIfReady]; +} + +- (void)appendPCM:(NSData *)pcm time:(CMTime)time { +#if SIMCAM_RECORDING_TESTS + if (getenv("SC_TEST_NO_AUDIO")) return; +#endif + if (_stopping || self.finished || CMTimeCompare(time, _origin) < 0) return; + if (!_writer) { + if (_pendingAudio.count < 100) [_pendingAudio addObject:@[pcm, [NSValue valueWithCMTime:time]]]; + return; + } + if (CMTIME_IS_VALID(_lastAudio) && CMTimeCompare(time, _lastAudio) <= 0) return; + if (!_audio.readyForMoreMediaData) { [self fail:SCMovieError(@"Audio encoder cannot keep up with microphone samples")]; return; } + AudioStreamBasicDescription asbd = {0}; + asbd.mSampleRate = _sampleRate; asbd.mFormatID = kAudioFormatLinearPCM; + asbd.mFormatFlags = kAudioFormatFlagIsFloat | kAudioFormatFlagIsPacked; + asbd.mBytesPerPacket = asbd.mBytesPerFrame = sizeof(float); asbd.mFramesPerPacket = 1; + asbd.mChannelsPerFrame = 1; asbd.mBitsPerChannel = 32; + CMAudioFormatDescriptionRef format = NULL; + CMBlockBufferRef block = NULL; CMSampleBufferRef sample = NULL; + OSStatus status = CMAudioFormatDescriptionCreate(kCFAllocatorDefault, &asbd, 0, NULL, 0, NULL, NULL, &format); + if (!status) status = CMBlockBufferCreateWithMemoryBlock(kCFAllocatorDefault, NULL, pcm.length, kCFAllocatorDefault, NULL, 0, pcm.length, 0, &block); + if (!status) status = CMBlockBufferReplaceDataBytes(pcm.bytes, block, 0, pcm.length); + if (!status) status = CMAudioSampleBufferCreateReadyWithPacketDescriptions(kCFAllocatorDefault, block, format, pcm.length / sizeof(float), time, NULL, &sample); + BOOL appended = !status && [_audio appendSampleBuffer:sample]; + if (sample) CFRelease(sample); if (block) CFRelease(block); if (format) CFRelease(format); + if (!appended) { [self fail:_writer.error ?: SCMovieError(@"Could not encode microphone PCM")]; return; } + _lastAudio = time; + _lastAudioEnd = CMTimeAdd(time, CMTimeMake(pcm.length / sizeof(float), (int32_t)_sampleRate)); + _hasAudio = YES; + [self notifyStartedIfReady]; +} + +- (void)notifyStartedIfReady { + if (_started || !_hasVideo || (_needsAudio && !_hasAudio)) return; + _started = YES; + SCRecordingLog(@"didStart"); + AVCaptureMovieFileOutput *output = _output; + id delegate = _delegate; + dispatch_async(dispatch_get_main_queue(), ^{ + if ([delegate respondsToSelector:@selector(captureOutput:didStartRecordingToOutputFileAtURL:fromConnections:)]) + [delegate captureOutput:output didStartRecordingToOutputFileAtURL:self->_url fromConnections:self->_connections]; + }); +} + +- (void)stopAudio { + SCRecordingLog(@"stopping audio"); +#if SIMCAM_RECORDING_TESTS + if (_testAudioTimer) { dispatch_source_cancel(_testAudioTimer); _testAudioTimer = nil; } +#endif + if (_interruptionObserver) [NSNotificationCenter.defaultCenter removeObserver:_interruptionObserver]; + if (_engineObserver) [NSNotificationCenter.defaultCenter removeObserver:_engineObserver]; + _interruptionObserver = _engineObserver = nil; + if (_engine) { [_engine stop]; if (_tapInstalled) [_engine.inputNode removeTapOnBus:0]; _tapInstalled = NO; _engine = nil; } + SCRecordingLog(@"audio stopped"); + [_pendingAudio removeAllObjects]; +} + +- (void)stop { + SCRecordingLog(@"stop requested"); + if (_stopping || self.finished) return; + _stopping = YES; self.recording = NO; + [self stopAudio]; + if (!_started) { [self fail:SCMovieError(@"Recording stopped before audio/video startup completed")]; return; } + [_video markAsFinished]; [_audio markAsFinished]; + [_writer finishWritingWithCompletionHandler:^{ + dispatch_async(self->_queue, ^{ + [self complete:self->_writer.status == AVAssetWriterStatusCompleted ? nil : self->_writer.error ?: SCMovieError(@"Movie finalization failed")]; + }); + }]; +} + +- (void)fail:(NSError *)error { + if (self.finished || (_stopping && _writer.status == AVAssetWriterStatusCompleted)) return; + _stopping = YES; self.recording = NO; + [self stopAudio]; + if (_writer.status == AVAssetWriterStatusWriting) [_writer cancelWriting]; + [self complete:error ?: SCMovieError(@"Movie recording failed")]; +} + +- (void)complete:(NSError *)error { + if (self.finished) return; + self.finished = YES; self.recording = NO; + SCRecordingLog([NSString stringWithFormat:@"didFinish error=%@", error]); + AVCaptureMovieFileOutput *output = _output; + id delegate = _delegate; + _delegate = nil; + _output = nil; + dispatch_async(dispatch_get_main_queue(), ^{ + [delegate captureOutput:output didFinishRecordingToOutputFileAtURL:self->_url fromConnections:self->_connections error:error]; + }); +} +@end diff --git a/SimulatorInjection/SCVideoGeometry.h b/SimulatorInjection/SCVideoGeometry.h new file mode 100644 index 0000000..383ccf5 --- /dev/null +++ b/SimulatorInjection/SCVideoGeometry.h @@ -0,0 +1,31 @@ +#pragma once +#import +#import +#import + +// Source metadata belongs to the copied frame, including cached/repeated frames. +#define SCFrameSourceKindKey CFSTR("SimulatorCamera.SourceKind") + +static inline CGFloat SCFramePortraitRotation(CVPixelBufferRef pixels) { + CFNumberRef kind = (CFNumberRef)CVBufferCopyAttachment(pixels, SCFrameSourceKindKey, NULL); + int value = 0; + if (kind && CFGetTypeID(kind) == CFNumberGetTypeID()) CFNumberGetValue(kind, kCFNumberIntType, &value); + if (kind) CFRelease(kind); + // SimCamSourceKind.macCamera = 1. Host webcam pixels are already upright. + // Treat that image as the emulated camera's portrait (90°) view, rather + // than pretending it is an unrotated iPhone sensor image. Other sources + // retain their existing sensor-coordinate contract. + return value == 1 ? 90 : 0; +} + +// Top-left image/display coordinates. The source-to-sensor normalization and +// requested connection rotation compose here; no pixels need resampling for MOV. +static inline CGAffineTransform SCVideoDisplayTransform(CGSize size, CGFloat requestedRotation, + BOOL mirrored, CGFloat portraitRotation) { + CGAffineTransform transform = CGAffineTransformMakeRotation((requestedRotation - portraitRotation) * M_PI / 180.0); + // Mirror about the displayed image's vertical axis, after rotation. + if (mirrored) { transform.a = -transform.a; transform.c = -transform.c; } + CGRect bounds = CGRectApplyAffineTransform((CGRect){CGPointZero, size}, transform); + transform.tx -= bounds.origin.x; transform.ty -= bounds.origin.y; + return transform; +} diff --git a/SimulatorInjection/SimCamInject.m b/SimulatorInjection/SimCamInject.m index 2d21c92..9324785 100644 --- a/SimulatorInjection/SimCamInject.m +++ b/SimulatorInjection/SimCamInject.m @@ -11,6 +11,7 @@ // AVCaptureDeviceInput, AVCaptureSession running state + notifications, // AVCaptureVideoDataOutput (BGRA sample buffers), AVCaptureVideoPreviewLayer, // AVCaptureMetadataOutput (QR codes), AVCapturePhotoOutput (JPEG/HEIF data). +// AVCaptureMovieFileOutput (H.264/AAC MOV, microphone via AVAudioEngine). // Without the Mac app running, a built-in animated test pattern is shown. // // Simulator only. Build: SimulatorInjection/build.sh @@ -22,7 +23,12 @@ #import #import #import +#import +#import +#import #import "SCFrameStore.h" +#import "SCMovieRecorder.h" +#import "SCVideoGeometry.h" #if !TARGET_OS_SIMULATOR #error "SimCamInject is for the iOS Simulator only" @@ -147,10 +153,12 @@ @implementation SCConnection { @public AVCaptureVideoOrientation _orientation; @public CGFloat _rotation; @public BOOL _mirrored; + @public AVMediaType _mediaType; + @public __weak AVCaptureOutput *_output; } - (NSArray *)inputPorts { return @[]; } -- (AVMediaType)mediaType { return AVMediaTypeVideo; } // private, read by AVCaptureSession internals -- (AVCaptureOutput *)output { return nil; } +- (AVMediaType)mediaType { return _mediaType ?: AVMediaTypeVideo; } // read by AVCaptureSession internals +- (AVCaptureOutput *)output { return _output; } - (AVCaptureVideoPreviewLayer *)videoPreviewLayer { return nil; } - (BOOL)isEnabled { return YES; } - (void)setEnabled:(BOOL)e {} @@ -177,6 +185,16 @@ - (void)setVideoMinFrameDuration:(CMTime)t {} @end static SCConnection *SCNewConnection(void) { + static dispatch_once_t once; + dispatch_once(&once, ^{ + // This fake never initializes AVCaptureConnection's native graph. Its + // dealloc assumes that graph exists and crashes when a preview releases + // its connection. NSObject still runs ARC ivar/weak/association cleanup. + SEL dealloc = sel_registerName("dealloc"); + Method rootDealloc = class_getInstanceMethod([NSObject class], dealloc); + class_addMethod([SCConnection class], dealloc, method_getImplementation(rootDealloc), + method_getTypeEncoding(rootDealloc)); + }); SCConnection *c = class_createInstance([SCConnection class], 0); c->_orientation = AVCaptureVideoOrientationPortrait; c->_rotation = 90; @@ -230,6 +248,50 @@ - (AVCaptureResolvedPhotoSettings *)resolvedSettings { return nil; } static double gStoreLatestAt; // when gStoreLatest was read (seconds since 1970) static uint64_t gFrameCount; static dispatch_queue_t gFrameQueue; +static int gBridgeSocket = -1; // A live socket is the host hardware lease; owned on gFrameQueue. + +static void SCResetStore(void) { + if (gStore) { munmap(gStore, SC_FRAME_FILE_SIZE); gStore = NULL; } + if (gLatest) { CVPixelBufferRelease(gLatest); gLatest = NULL; } + if (gStoreLatest) { CVPixelBufferRelease(gStoreLatest); gStoreLatest = NULL; } + gLastSeq = 0; + gStoreLatestAt = 0; +} + +static void SCReleaseBridge(void) { + if (gBridgeSocket < 0) return; + // The next take gets a new frame-store inode. Drop mappings before releasing + // the host writer, rather than showing a stale frame from the previous take. + SCResetStore(); + close(gBridgeSocket); + gBridgeSocket = -1; +} + +static BOOL SCAcquireBridge(void) { + const char *path = getenv("SIMCAM_BRIDGE_SOCKET"); + if (!path || !*path || gBridgeSocket >= 0) return YES; + struct sockaddr_un address = { .sun_family = AF_UNIX }; + if (strlen(path) >= sizeof(address.sun_path)) return NO; + strlcpy(address.sun_path, path, sizeof(address.sun_path)); + int fd = socket(AF_UNIX, SOCK_STREAM, 0); + if (fd < 0) return NO; + fcntl(fd, F_SETFD, FD_CLOEXEC); + int noSignal = 1; + setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &noSignal, sizeof(noSignal)); + struct timeval timeout = { .tv_sec = 12 }; + setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &timeout, sizeof(timeout)); + setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &timeout, sizeof(timeout)); + char ready = 0; + if (connect(fd, (struct sockaddr *)&address, sizeof(address)) != 0 || + send(fd, "CAPTURE\n", 8, 0) != 8 || recv(fd, &ready, 1, 0) != 1 || ready != 'R') { + close(fd); + SCLog(@"host camera bridge unavailable or failed to start"); + return NO; + } + SCResetStore(); + gBridgeSocket = fd; + return YES; +} static void SCMapStore(void) { if (gStore) return; @@ -259,7 +321,7 @@ static CVPixelBufferRef SCReadStoreOnce(void) { if (s1 & 1) return NULL; // Treat a writer that stopped more than 3 s ago as gone. if ([NSDate date].timeIntervalSince1970 - h->hostTime > 3 && h->hostTime > 1e9) return NULL; - uint32_t w = h->width, ht = h->height, bpr = h->bytesPerRow; + uint32_t w = h->width, ht = h->height, bpr = h->bytesPerRow, sourceKind = h->sourceKind; if (!w || !ht || w > SC_FRAME_MAX_WIDTH || ht > SC_FRAME_MAX_HEIGHT || bpr < w * 4 || (uint64_t)bpr * ht > SC_FRAME_MAX_BYTES) return NULL; if (s1 == gLastSeq && gStoreLatest && CVPixelBufferGetWidth(gStoreLatest) == w) return CVPixelBufferRetain(gStoreLatest); CVPixelBufferRef pb = SCNewBuffer(w, ht); @@ -272,6 +334,7 @@ static CVPixelBufferRef SCReadStoreOnce(void) { CVPixelBufferUnlockBaseAddress(pb, 0); uint64_t s2 = atomic_load_explicit((_Atomic uint64_t *)&h->seq, memory_order_acquire); if (s1 != s2) { CVPixelBufferRelease(pb); return NULL; } // torn; keep previous + CVBufferSetAttachment(pb, SCFrameSourceKindKey, (__bridge CFNumberRef)@(sourceKind), kCVAttachmentMode_ShouldPropagate); gLastSeq = s1; return pb; } @@ -323,19 +386,36 @@ static CVPixelBufferRef SCTestPattern(uint64_t n) { #pragma mark - Session registry +static const void *kSCMovieKey = &kSCMovieKey; +static void SCUpdateTimer(void); +static void sc_stopRunning(AVCaptureSession *self, SEL cmd); +@interface SCWeakSession : NSObject +@property (nonatomic, weak) AVCaptureSession *session; +@end +@implementation SCWeakSession @end + @interface SCSessionState : NSObject -@property (nonatomic) BOOL usesFake; +@property (nonatomic) BOOL usesFake; // Sticky: an injected session never resumes the native stub graph. @property (nonatomic) BOOL running; @property (nonatomic, strong) NSMutableArray *inputs; -@property (nonatomic, strong) NSHashTable *outputs; // weak +@property (nonatomic, strong) NSHashTable *outputs; @end @implementation SCSessionState -- (instancetype)init { if ((self = [super init])) { _inputs = [NSMutableArray array]; _outputs = [NSHashTable weakObjectsHashTable]; } return self; } +- (instancetype)init { if ((self = [super init])) { _inputs = [NSMutableArray array]; _outputs = [NSHashTable hashTableWithOptions:NSPointerFunctionsStrongMemory]; } return self; } +- (void)dealloc { + for (AVCaptureOutput *output in _outputs.allObjects) { + SCMovieRecorder *recorder = objc_getAssociatedObject(output, kSCMovieKey); + if (recorder) dispatch_async(gFrameQueue, ^{ [recorder stop]; }); + } + dispatch_async(gFrameQueue, ^{ SCUpdateTimer(); }); +} @end static const void *kSCStateKey = &kSCStateKey; static const void *kSCSessionKey = &kSCSessionKey; // output/layer -> session (weak box) static const void *kSCConnKey = &kSCConnKey; // output/layer -> SCConnection +static const void *kSCAudioConnKey = &kSCAudioConnKey; +static const void *kSCPreviewSessionKey = &kSCPreviewSessionKey; static const void *kSCMetaTypesKey = &kSCMetaTypesKey; static const void *kSCRectKey = &kSCRectKey; static NSHashTable *gRunning; // running fake sessions (weak) @@ -350,12 +430,43 @@ - (instancetype)init { if ((self = [super init])) { _inputs = [NSMutableArray ar return st; } +static AVCaptureSession *SCSessionFor(id object) { + SCWeakSession *box = objc_getAssociatedObject(object, kSCSessionKey); + return box.session; +} + +static NSArray *SCOutputs(AVCaptureSession *session) { + SCSessionState *state = SCState(session); + @synchronized (state) { return state.outputs.allObjects; } +} + static SCConnection *SCConnectionFor(id obj) { SCConnection *c = objc_getAssociatedObject(obj, kSCConnKey); if (!c) { c = SCNewConnection(); objc_setAssociatedObject(obj, kSCConnKey, c, OBJC_ASSOCIATION_RETAIN); } return c; } +static BOOL SCHasAudio(AVCaptureSession *session) { + for (AVCaptureInput *input in session.inputs) + if ([input isKindOfClass:AVCaptureDeviceInput.class] && [((AVCaptureDeviceInput *)input).device hasMediaType:AVMediaTypeAudio]) return YES; + return NO; +} + +static BOOL SCHasVideo(AVCaptureSession *session) { + for (AVCaptureInput *input in SCState(session).inputs) + if ([input isKindOfClass:SCDeviceInput.class]) return YES; + return NO; +} + +static SCConnection *SCAudioConnectionFor(AVCaptureOutput *output) { + SCConnection *c = objc_getAssociatedObject(output, kSCAudioConnKey); + if (!c) { + c = SCNewConnection(); c->_mediaType = AVMediaTypeAudio; c->_output = output; + objc_setAssociatedObject(output, kSCAudioConnKey, c, OBJC_ASSOCIATION_RETAIN); + } + return c; +} + static CMSampleBufferRef SCSampleBuffer(CVPixelBufferRef pb, CMTime pts) { CMVideoFormatDescriptionRef fmt = NULL; if (CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pb, &fmt) != noErr) return NULL; @@ -395,9 +506,25 @@ static void SCDeliverMetadata(AVCaptureMetadataOutput *out, AVCaptureSession *se } static void SCTick(void) { + if (gBridgeSocket >= 0) { + char byte; + ssize_t count = recv(gBridgeSocket, &byte, 1, MSG_PEEK | MSG_DONTWAIT); + if (count == 0 || (count < 0 && errno != EAGAIN && errno != EWOULDBLOCK && errno != EINTR)) { + SCReleaseBridge(); + [gLock lock]; NSArray *sessions = gRunning.allObjects; [gLock unlock]; + NSError *error = [NSError errorWithDomain:AVFoundationErrorDomain code:AVErrorUnknown + userInfo:@{NSLocalizedDescriptionKey: @"The host camera bridge disconnected"}]; + for (AVCaptureSession *session in sessions) { + sc_stopRunning(session, @selector(stopRunning)); + [[NSNotificationCenter defaultCenter] postNotificationName:AVCaptureSessionRuntimeErrorNotification + object:session userInfo:@{AVCaptureSessionErrorKey: error}]; + } + return; + } + } gFrameCount++; CVPixelBufferRef pb = SCReadStore(); - if (!pb) pb = SCTestPattern(gFrameCount); + if (!pb && !getenv("SIMCAM_BRIDGE_SOCKET")) pb = SCTestPattern(gFrameCount); if (!pb) return; if (gLatest) CVPixelBufferRelease(gLatest); gLatest = CVPixelBufferRetain(pb); @@ -409,8 +536,14 @@ static void SCTick(void) { [gLock unlock]; for (AVCaptureSession *s in sessions) { - for (AVCaptureOutput *out in SCState(s).outputs.allObjects) { - if ([out isKindOfClass:[AVCaptureVideoDataOutput class]]) { + for (AVCaptureOutput *out in SCOutputs(s)) { + if ([out isKindOfClass:[AVCaptureMovieFileOutput class]]) { +#if SIMCAM_RECORDING_TESTS + if (getenv("SC_TEST_NO_VIDEO")) continue; +#endif + SCMovieRecorder *recorder = objc_getAssociatedObject(out, kSCMovieKey); + [recorder appendVideo:pb time:pts]; + } else if ([out isKindOfClass:[AVCaptureVideoDataOutput class]]) { AVCaptureVideoDataOutput *vo = (AVCaptureVideoDataOutput *)out; id d = vo.sampleBufferDelegate; dispatch_queue_t q = vo.sampleBufferCallbackQueue; @@ -428,29 +561,48 @@ static void SCTick(void) { if (layers.count && gFrameCount % 2 == 0) { // preview at 15 fps is plenty CIImage *ci = [CIImage imageWithCVPixelBuffer:pb]; static CIContext *ctx; if (!ctx) ctx = [CIContext context]; - CGImageRef img = [ctx createCGImage:ci fromRect:ci.extent]; + NSMutableArray *frames = [NSMutableArray array]; + CGFloat portraitRotation = SCFramePortraitRotation(pb); + CGSize size = CGSizeMake(CVPixelBufferGetWidth(pb), CVPixelBufferGetHeight(pb)); + for (AVCaptureVideoPreviewLayer *layer in layers) { + CIImage *display = ci; + if (portraitRotation != 0) { + SCConnection *connection = SCConnectionFor(layer); + CGAffineTransform transform = SCVideoDisplayTransform(size, connection->_rotation, connection->_mirrored, portraitRotation); + CGRect bounds = CGRectApplyAffineTransform((CGRect){CGPointZero, size}, transform); + // Core Image uses bottom-left coordinates; the movie/display transform + // uses top-left coordinates. Convert both ends, not the rotation rule. + CGAffineTransform ciTransform = CGAffineTransformConcat(CGAffineTransformMake(1, 0, 0, -1, 0, size.height), transform); + ciTransform = CGAffineTransformConcat(ciTransform, CGAffineTransformMake(1, 0, 0, -1, 0, bounds.size.height)); + display = [ci imageByApplyingTransform:ciTransform]; + } + CGImageRef image = [ctx createCGImage:display fromRect:display.extent]; + if (image) { [frames addObject:@[layer, (__bridge id)image]]; CGImageRelease(image); } + } dispatch_async(dispatch_get_main_queue(), ^{ [CATransaction begin]; [CATransaction setDisableActions:YES]; // no cross-fade between frames - for (AVCaptureVideoPreviewLayer *l in layers) { + for (NSArray *frame in frames) { + AVCaptureVideoPreviewLayer *l = frame[0]; AVCaptureSession *s = l.session; if (!s || !SCState(s).running) continue; NSString *g = l.videoGravity; l.contentsGravity = [g isEqualToString:AVLayerVideoGravityResize] ? kCAGravityResize : [g isEqualToString:AVLayerVideoGravityResizeAspect] ? kCAGravityResizeAspect : kCAGravityResizeAspectFill; - l.contents = (__bridge id)img; + l.contents = frame[1]; } [CATransaction commit]; - CGImageRelease(img); }); } CVPixelBufferRelease(pb); } static void SCUpdateTimer(void) { + // Sessions are controlled on caller queues; timer ownership stays on the frame queue. + dispatch_async(gFrameQueue, ^{ [gLock lock]; - BOOL any = gRunning.count > 0; + BOOL any = gRunning.allObjects.count > 0; [gLock unlock]; if (any && !gTimer) { gTimer = dispatch_source_create(DISPATCH_SOURCE_TYPE_TIMER, 0, 0, gFrameQueue); @@ -461,6 +613,8 @@ static void SCUpdateTimer(void) { dispatch_source_cancel(gTimer); gTimer = nil; } + if (!any) SCReleaseBridge(); + }); } #pragma mark - AVCaptureDevice discovery @@ -505,9 +659,12 @@ static void SCUpdateTimer(void) { in->_scDevice = (SCDevice *)d; return in; } -static id sc_inputInit(id self, SEL _cmd, AVCaptureDevice *d, NSError **e) { +// IMPs replacing init-family methods must preserve ARC's consumed-self/+1-return ABI. +typedef id (*SCInputInitializer)(id __attribute__((ns_consumed)), SEL, AVCaptureDevice *, NSError **) __attribute__((ns_returns_retained)); +static id sc_inputInit(id __attribute__((ns_consumed)) self, SEL _cmd, AVCaptureDevice *d, NSError **e) __attribute__((ns_returns_retained)); +static id sc_inputInit(id __attribute__((ns_consumed)) self, SEL _cmd, AVCaptureDevice *d, NSError **e) { if ([d isKindOfClass:[SCDevice class]]) return SCMakeInput(d); - return ((id (*)(id, SEL, AVCaptureDevice *, NSError **))o_inputInit)(self, _cmd, d, e); + return ((SCInputInitializer)o_inputInit)(self, _cmd, d, e); } static id sc_inputFactory(id self, SEL _cmd, AVCaptureDevice *d, NSError **e) { if ([d isKindOfClass:[SCDevice class]]) return SCMakeInput(d); @@ -516,27 +673,33 @@ static id sc_inputFactory(id self, SEL _cmd, AVCaptureDevice *d, NSError **e) { #pragma mark - AVCaptureSession -static IMP o_canAddInput, o_addInput, o_removeInput, o_inputs, o_addOutput, o_removeOutput, - o_startRunning, o_stopRunning, o_isRunning, o_canAddOutput, o_addInputNoConn; +static IMP o_canAddInput, o_addInput, o_removeInput, o_inputs, o_outputs, o_addOutput, o_removeOutput, + o_startRunning, o_stopRunning, o_isRunning, o_canAddOutput, o_addInputNoConn; +static void sc_stopRunning(AVCaptureSession *self, SEL cmd); static BOOL sc_canAddInput(AVCaptureSession *self, SEL _cmd, AVCaptureInput *in) { if ([in isKindOfClass:[SCDeviceInput class]]) return YES; + if (SCState(self).usesFake && [in isKindOfClass:AVCaptureDeviceInput.class] && [((AVCaptureDeviceInput *)in).device hasMediaType:AVMediaTypeAudio]) return YES; return ((BOOL (*)(id, SEL, id))o_canAddInput)(self, _cmd, in); } static void sc_addInput(AVCaptureSession *self, SEL _cmd, AVCaptureInput *in) { - if ([in isKindOfClass:[SCDeviceInput class]]) { - SCSessionState *st = SCState(self); + SCSessionState *st = SCState(self); + BOOL fakeVideo = [in isKindOfClass:SCDeviceInput.class]; + BOOL audio = [in isKindOfClass:AVCaptureDeviceInput.class] && [((AVCaptureDeviceInput *)in).device hasMediaType:AVMediaTypeAudio]; + if (fakeVideo || (st.usesFake && audio)) { if (![st.inputs containsObject:in]) [st.inputs addObject:in]; - st.usesFake = YES; + if (fakeVideo) st.usesFake = YES; + // Keep a real, permission-checked microphone input, but do not add it to + // the Simulator's stub graph: that graph emits spurious runtime failures. return; } ((void (*)(id, SEL, id))o_addInput)(self, _cmd, in); } static void sc_removeInput(AVCaptureSession *self, SEL _cmd, AVCaptureInput *in) { - if ([in isKindOfClass:[SCDeviceInput class]]) { - SCSessionState *st = SCState(self); + SCSessionState *st = SCState(self); + if ([st.inputs containsObject:in]) { + if (st.running) sc_stopRunning(self, @selector(stopRunning)); [st.inputs removeObject:in]; - st.usesFake = st.inputs.count > 0; return; } ((void (*)(id, SEL, id))o_removeInput)(self, _cmd, in); @@ -551,21 +714,40 @@ static BOOL sc_canAddOutput(AVCaptureSession *self, SEL _cmd, AVCaptureOutput *o return ok || SCState(self).usesFake; } static void sc_addOutput(AVCaptureSession *self, SEL _cmd, AVCaptureOutput *o) { - [SCState(self).outputs addObject:o]; - objc_setAssociatedObject(o, kSCSessionKey, self, OBJC_ASSOCIATION_ASSIGN); + SCSessionState *state = SCState(self); + @synchronized (state) { [state.outputs addObject:o]; } + SCWeakSession *box = [SCWeakSession new]; box.session = self; + objc_setAssociatedObject(o, kSCSessionKey, box, OBJC_ASSOCIATION_RETAIN); + if (state.usesFake) return; @try { ((void (*)(id, SEL, id))o_addOutput)(self, _cmd, o); } @catch (NSException *ex) { SCLog(@"addOutput fallback (%@)", ex.reason); } } static void sc_removeOutput(AVCaptureSession *self, SEL _cmd, AVCaptureOutput *o) { - [SCState(self).outputs removeObject:o]; - @try { ((void (*)(id, SEL, id))o_removeOutput)(self, _cmd, o); } @catch (NSException *ex) {} + SCMovieRecorder *recorder = objc_getAssociatedObject(o, kSCMovieKey); + if (recorder) dispatch_async(gFrameQueue, ^{ [recorder stop]; }); + SCSessionState *state = SCState(self); + @synchronized (state) { [state.outputs removeObject:o]; } + if (!state.usesFake) { @try { ((void (*)(id, SEL, id))o_removeOutput)(self, _cmd, o); } @catch (NSException *ex) {} } + objc_setAssociatedObject(o, kSCSessionKey, nil, OBJC_ASSOCIATION_RETAIN); +} +static NSArray *sc_outputs(AVCaptureSession *self, SEL cmd) { + if (SCState(self).usesFake) return SCOutputs(self); + return ((id (*)(id, SEL))o_outputs)(self, cmd); } static void sc_startRunning(AVCaptureSession *self, SEL _cmd) { SCSessionState *st = SCState(self); if (!st.usesFake) { ((void (*)(id, SEL))o_startRunning)(self, _cmd); return; } + if (!SCHasVideo(self)) return; if (st.running) return; - st.running = YES; - [gLock lock]; [gRunning addObject:self]; [gLock unlock]; + __block BOOL ready = NO; + dispatch_sync(gFrameQueue, ^{ + ready = SCAcquireBridge(); + if (ready) { + st.running = YES; + [gLock lock]; [gRunning addObject:self]; [gLock unlock]; + } + }); + if (!ready) return; SCUpdateTimer(); SCLog(@"session running with the injected camera"); [[NSNotificationCenter defaultCenter] postNotificationName:AVCaptureSessionDidStartRunningNotification object:self]; @@ -574,21 +756,27 @@ static void sc_stopRunning(AVCaptureSession *self, SEL _cmd) { SCSessionState *st = SCState(self); if (!st.usesFake) { ((void (*)(id, SEL))o_stopRunning)(self, _cmd); return; } if (!st.running) return; + for (AVCaptureOutput *output in SCOutputs(self)) { + SCMovieRecorder *recorder = objc_getAssociatedObject(output, kSCMovieKey); + if (recorder) dispatch_async(gFrameQueue, ^{ [recorder stop]; }); + } st.running = NO; [gLock lock]; [gRunning removeObject:self]; [gLock unlock]; SCUpdateTimer(); [[NSNotificationCenter defaultCenter] postNotificationName:AVCaptureSessionDidStopRunningNotification object:self]; } static IMP o_beginConfig, o_commitConfig; -// A session on the injected camera has no real graph to configure; the stock -// commit walks output connections and would hit the fake ones. +// Never commit an injected graph into the native simulator stub. Even an empty +// native commit emits runtime errors that would interrupt a valid emulated take. +// If begin preceded the fake input, its native transaction stays quarantined +// until session destruction. usesFake stays set even after removing the camera. static void sc_beginConfig(AVCaptureSession *self, SEL _cmd) { - SCSessionState *st = objc_getAssociatedObject(self, kSCStateKey); + SCSessionState *st = SCState(self); if (st.usesFake) return; ((void (*)(id, SEL))o_beginConfig)(self, _cmd); } static void sc_commitConfig(AVCaptureSession *self, SEL _cmd) { - SCSessionState *st = objc_getAssociatedObject(self, kSCStateKey); + SCSessionState *st = SCState(self); if (st.usesFake) return; ((void (*)(id, SEL))o_commitConfig)(self, _cmd); } @@ -601,20 +789,32 @@ static BOOL sc_isRunning(AVCaptureSession *self, SEL _cmd) { #pragma mark - Outputs and preview layer static IMP o_connectionWithMediaType, o_connections, o_setMetaTypes, o_metaTypes, o_availableMetaTypes, - o_layerConnection, o_layerSetSession, o_layerInitSession, o_layerInitNoConn, o_transformMeta, + o_layerConnection, o_layerSession, o_layerSetSession, o_layerInitSession, o_layerInitNoConn, o_transformMeta, o_capturePhoto, o_setRectOfInterest, o_rectOfInterest; static BOOL SCOutputIsFake(id out) { - AVCaptureSession *s = objc_getAssociatedObject(out, kSCSessionKey); + AVCaptureSession *s = SCSessionFor(out); SCSessionState *st = s ? objc_getAssociatedObject(s, kSCStateKey) : nil; return st.usesFake; } static AVCaptureConnection *sc_connectionWithMediaType(AVCaptureOutput *self, SEL _cmd, AVMediaType t) { - if (SCOutputIsFake(self)) return SCConnectionFor(self); + if (SCOutputIsFake(self)) { + if ([t isEqualToString:AVMediaTypeVideo]) { + if (!SCHasVideo(SCSessionFor(self))) return nil; + SCConnection *c = SCConnectionFor(self); c->_output = self; return c; + } + if ([t isEqualToString:AVMediaTypeAudio] && [self isKindOfClass:AVCaptureMovieFileOutput.class] && SCHasAudio(SCSessionFor(self))) return SCAudioConnectionFor(self); + return nil; + } return ((id (*)(id, SEL, id))o_connectionWithMediaType)(self, _cmd, t); } static NSArray *sc_connections(AVCaptureOutput *self, SEL _cmd) { - if (SCOutputIsFake(self)) return @[SCConnectionFor(self)]; + if (SCOutputIsFake(self)) { + AVCaptureConnection *video = sc_connectionWithMediaType(self, @selector(connectionWithMediaType:), AVMediaTypeVideo); + AVCaptureConnection *audio = sc_connectionWithMediaType(self, @selector(connectionWithMediaType:), AVMediaTypeAudio); + if (!video) return audio ? @[audio] : @[]; + return audio ? @[video, audio] : @[video]; + } return ((id (*)(id, SEL))o_connections)(self, _cmd); } static void sc_setMetaTypes(AVCaptureMetadataOutput *self, SEL _cmd, NSArray *types) { @@ -646,20 +846,41 @@ static void SCTrackLayer(AVCaptureVideoPreviewLayer *l) { static AVCaptureConnection *sc_layerConnection(AVCaptureVideoPreviewLayer *self, SEL _cmd) { AVCaptureSession *s = self.session; SCSessionState *st = s ? objc_getAssociatedObject(s, kSCStateKey) : nil; - if (st.usesFake) return SCConnectionFor(self); + if (st.usesFake || objc_getAssociatedObject(self, kSCPreviewSessionKey)) return SCConnectionFor(self); return ((id (*)(id, SEL))o_layerConnection)(self, _cmd); } +static AVCaptureSession *sc_layerSession(AVCaptureVideoPreviewLayer *self, SEL _cmd) { + AVCaptureSession *session = objc_getAssociatedObject(self, kSCPreviewSessionKey); + return session ?: ((id (*)(id, SEL))o_layerSession)(self, _cmd); +} static void sc_layerSetSession(AVCaptureVideoPreviewLayer *self, SEL _cmd, AVCaptureSession *s) { + // SwiftUI installs its preview while capture authorization is still pending. + // Attaching an empty session through the native setter commits the Simulator + // stub graph and emits a runtime error before the fake camera is configured. + // Keep the preview's normal strong session ownership without that native graph. + if (s && (SCState(s).usesFake || s.inputs.count == 0)) { + objc_setAssociatedObject(self, kSCPreviewSessionKey, s, OBJC_ASSOCIATION_RETAIN); + SCTrackLayer(self); + return; + } + if (objc_getAssociatedObject(self, kSCPreviewSessionKey)) { + objc_setAssociatedObject(self, kSCPreviewSessionKey, nil, OBJC_ASSOCIATION_RETAIN); + self.contents = nil; + if (!s) return; + } ((void (*)(id, SEL, id))o_layerSetSession)(self, _cmd, s); SCTrackLayer(self); } -static id sc_layerInitSession(AVCaptureVideoPreviewLayer *self, SEL _cmd, AVCaptureSession *s) { - id r = ((id (*)(id, SEL, id))o_layerInitSession)(self, _cmd, s); +typedef id (*SCLayerInitializer)(id __attribute__((ns_consumed)), SEL, id) __attribute__((ns_returns_retained)); +static id sc_layerInitSession(AVCaptureVideoPreviewLayer * __attribute__((ns_consumed)) self, SEL _cmd, AVCaptureSession *s) __attribute__((ns_returns_retained)); +static id sc_layerInitSession(AVCaptureVideoPreviewLayer * __attribute__((ns_consumed)) self, SEL _cmd, AVCaptureSession *s) { + id r = ((SCLayerInitializer)o_layerInitSession)(self, _cmd, s); if (r) SCTrackLayer(r); return r; } -static id sc_layerInitNoConn(AVCaptureVideoPreviewLayer *self, SEL _cmd, AVCaptureSession *s) { - id r = ((id (*)(id, SEL, id))o_layerInitNoConn)(self, _cmd, s); +static id sc_layerInitNoConn(AVCaptureVideoPreviewLayer * __attribute__((ns_consumed)) self, SEL _cmd, AVCaptureSession *s) __attribute__((ns_returns_retained)); +static id sc_layerInitNoConn(AVCaptureVideoPreviewLayer * __attribute__((ns_consumed)) self, SEL _cmd, AVCaptureSession *s) { + id r = ((SCLayerInitializer)o_layerInitNoConn)(self, _cmd, s); if (r) SCTrackLayer(r); return r; } @@ -702,6 +923,34 @@ static void sc_capturePhoto(AVCapturePhotoOutput *self, SEL _cmd, AVCapturePhoto #pragma mark - Install +static IMP o_movieStart, o_movieStop, o_movieRecording; +static void sc_movieStart(AVCaptureMovieFileOutput *self, SEL cmd, NSURL *url, id delegate) { + if (!SCOutputIsFake(self)) { ((void (*)(id, SEL, id, id))o_movieStart)(self, cmd, url, delegate); return; } + AVCaptureSession *session = SCSessionFor(self); + BOOL running = SCState(session).running; + BOOL audio = SCHasAudio(session); + SCConnection *connection = SCConnectionFor(self); + CGFloat rotation = connection->_rotation; BOOL mirrored = connection->_mirrored; + NSArray *connections = self.connections; + dispatch_async(gFrameQueue, ^{ + SCMovieRecorder *previous = objc_getAssociatedObject(self, kSCMovieKey); + if (previous && !previous.finished) return; // Native API requires stop/finish before another start. + SCMovieRecorder *recorder = [[SCMovieRecorder alloc] initWithOutput:self URL:url delegate:delegate connections:connections audio:audio rotation:rotation mirrored:mirrored queue:gFrameQueue]; + objc_setAssociatedObject(self, kSCMovieKey, recorder, OBJC_ASSOCIATION_RETAIN); + [recorder start]; + if (!running) [recorder stop]; + }); +} +static void sc_movieStop(AVCaptureMovieFileOutput *self, SEL cmd) { + if (!SCOutputIsFake(self) && !objc_getAssociatedObject(self, kSCMovieKey)) { ((void (*)(id, SEL))o_movieStop)(self, cmd); return; } + dispatch_async(gFrameQueue, ^{ [objc_getAssociatedObject(self, kSCMovieKey) stop]; }); +} +static BOOL sc_movieRecording(AVCaptureMovieFileOutput *self, SEL cmd) { + SCMovieRecorder *recorder = objc_getAssociatedObject(self, kSCMovieKey); + if (SCOutputIsFake(self) || recorder) return recorder.isRecording; + return ((BOOL (*)(id, SEL))o_movieRecording)(self, cmd); +} + __attribute__((constructor)) static void SCInstall(void) { if (getenv("SIMCAM_DISABLE")) return; gLock = [NSLock new]; @@ -727,6 +976,7 @@ static void sc_capturePhoto(AVCapturePhotoOutput *self, SEL _cmd, AVCapturePhoto SCSwizzleInstance(S, @selector(addInput:), (IMP)sc_addInput, &o_addInput); SCSwizzleInstance(S, @selector(removeInput:), (IMP)sc_removeInput, &o_removeInput); SCSwizzleInstance(S, @selector(inputs), (IMP)sc_inputs, &o_inputs); + SCSwizzleInstance(S, @selector(outputs), (IMP)sc_outputs, &o_outputs); SCSwizzleInstance(S, @selector(canAddOutput:), (IMP)sc_canAddOutput, &o_canAddOutput); SCSwizzleInstance(S, @selector(addOutput:), (IMP)sc_addOutput, &o_addOutput); SCSwizzleInstance(S, @selector(removeOutput:), (IMP)sc_removeOutput, &o_removeOutput); @@ -746,8 +996,13 @@ static void sc_capturePhoto(AVCapturePhotoOutput *self, SEL _cmd, AVCapturePhoto SCSwizzleInstance(M, @selector(setRectOfInterest:), (IMP)sc_setRectOfInterest, &o_setRectOfInterest); SCSwizzleInstance(M, @selector(rectOfInterest), (IMP)sc_rectOfInterest, &o_rectOfInterest); SCSwizzleInstance([AVCapturePhotoOutput class], @selector(capturePhotoWithSettings:delegate:), (IMP)sc_capturePhoto, &o_capturePhoto); + Class F = [AVCaptureMovieFileOutput class]; + SCSwizzleInstance(F, @selector(startRecordingToOutputFileURL:recordingDelegate:), (IMP)sc_movieStart, &o_movieStart); + SCSwizzleInstance(F, @selector(stopRecording), (IMP)sc_movieStop, &o_movieStop); + SCSwizzleInstance(F, @selector(isRecording), (IMP)sc_movieRecording, &o_movieRecording); Class L = [AVCaptureVideoPreviewLayer class]; + SCSwizzleInstance(L, @selector(session), (IMP)sc_layerSession, &o_layerSession); SCSwizzleInstance(L, @selector(connection), (IMP)sc_layerConnection, &o_layerConnection); SCSwizzleInstance(L, @selector(setSession:), (IMP)sc_layerSetSession, &o_layerSetSession); SCSwizzleInstance(L, @selector(initWithSession:), (IMP)sc_layerInitSession, &o_layerInitSession); diff --git a/SimulatorInjection/Tests/AnalyzeMovie.m b/SimulatorInjection/Tests/AnalyzeMovie.m new file mode 100644 index 0000000..d9e722e --- /dev/null +++ b/SimulatorInjection/Tests/AnalyzeMovie.m @@ -0,0 +1,85 @@ +// Decode the actual finalized file; track metadata alone is not recording proof. +#import +#import +#import +int main(int argc, char **argv) { @autoreleasepool { + if (argc != 2) return 2; + AVURLAsset *asset = [AVURLAsset URLAssetWithURL:[NSURL fileURLWithPath:@(argv[1])] options:nil]; + NSMutableDictionary *result = [NSMutableDictionary dictionary]; + result[@"duration"] = @(CMTimeGetSeconds(asset.duration)); + BOOL pass = YES; + for (AVMediaType type in @[AVMediaTypeVideo, AVMediaTypeAudio]) { + AVAssetTrack *track = [asset tracksWithMediaType:type].firstObject; + if (!track) { result[type] = @{@"error": @"missing track"}; pass = NO; continue; } + NSError *error = nil; + AVAssetReader *reader = [[AVAssetReader alloc] initWithAsset:asset error:&error]; + NSDictionary *settings = [type isEqual:AVMediaTypeVideo] + ? @{(id)kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_32BGRA)} + : @{AVFormatIDKey: @(kAudioFormatLinearPCM), AVLinearPCMBitDepthKey: @32, + AVLinearPCMIsFloatKey: @YES, AVLinearPCMIsNonInterleaved: @NO}; + AVAssetReaderTrackOutput *out = [[AVAssetReaderTrackOutput alloc] initWithTrack:track outputSettings:settings]; + [reader addOutput:out]; + [reader startReading]; + NSUInteger count = 0, changes = 0, audioValues = 0, crossings = 0; + double previousAudio = 0, sampleRate = 0; + double first = -1, last = -1, end = -1, sum = 0, peak = 0; + BOOL monotonic = YES; + uint64_t previousHash = 0; + CMSampleBufferRef sample; + while ((sample = [out copyNextSampleBuffer])) { + double pts = CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sample)); + if (!count) first = pts; + if (count && pts <= last) monotonic = NO; + last = pts; + double duration = CMTimeGetSeconds(CMSampleBufferGetDuration(sample)); + end = pts + (isfinite(duration) ? duration : 0); + count++; + CVPixelBufferRef pb = CMSampleBufferGetImageBuffer(sample); + if (pb) { + CVPixelBufferLockBaseAddress(pb, kCVPixelBufferLock_ReadOnly); + const uint8_t *bytes = CVPixelBufferGetBaseAddress(pb); + size_t length = CVPixelBufferGetBytesPerRow(pb) * CVPixelBufferGetHeight(pb); + uint64_t hash = 1469598103934665603ULL; + for (size_t i = 0; i < length; i += 97) hash = (hash ^ bytes[i]) * 1099511628211ULL; + if (count > 1 && hash != previousHash) changes++; + previousHash = hash; + CVPixelBufferUnlockBaseAddress(pb, kCVPixelBufferLock_ReadOnly); + } else { + const AudioStreamBasicDescription *asbd = CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sample)); + sampleRate = asbd->mSampleRate; + CMBlockBufferRef block = CMSampleBufferGetDataBuffer(sample); + size_t length = CMBlockBufferGetDataLength(block); + float *values = malloc(length); + if (CMBlockBufferCopyDataBytes(block, 0, length, values) == noErr) { + for (size_t i = 0; i < length / sizeof(float); i++) { + double value = values[i]; sum += value * value; peak = fmax(peak, fabs(value)); + if (audioValues && previousAudio <= 0 && value > 0) crossings++; + previousAudio = value; audioValues++; + } + } + free(values); + } + CFRelease(sample); + } + NSMutableDictionary *metrics = [@{@"samples": @(count), @"firstPTS": @(first), @"lastPTS": @(last), @"endPTS": @(end), @"monotonic": @(monotonic), @"readerStatus": @(reader.status)} mutableCopy]; + if ([type isEqual:AVMediaTypeVideo]) { + metrics[@"changedFrames"] = @(changes); + CGAffineTransform t = track.preferredTransform; + metrics[@"transform"] = @[@(t.a), @(t.b), @(t.c), @(t.d), @(t.tx), @(t.ty)]; + metrics[@"width"] = @(track.naturalSize.width); metrics[@"height"] = @(track.naturalSize.height); + pass &= changes > 0; + } else { + double rms = audioValues ? sqrt(sum / audioValues) : 0; + metrics[@"rms"] = @(rms); metrics[@"peak"] = @(peak); + metrics[@"positiveCrossingFrequency"] = @(audioValues ? crossings * sampleRate / audioValues : 0); + pass &= rms > 0.00001; + } + if (reader.error) metrics[@"error"] = reader.error.description; + result[type] = metrics; + pass &= count > 1 && monotonic && reader.status == AVAssetReaderStatusCompleted; + } + result[@"pass"] = @(pass); + NSData *json = [NSJSONSerialization dataWithJSONObject:result options:NSJSONWritingPrettyPrinted error:nil]; + puts([[NSString alloc] initWithData:json encoding:NSUTF8StringEncoding].UTF8String); + return pass ? 0 : 1; +} } diff --git a/SimulatorInjection/Tests/CheckOrientation.m b/SimulatorInjection/Tests/CheckOrientation.m new file mode 100644 index 0000000..5108df2 --- /dev/null +++ b/SimulatorInjection/Tests/CheckOrientation.m @@ -0,0 +1,40 @@ +#import +#import +#import + +static NSString *corners(CGImageRef image) { + NSBitmapImageRep *bitmap = [[NSBitmapImageRep alloc] initWithCGImage:image]; + NSMutableString *result = [NSMutableString string]; + for (NSUInteger i = 0; i < 4; i++) { + NSColor *color = [[bitmap colorAtX:(NSInteger)(bitmap.pixelsWide * (i % 2 ? .8 : .2)) + y:(NSInteger)(bitmap.pixelsHigh * (i / 2 ? .8 : .2))] colorUsingColorSpace:NSColorSpace.sRGBColorSpace]; + CGFloat r = color.redComponent, g = color.greenComponent, b = color.blueComponent; + NSString *name = r > .6 && g > .6 && b < .3 ? @"Y" : r > .6 && g < .3 && b < .3 ? @"R" : g > .6 && r < .3 && b < .3 ? @"G" : b > .6 && r < .3 && g < .3 ? @"B" : @"?"; + [result appendString:name]; + } + return result; +} +int main(int argc, char **argv) { @autoreleasepool { + if (argc != 6) { fprintf(stderr, "usage: CheckOrientation movie angle mirrored preview.png extracted.png\n"); return 2; } + int angle = atoi(argv[2]); BOOL mirrored = atoi(argv[3]); + // Independently specified visual corners TL,TR,BL,BR. A portrait (90°) + // connection must show the upright host image; other requests remain rotations. + NSDictionary *expected = @{@0: @[@"GYRB", @"YGBR"], @90: @[@"RGBY", @"GRYB"], + @180: @[@"BRYG", @"RBGY"], @270: @[@"YBGR", @"BYRG"]}; + NSString *wanted = expected[@(angle)][mirrored ? 1 : 0]; + AVURLAsset *asset = [AVURLAsset URLAssetWithURL:[NSURL fileURLWithPath:@(argv[1])] options:nil]; + AVAssetImageGenerator *generator = [AVAssetImageGenerator assetImageGeneratorWithAsset:asset]; + generator.appliesPreferredTrackTransform = YES; // Normal player semantics; no corrective transform. + NSError *error = nil; + CGImageRef movie = [generator copyCGImageAtTime:CMTimeMake(1, 2) actualTime:NULL error:&error]; + if (!movie) { NSLog(@"decode failed %@", error); return 1; } + CGImageDestinationRef destination = CGImageDestinationCreateWithURL((__bridge CFURLRef)[NSURL fileURLWithPath:@(argv[5])], CFSTR("public.png"), 1, NULL); + CGImageDestinationAddImage(destination, movie, NULL); CGImageDestinationFinalize(destination); CFRelease(destination); + CGImageSourceRef source = CGImageSourceCreateWithURL((__bridge CFURLRef)[NSURL fileURLWithPath:@(argv[4])], NULL); + CGImageRef preview = source ? CGImageSourceCreateImageAtIndex(source, 0, NULL) : NULL; + NSString *movieCorners = corners(movie), *previewCorners = preview ? corners(preview) : @"missing"; + BOOL pass = [movieCorners isEqual:wanted] && [previewCorners isEqual:wanted]; + printf("%s angle=%d mirrored=%d expected=%s movie=%s preview=%s display=%zux%zu\n", pass ? "PASS" : "FAIL", angle, mirrored, wanted.UTF8String, movieCorners.UTF8String, previewCorners.UTF8String, CGImageGetWidth(movie), CGImageGetHeight(movie)); + CGImageRelease(movie); if (preview) CGImageRelease(preview); if (source) CFRelease(source); + return pass ? 0 : 1; +} } diff --git a/SimulatorInjection/Tests/ExtractMovieFrames.m b/SimulatorInjection/Tests/ExtractMovieFrames.m new file mode 100644 index 0000000..3029423 --- /dev/null +++ b/SimulatorInjection/Tests/ExtractMovieFrames.m @@ -0,0 +1,21 @@ +// Export the original asset as an ordinary player displays it. No corrective rotation. +#import +#import +int main(int argc, char **argv) { @autoreleasepool { + if (argc != 3) return 2; + AVURLAsset *asset = [AVURLAsset URLAssetWithURL:[NSURL fileURLWithPath:@(argv[1])] options:nil]; + AVAssetImageGenerator *generator = [AVAssetImageGenerator assetImageGeneratorWithAsset:asset]; + generator.appliesPreferredTrackTransform = YES; + for (int i = 0; i < 3; i++) { + NSError *error = nil; + CGImageRef image = [generator copyCGImageAtTime:CMTimeMake(3 + 7 * i, 10) actualTime:NULL error:&error]; + if (!image) { NSLog(@"%@", error); return 1; } + NSURL *url = [NSURL fileURLWithPath:[NSString stringWithFormat:@"%s-%d.png", argv[2], i + 1]]; + CGImageDestinationRef destination = CGImageDestinationCreateWithURL((__bridge CFURLRef)url, CFSTR("public.png"), 1, NULL); + CGImageDestinationAddImage(destination, image, NULL); + BOOL saved = CGImageDestinationFinalize(destination); + CFRelease(destination); CGImageRelease(image); + if (!saved) return 1; + } + return 0; +} } diff --git a/SimulatorInjection/Tests/HostAudioFeed.m b/SimulatorInjection/Tests/HostAudioFeed.m new file mode 100644 index 0000000..79bed08 --- /dev/null +++ b/SimulatorInjection/Tests/HostAudioFeed.m @@ -0,0 +1,115 @@ +// Optional hardware-boundary fixture. Selects devices only for this process; +// never changes the host's default input/output. Requires an existing loopback device. +#import +#import +#import +#import + +@interface HostMicFeed : NSObject +@property AVAudioPlayerNode *player; +@property double sum; +@property NSUInteger count; +@property BOOL ready; +@end +@implementation HostMicFeed +- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sample fromConnection:(AVCaptureConnection *)connection { + AVAudioFormat *format = [[AVAudioFormat alloc] initWithStreamDescription:CMAudioFormatDescriptionGetStreamBasicDescription(CMSampleBufferGetFormatDescription(sample))]; + AVAudioPCMBuffer *buffer = [[AVAudioPCMBuffer alloc] initWithPCMFormat:format frameCapacity:(AVAudioFrameCount)CMSampleBufferGetNumSamples(sample)]; + buffer.frameLength = buffer.frameCapacity; + if (CMSampleBufferCopyPCMDataIntoAudioBufferList(sample, 0, buffer.frameLength, buffer.mutableAudioBufferList) != noErr) return; + if (buffer.floatChannelData) { + for (AVAudioFrameCount i = 0; i < buffer.frameLength; i++) { double v = buffer.floatChannelData[0][i * buffer.stride]; self.sum += v * v; self.count++; } + } + [self.player scheduleBuffer:buffer completionHandler:nil]; + if (!self.ready) { self.ready = YES; fprintf(stderr, "READY microphone samples\n"); } +} +@end + +static AudioDeviceID findDevice(NSString *wanted) { + AudioObjectPropertyAddress address = {kAudioHardwarePropertyDevices, kAudioObjectPropertyScopeGlobal, kAudioObjectPropertyElementMain}; + UInt32 size = 0; + AudioObjectGetPropertyDataSize(kAudioObjectSystemObject, &address, 0, NULL, &size); + AudioDeviceID *devices = malloc(size); + AudioObjectGetPropertyData(kAudioObjectSystemObject, &address, 0, NULL, &size, devices); + AudioDeviceID result = kAudioObjectUnknown; + for (UInt32 i = 0; i < size / sizeof(AudioDeviceID); i++) { + AudioObjectPropertyAddress nameAddress = {kAudioObjectPropertyName, kAudioObjectPropertyScopeGlobal, kAudioObjectPropertyElementMain}; + CFStringRef name = NULL; UInt32 nameSize = sizeof(name); + AudioObjectGetPropertyData(devices[i], &nameAddress, 0, NULL, &nameSize, &name); + if ([(__bridge NSString *)name isEqual:wanted]) result = devices[i]; + if (name) CFRelease(name); + } + free(devices); return result; +} + +int main(int argc, char **argv) { @autoreleasepool { + if (argc < 3) { fprintf(stderr, "usage: HostAudioFeed [input-device-name]\n"); return 2; } + NSString *outputName = @(argv[1]); double seconds = atof(argv[2]); + dispatch_source_t parentWatch = nil; + if (getenv("SIMCAM_STOP_ON_STDIN_EOF")) { + parentWatch = dispatch_source_create(DISPATCH_SOURCE_TYPE_READ, STDIN_FILENO, 0, dispatch_get_global_queue(QOS_CLASS_DEFAULT, 0)); + dispatch_source_set_event_handler(parentWatch, ^{ char byte; if (read(STDIN_FILENO, &byte, 1) == 0) exit(0); }); + dispatch_resume(parentWatch); + } + AudioDeviceID output = findDevice(outputName); + if (!output) { fprintf(stderr, "No output device %s\n", argv[1]); return 2; } + AVAudioEngine *engine = [AVAudioEngine new]; + AudioUnit outputUnit = engine.outputNode.audioUnit; + OSStatus status = AudioUnitSetProperty(outputUnit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0, &output, sizeof(output)); + if (status) { fprintf(stderr, "Output selection failed %d\n", (int)status); return 2; } + AVAudioPlayerNode *player = nil; + AVCaptureSession *capture = nil; + HostMicFeed *feed = nil; + if (argc > 3) { + if ([AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeAudio] != AVAuthorizationStatusAuthorized) { + fprintf(stderr, "BLOCKED: host microphone permission is not authorized for HostAudioFeed (no permission bypass attempted)\n"); return 3; + } + AVCaptureDevice *mic = nil; + for (AVCaptureDevice *device in [AVCaptureDevice devicesWithMediaType:AVMediaTypeAudio]) + if ([device.localizedName isEqual:@(argv[3])]) mic = device; + if (!mic) { fprintf(stderr, "No microphone named %s\n", argv[3]); return 2; } + capture = [AVCaptureSession new]; + NSError *error = nil; + AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:mic error:&error]; + if (!input || ![capture canAddInput:input]) { fprintf(stderr, "%s\n", error.description.UTF8String); return 2; } + [capture addInput:input]; + AVCaptureAudioDataOutput *audio = [AVCaptureAudioDataOutput new]; + audio.audioSettings = @{AVFormatIDKey: @(kAudioFormatLinearPCM), AVSampleRateKey: @48000, AVNumberOfChannelsKey: @1, AVLinearPCMBitDepthKey: @32, AVLinearPCMIsFloatKey: @YES, AVLinearPCMIsNonInterleaved: @NO}; + player = [AVAudioPlayerNode new]; [engine attachNode:player]; + [engine connect:player to:engine.mainMixerNode format:[[AVAudioFormat alloc] initStandardFormatWithSampleRate:48000 channels:1]]; + feed = [HostMicFeed new]; feed.player = player; + [audio setSampleBufferDelegate:feed queue:dispatch_queue_create("probe.microphone", DISPATCH_QUEUE_SERIAL)]; + [capture addOutput:audio]; + fprintf(stderr, "LIVE input=%s -> output=%s\n", argv[3], argv[1]); + } else { + player = [AVAudioPlayerNode new]; [engine attachNode:player]; + AVAudioFormat *format = [[AVAudioFormat alloc] initStandardFormatWithSampleRate:48000 channels:2]; + [engine connect:player to:engine.mainMixerNode format:format]; + AVAudioPCMBuffer *buffer = [[AVAudioPCMBuffer alloc] initWithPCMFormat:format frameCapacity:48000]; + buffer.frameLength = 48000; + for (NSUInteger channel = 0; channel < 2; channel++) + for (NSUInteger frame = 0; frame < 48000; frame++) buffer.floatChannelData[channel][frame] = 0.2 * sin(2 * M_PI * 440 * frame / 48000.0); + [player scheduleBuffer:buffer atTime:nil options:AVAudioPlayerNodeBufferLoops completionHandler:nil]; + fprintf(stderr, "GENERATED 440Hz tone -> output=%s\n", argv[1]); + } + NSError *error = nil; + __block double renderedSum = 0; + __block NSUInteger renderedCount = 0; + [engine.mainMixerNode installTapOnBus:0 bufferSize:1024 format:nil block:^(AVAudioPCMBuffer *buffer, AVAudioTime *time) { + for (AVAudioFrameCount i = 0; i < buffer.frameLength; i++) { + double value = buffer.floatChannelData[0][i * buffer.stride]; + renderedSum += value * value; renderedCount++; + } + }]; + if (![engine startAndReturnError:&error]) { fprintf(stderr, "%s\n", error.description.UTF8String); return 1; } + [player play]; + [capture startRunning]; + if (seconds > 0) [NSThread sleepForTimeInterval:seconds]; + else while (YES) pause(); // Managed bridge closes stdin or sends SIGTERM to release hardware. + [engine stop]; + [engine.mainMixerNode removeTapOnBus:0]; + [capture stopRunning]; + if (feed) fprintf(stderr, "LIVE microphone samples=%lu rms=%g\n", (unsigned long)feed.count, feed.count ? sqrt(feed.sum / feed.count) : 0); + fprintf(stderr, "RENDERED to selected output samples=%lu rms=%g\n", (unsigned long)renderedCount, renderedCount ? sqrt(renderedSum / renderedCount) : 0); + return 0; +} } diff --git a/SimulatorInjection/Tests/NativeDriverProbe.swift b/SimulatorInjection/Tests/NativeDriverProbe.swift new file mode 100644 index 0000000..bee367f --- /dev/null +++ b/SimulatorInjection/Tests/NativeDriverProbe.swift @@ -0,0 +1,116 @@ +import UIKit +@preconcurrency import AVFoundation + +// Compiled with the caller's actual Media sources, not a copy or replacement recorder. +@main +@MainActor +final class NativeDriverProbe: UIResponder, UIApplicationDelegate { + var window: UIWindow? + private let driver = NativeVideoCaptureDriver() + private let label = UILabel() + private var player: AVPlayer? + private var playerLayer: AVPlayerLayer? + + func application(_ application: UIApplication, didFinishLaunchingWithOptions options: [UIApplication.LaunchOptionsKey: Any]? = nil) -> Bool { + let window = UIWindow(frame: UIScreen.main.bounds) + let view = UIViewController() + view.view.backgroundColor = .black + label.frame = CGRect(x: 24, y: 70, width: 340, height: 90) + label.textColor = .white + label.numberOfLines = 0 + label.text = "Native Small Hello driver\nPreparing recording" + view.view.addSubview(label) + let preview = AVCaptureVideoPreviewLayer() + preview.frame = CGRect(x: 20, y: 180, width: 350, height: 450) + preview.videoGravity = .resizeAspect + driver.previewSource.connect(to: preview) + view.view.layer.addSublayer(preview) + window.rootViewController = view + window.makeKeyAndVisible() + self.window = window + Task { + try? await Task.sleep(for: .seconds(20)) + finish("FAIL native driver watchdog", success: false) + } + Task { + do { + try await Task.sleep(for: .seconds(1)) + let documents = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] + if ProcessInfo.processInfo.environment["NATIVE_CANCEL"] == "1" { + let url = documents.appendingPathComponent("native-cancel.mov") + try? FileManager.default.removeItem(at: url) + let start = Task { try await driver.beginRecording(to: url) } + try await Task.sleep(for: .milliseconds(30)) + await driver.stopRecording() + do { + let completion = try await start.value + try await completion.value + } catch { print("NATIVE cancellation settled: \(error)") } + let retryURL = documents.appendingPathComponent("native-cancel-retry.mov") + try? FileManager.default.removeItem(at: retryURL) + let retry = try await driver.beginRecording(to: retryURL) + try await Task.sleep(for: .seconds(1)) + await driver.stopRecording() + try await retry.value + let retryAsset = AVURLAsset(url: retryURL) + guard try await retryAsset.loadTracks(withMediaType: .video).count == 1, + try await retryAsset.loadTracks(withMediaType: .audio).count == 1 else { throw MediaError.unusableVideo } + finish("PASS exact driver startup cancellation and subsequent recording", success: true) + return + } + for take in 1...2 { + let url = documents.appendingPathComponent("native-\(take).mov") + try? FileManager.default.removeItem(at: url) + label.text = "Native Small Hello driver\nRecording take \(take)" + print("NATIVE requesting take \(take)") + let completion = try await driver.beginRecording(to: url) + print("NATIVE started take \(take)") + if take == 1 { try "ready".write(to: documents.appendingPathComponent("native-recording-ready"), atomically: true, encoding: .utf8) } + try await Task.sleep(for: .seconds(2)) + await driver.stopRecording() + print("NATIVE stop requested take \(take)") + try await completion.value + let asset = AVURLAsset(url: url) + let video = try await asset.loadTracks(withMediaType: .video) + let audio = try await asset.loadTracks(withMediaType: .audio) + let duration = try await asset.load(.duration).seconds + guard video.count == 1, audio.count == 1, duration > 1.5 else { + print("NATIVE unexpected short capture duration=\(duration)") + throw MediaError.unusableVideo + } + print("NATIVE finalized \(url.path)") + } + label.text = "Native Small Hello driver\nReplaying finalized video + audio" + preview.removeFromSuperlayer() + let playbackAudio = AudioSessionLease() + try playbackAudio.activateForPlayback() + defer { playbackAudio.restore() } + let player = AVPlayer(url: documents.appendingPathComponent("native-2.mov")) + player.isMuted = true // Decode playback without playing a fixture into the user's speakers. + self.player = player + let layer = AVPlayerLayer(player: player) + layer.frame = preview.frame + view.view.layer.addSublayer(layer) + self.playerLayer = layer + player.play() + let playbackDeadline = ContinuousClock.now.advanced(by: .seconds(5)) + while (player.currentTime().seconds <= 0.2 || !layer.isReadyForDisplay) && ContinuousClock.now < playbackDeadline { + try await Task.sleep(for: .milliseconds(100)) + } + print("NATIVE replay time=\(player.currentTime().seconds) readyForDisplay=\(layer.isReadyForDisplay) error=\(String(describing: player.error))") + guard player.currentTime().seconds > 0.2, layer.isReadyForDisplay else { throw MediaError.unusableVideo } + try "ready".write(to: documents.appendingPathComponent("native-replay-ready"), atomically: true, encoding: .utf8) + try await Task.sleep(for: .seconds(2)) + finish("PASS exact native driver two takes and replay", success: true) + } catch { finish("FAIL native driver \(error)", success: false) } + } + return true + } + + private func finish(_ message: String, success: Bool) { + print(message) + let documents = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0] + try? message.write(to: documents.appendingPathComponent("native-result.txt"), atomically: true, encoding: .utf8) + exit(success ? 0 : 1) + } +} diff --git a/SimulatorInjection/Tests/README.md b/SimulatorInjection/Tests/README.md new file mode 100644 index 0000000..547501b --- /dev/null +++ b/SimulatorInjection/Tests/README.md @@ -0,0 +1,144 @@ +# Movie recording verification + +Run only on an isolated Simulator with disposable app data. No system extension, +host camera, or microphone capture is needed for the deterministic suite. The +probe imports only UIKit/AVFoundation and calls the public movie-output API. + +## Deterministic public-API suite + +```sh +export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer +xcrun simctl list runtimes +xcrun simctl create SimulatorCamera-Recording-Probe \ + com.apple.CoreSimulator.SimDeviceType.iPhone-17 \ + com.apple.CoreSimulator.SimRuntime.iOS-26-5 +# Use the returned dedicated ID, not "booted". +DEVICE= +xcrun simctl boot "$DEVICE" +xcrun simctl bootstatus "$DEVICE" -b +bash SimulatorInjection/Tests/run-probe.sh "$DEVICE" +``` + +The suite builds a **test-only** injection library under +`build/RecordingTestInjection/`. Its synthetic 440 Hz PCM source replaces only +the microphone boundary; encoding, timestamps, file finalization, session hooks, +and recording delegates use production code. Test-only switches also suppress +audio/video delivery to exercise startup timeout failures. These switches are +compiled out of normal `SimulatorInjection/build.sh` builds. + +Cases cover attaching a preview before session configuration and releasing its +connection after recording, two takes with different rotation/mirroring, immediate repeated stop, +not-running session, existing/missing destination, session stop, output removal, +session release, camera-input removal, interruption, missing audio/video, and +truthful native microphone permission denial. Permission denial is rejected by +the real microphone-input initializer, before recording begins. + +`AnalyzeMovie.m` uses real `AVAssetReader` decoding. The checks require changing +video pixels, nonzero audio with the expected RMS/frequency, monotonic sample +timestamps, independent transform expectations, and bounded track start/end +differences. Track presence or successful callbacks alone are not sufficient. +Timing checks establish muxed track alignment, **not physical camera/microphone +lip-sync latency**. Artifacts and logs go to ignored `build/evidence/`. + +Run one case by appending its name, for example: + +```sh +bash SimulatorInjection/Tests/run-probe.sh "$DEVICE" immediate +``` + +## Exact caller integration + +After the public suite has built its tools: + +```sh +bash SimulatorInjection/Tests/run-native-driver-probe.sh "$DEVICE" \ + /path/to/small-hello/app/Sources/Media fixture +``` + +This compiles the actual caller's Media sources in place, read-only, with Swift 6 +complete concurrency checking; hashes are retained with the evidence. The app +uses `NativeVideoCaptureDriver` and `AudioSessionLease` unchanged to record twice, +stop/finalize, and replay with `AVPlayer`. A separate launch cancels startup and +then records again. Playback is muted to keep fixtures out of the host speakers. +Screenshots show only this isolated app and must be inspected separately. + +## Real microphone proof + +Use a normal production library build and a live host microphone route for this +proof. Fixtures do not establish real microphone capture. On the assessment +machine the default input is an existing **BlackHole 2ch** virtual device. The +optional `live` mode uses `HostAudioFeed.m` to bridge **MacBook Air Microphone** +into that device for 15 seconds, selecting both devices only inside the helper +process. It does not change global audio defaults or install a driver: + +```sh +bash SimulatorInjection/Tests/run-native-driver-probe.sh "$DEVICE" \ + /path/to/small-hello/app/Sources/Media live +``` + +This machine-specific mode requires those two existing devices and already +authorized host microphone access. If unavailable, configure a suitable route +yourself or report the remaining live-capture gate; do not bypass permissions. +The helper reports live input sample count/RMS; finalized movies must separately +pass nonzero-audio decoding. Quiet surroundings can legitimately fail that proof. +Live video in this harness is the animated injection test pattern, not a claim +of Mac webcam capture. + +## Webcam orientation regression + +The camera transport publishes an already-upright host image. Its portrait +reference is 90°; the effective display rotation is the requested connection +angle minus that reference, followed by display-space mirroring. The ordinary +test pattern's sensor-coordinate transform tests retain their zero reference. + +```sh +bash SimulatorInjection/Tests/run-orientation-probe.sh "$DEVICE" \ + "$PWD/build/evidence/new-orientation-run" +``` + +This publishes a labeled, asymmetric four-color image through the actual frame +store with the camera tag, then records via the public movie-output API. It +independently checks the expected visual corner positions at 0/90/180/270°, +mirrored and unmirrored, in both preview pixels and the decoded movie. Extraction +uses `AVAssetImageGenerator.appliesPreferredTrackTransform`, as a normal player +does; no corrective rotation is applied. The fixture refuses to overwrite an +existing frame store and cleans up only its own inode. + +For a separately authorized short **physical webcam plus microphone** check, +build the production dylib, native-driver probe and analysis tools, then: + +```sh +DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcodebuild \ + -project SimulatorCamera.xcodeproj -scheme simcamctl -configuration Debug \ + -destination 'platform=macOS' -derivedDataPath build/WebcamVerification \ + CODE_SIGNING_ALLOWED=NO build +DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcrun clang \ + -fobjc-arc -fmodules -Wno-deprecated-declarations -framework AVFoundation \ + -framework CoreMedia -framework ImageIO SimulatorInjection/Tests/ExtractMovieFrames.m \ + -o build/ExtractMovieFrames +python3 SimulatorInjection/Tests/run-webcam-probe.py "$DEVICE" \ + "$PWD/build/evidence/new-physical-webcam-run" +``` + +The runner waits for fresh advancing camera-tagged frames before launching the +unchanged caller. It retains original movies, screenshots, decoded frames and +audio/video measurements, waits for its short capture processes to end, and +removes only its own raw frame file. Inspect the screenshots and extracted frames +yourself before claiming upright webcam behavior. The microphone bridge has the +same machine-specific prerequisites described above; playback is muted and this +does not establish perceptual audio quality or physical lip-sync. + +## Existing contract suite and cleanup + +```sh +DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer xcodebuild \ + -project SimulatorCamera.xcodeproj -scheme SimulatorCamera \ + -destination 'platform=macOS' -derivedDataPath build/MacTests \ + -only-testing:SimulatorCameraTests CODE_SIGNING_ALLOWED=NO test +xcrun simctl shutdown "$DEVICE" +``` + +Keep the dedicated device and local evidence until review is complete. No normal +app data, other simulators, system-extension registrations, or security settings +need changing. Normal production builds create both arm64 and x86_64 slices; +the current probe app build targets arm64. diff --git a/SimulatorInjection/Tests/RecordingProbe.m b/SimulatorInjection/Tests/RecordingProbe.m new file mode 100644 index 0000000..c31b27b --- /dev/null +++ b/SimulatorInjection/Tests/RecordingProbe.m @@ -0,0 +1,191 @@ +#import +#import + +// Public-API acceptance probe: no imports or calls into the injection library. +@interface Probe : UIResponder +@property (nonatomic, strong) UIWindow *window; +@property AVCaptureSession *session; +@property AVCaptureMovieFileOutput *output; +@property NSURL *url; +@property NSString *scenario; +@property NSUInteger iteration; +@property NSUInteger starts; +@property NSUInteger finishes; +@property NSUInteger failures; +@property AVCaptureVideoPreviewLayer *preview; +@property id runtimeErrorObserver; +@end + +@implementation Probe +- (NSURL *)documents { return [[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask].firstObject; } +- (void)log:(NSString *)message { NSLog(@"PROBE %@", message); } +- (void)check:(BOOL)condition message:(NSString *)message { + if (!condition) { self.failures++; [self log:[@"FAIL " stringByAppendingString:message]]; } +} +- (void)done { + if (self.runtimeErrorObserver) [NSNotificationCenter.defaultCenter removeObserver:self.runtimeErrorObserver]; + NSDictionary *result = @{@"case": self.scenario, @"starts": @(self.starts), @"finishes": @(self.finishes), @"failures": @(self.failures), @"pass": @(self.failures == 0)}; + NSData *data = [NSJSONSerialization dataWithJSONObject:result options:NSJSONWritingPrettyPrinted error:nil]; + [data writeToURL:[[self documents] URLByAppendingPathComponent:[self.scenario stringByAppendingString:@".json"]] atomically:YES]; + [self log:[[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding]]; + exit(self.failures ? 1 : 0); +} +- (BOOL)is:(NSString *)name { return [self.scenario isEqual:name]; } +- (BOOL)expectsFailure { return [@[@"immediate", @"not-running", @"existing-file", @"missing-directory", @"no-video", @"no-audio", @"denied-audio"] containsObject:self.scenario]; } +- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)options { + self.scenario = getenv("PROBE_CASE") ? @(getenv("PROBE_CASE")) : @"success"; + self.window = [[UIWindow alloc] initWithFrame:UIScreen.mainScreen.bounds]; + UIViewController *vc = [UIViewController new]; vc.view.backgroundColor = UIColor.darkGrayColor; + self.window.rootViewController = vc; [self.window makeKeyAndVisible]; + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 15 * NSEC_PER_SEC), dispatch_get_main_queue(), ^{ [self check:NO message:@"callback watchdog expired"]; [self done]; }); + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, NSEC_PER_SEC), dispatch_get_main_queue(), ^{ [self configure]; }); + return YES; +} +- (void)configure { + @try { + NSError *error = nil; + AVCaptureDevice *camera = [AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInWideAngleCamera mediaType:AVMediaTypeVideo position:AVCaptureDevicePositionFront] ?: [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; + AVCaptureDevice *mic = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeAudio]; + [self log:[NSString stringWithFormat:@"devices camera=%@ microphone=%@", camera, mic]]; + [self check:camera && mic message:@"native camera/microphone discovery"]; + if (!camera || !mic) { [self done]; return; } + AVCaptureDeviceInput *video = [[AVCaptureDeviceInput alloc] initWithDevice:camera error:&error]; + AVCaptureDeviceInput *audio = [[AVCaptureDeviceInput alloc] initWithDevice:mic error:&error]; + if ([self is:@"denied-audio"]) { + // AVFoundation rejects input creation before a recorder exists. Preserve that gate. + [self check:video && !audio && error != nil message:@"native denied microphone input must fail"]; + [self check:[AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeAudio] == AVAuthorizationStatusDenied message:@"permission decision remains truthful"]; + [self done]; return; + } + [self check:video && audio message:@"native input constructors"]; + if (!video || !audio) { [self done]; return; } + AVAudioSession *as = AVAudioSession.sharedInstance; + [self check:[as setCategory:AVAudioSessionCategoryPlayAndRecord mode:AVAudioSessionModeVideoRecording options:AVAudioSessionCategoryOptionDefaultToSpeaker error:&error] message:@"audio category"]; + [self check:[as setActive:YES error:&error] message:@"audio activation"]; + self.session = [AVCaptureSession new]; self.output = [AVCaptureMovieFileOutput new]; + if ([self is:@"preview-lifecycle"]) { + __weak Probe *weakSelf = self; + self.runtimeErrorObserver = [NSNotificationCenter.defaultCenter addObserverForName:AVCaptureSessionRuntimeErrorNotification object:self.session queue:NSOperationQueue.mainQueue usingBlock:^(NSNotification *note) { + [weakSelf check:NO message:@"preview attachment must not emit a capture runtime error"]; + }]; + // Match SwiftUI: bind an empty session, then configure its inputs. + self.preview = [AVCaptureVideoPreviewLayer new]; + self.preview.session = self.session; + [self check:self.preview.session == self.session message:@"preview retains its session"]; + self.preview.frame = CGRectMake(20, 100, 320, 320); + [self.window.rootViewController.view.layer addSublayer:self.preview]; + } + [self.session beginConfiguration]; + self.session.automaticallyConfiguresApplicationAudioSession = NO; + if ([self.session canSetSessionPreset:AVCaptureSessionPresetHigh]) self.session.sessionPreset = AVCaptureSessionPresetHigh; + for (AVCaptureDeviceInput *input in @[video, audio]) { + [self check:[self.session canAddInput:input] message:@"canAddInput"]; + [self.session addInput:input]; + } + [self check:[self.session canAddOutput:self.output] message:@"canAddOutput"]; + [self.session addOutput:self.output]; [self.session commitConfiguration]; + [self check:[self.session.outputs containsObject:self.output] message:@"session owns configured output"]; + AVCaptureConnection *v = [self.output connectionWithMediaType:AVMediaTypeVideo]; + AVCaptureConnection *a = [self.output connectionWithMediaType:AVMediaTypeAudio]; + [self check:v && a && v != a message:@"distinct video/audio connections"]; + [self check:self.output.connections.count == 2 message:@"delegate connection list"]; + [self check:[self.output connectionWithMediaType:AVMediaTypeMetadata] == nil message:@"unsupported media connection absent"]; + if (getenv("PROBE_ORIENTATION_ANGLE")) { + self.preview = [AVCaptureVideoPreviewLayer layerWithSession:self.session]; + self.preview.frame = CGRectMake(20, 100, 320, 320); + self.preview.videoGravity = AVLayerVideoGravityResizeAspect; + [self.window.rootViewController.view.layer addSublayer:self.preview]; + } + [self record]; + } @catch (NSException *e) { [self check:NO message:e.description]; [self done]; } +} +- (void)record { + self.iteration++; + self.url = [[self documents] URLByAppendingPathComponent:[NSString stringWithFormat:@"%@-%lu.mov", self.scenario, (unsigned long)self.iteration]]; + [[NSFileManager defaultManager] removeItemAtURL:self.url error:nil]; + if ([self is:@"missing-directory"]) self.url = [[[self documents] URLByAppendingPathComponent:NSUUID.UUID.UUIDString] URLByAppendingPathComponent:@"absent.mov"]; + if ([self is:@"existing-file"]) [@"do not overwrite" writeToURL:self.url atomically:YES encoding:NSUTF8StringEncoding error:nil]; + AVCaptureConnection *v = [self.output connectionWithMediaType:AVMediaTypeVideo]; + if (@available(iOS 17, *)) { + CGFloat angle = self.iteration == 1 ? 90 : 0; + if (getenv("PROBE_ORIENTATION_ANGLE")) angle = atof(getenv("PROBE_ORIENTATION_ANGLE")); + [self check:[v isVideoRotationAngleSupported:angle] message:@"rotation supported"]; + v.videoRotationAngle = angle; + self.preview.connection.videoRotationAngle = angle; + } + v.automaticallyAdjustsVideoMirroring = NO; + v.videoMirrored = self.iteration == 1; + if (getenv("PROBE_ORIENTATION_ANGLE")) { + v.videoMirrored = getenv("PROBE_ORIENTATION_MIRROR") && atoi(getenv("PROBE_ORIENTATION_MIRROR")); + self.preview.connection.automaticallyAdjustsVideoMirroring = NO; + self.preview.connection.videoMirrored = v.videoMirrored; + } + if (![self is:@"not-running"]) [self.session startRunning]; + [self check:self.session.isRunning == ![self is:@"not-running"] message:@"session running state"]; + [self.output startRecordingToOutputFileURL:self.url recordingDelegate:self]; + if ([self is:@"immediate"]) { [self.output stopRecording]; [self.output stopRecording]; } +} +- (void)captureOutput:(AVCaptureFileOutput *)output didStartRecordingToOutputFileAtURL:(NSURL *)url fromConnections:(NSArray *)connections { + self.starts++; + [self log:[NSString stringWithFormat:@"didStart %@", url.lastPathComponent]]; + [self check:!self.expectsFailure message:@"failure case must not report didStart"]; + [self check:self.starts == self.iteration && self.finishes == self.iteration - 1 message:@"exactly one ordered start"]; + [self check:output == self.output && [url isEqual:self.url] && connections.count == 2 message:@"start delegate arguments"]; + [self check:self.output.isRecording message:@"isRecording at didStart"]; + if (self.preview) dispatch_after(dispatch_time(DISPATCH_TIME_NOW, NSEC_PER_SEC), dispatch_get_main_queue(), ^{ + CGImageRef image = (__bridge CGImageRef)self.preview.contents; + [self check:image != NULL message:@"preview has a rendered frame"]; + if (image) [UIImagePNGRepresentation([UIImage imageWithCGImage:image]) writeToURL:[[self documents] URLByAppendingPathComponent:[self.scenario stringByAppendingString:@"-preview.png"]] atomically:YES]; + }); + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 2 * NSEC_PER_SEC), dispatch_get_main_queue(), ^{ + if ([self is:@"session-stop"]) [self.session stopRunning]; + else if ([self is:@"remove-output"]) [self.session removeOutput:self.output]; + else if ([self is:@"drop-session"]) self.session = nil; + else if ([self is:@"remove-input"]) { + for (AVCaptureDeviceInput *input in self.session.inputs) + if ([input.device hasMediaType:AVMediaTypeVideo]) [self.session removeInput:input]; + } + else if ([self is:@"interruption"]) [NSNotificationCenter.defaultCenter postNotificationName:AVAudioSessionInterruptionNotification object:AVAudioSession.sharedInstance userInfo:@{AVAudioSessionInterruptionTypeKey: @(AVAudioSessionInterruptionTypeBegan)}]; + else { [self.output stopRecording]; [self.output stopRecording]; } + }); +} +- (void)captureOutput:(AVCaptureFileOutput *)output didFinishRecordingToOutputFileAtURL:(NSURL *)url fromConnections:(NSArray *)connections error:(NSError *)error { + self.finishes++; + [self log:[NSString stringWithFormat:@"didFinish %@ error=%@", url.lastPathComponent, error]]; + [self check:self.finishes == self.iteration message:@"exactly one finish per start request"]; + [self check:output == self.output && [url isEqual:self.url] && connections.count == 2 message:@"finish delegate arguments"]; + [self check:!self.output.isRecording message:@"isRecording false at didFinish"]; + [self check:(error != nil) == self.expectsFailure message:@"expected terminal error semantics"]; + [self.session stopRunning]; + if (self.expectsFailure) { + [self check:self.starts == 0 message:@"no spurious start on startup failure"]; + [self check:![error.userInfo[AVErrorRecordingSuccessfullyFinishedKey] boolValue] message:@"failure must not claim saved success"]; + if ([self is:@"existing-file"]) [self check:[[NSString stringWithContentsOfURL:url encoding:NSUTF8StringEncoding error:nil] isEqual:@"do not overwrite"] message:@"existing destination preserved"]; + } else { + AVURLAsset *asset = [AVURLAsset URLAssetWithURL:url options:nil]; + [self check:[asset tracksWithMediaType:AVMediaTypeVideo].count == 1 && [asset tracksWithMediaType:AVMediaTypeAudio].count == 1 message:@"finalized audiovisual file readable inside didFinish"]; + [self check:CMTimeGetSeconds(asset.duration) > 1 message:@"nontrivial movie duration"]; + } + if ([self is:@"success"] && self.iteration == 1) { [self record]; return; } + if ([self is:@"preview-lifecycle"]) { + dispatch_async(dispatch_get_main_queue(), ^{ + __weak AVCaptureConnection *releasedConnection; + @autoreleasepool { + releasedConnection = self.preview.connection; + [self check:releasedConnection != nil message:@"preview exposes a connection"]; + self.preview.session = nil; + [self.preview removeFromSuperlayer]; + self.preview = nil; + [CATransaction flush]; + } + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, NSEC_PER_SEC / 2), dispatch_get_main_queue(), ^{ + [self check:releasedConnection == nil message:@"dismissed preview releases its connection without crashing or leaking"]; + [self done]; + }); + }); + return; + } + dispatch_after(dispatch_time(DISPATCH_TIME_NOW, NSEC_PER_SEC / 2), dispatch_get_main_queue(), ^{ [self done]; }); +} +@end +int main(int argc, char **argv) { @autoreleasepool { return UIApplicationMain(argc, argv, nil, NSStringFromClass(Probe.class)); } } diff --git a/SimulatorInjection/Tests/build-native-driver-probe.sh b/SimulatorInjection/Tests/build-native-driver-probe.sh new file mode 100644 index 0000000..41a9167 --- /dev/null +++ b/SimulatorInjection/Tests/build-native-driver-probe.sh @@ -0,0 +1,18 @@ +#!/bin/bash +set -euo pipefail +export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +SOURCE="${1:?usage: build-native-driver-probe.sh /path/to/small-hello/app/Sources/Media}" +OUT="$ROOT/build/NativeDriverProbe.app" +mkdir -p "$OUT" "$ROOT/build/evidence" +SDK="$(xcrun --sdk iphonesimulator --show-sdk-path)" +SDKROOT="$SDK" xcrun --sdk iphonesimulator swiftc -swift-version 6 -strict-concurrency=complete -parse-as-library \ + -sdk "$SDK" -target arm64-apple-ios18.0-simulator \ + "$SOURCE"/Adapters/*.swift "$SOURCE"/Service/*.swift "$SOURCE"/Types/*.swift "$SOURCE"/UI/*.swift \ + "$ROOT/SimulatorInjection/Tests/NativeDriverProbe.swift" -o "$OUT/NativeDriverProbe" +shasum -a 256 "$SOURCE"/Adapters/*.swift "$SOURCE"/Service/*.swift "$SOURCE"/Types/*.swift "$SOURCE"/UI/*.swift > "$ROOT/build/evidence/native-source-hashes.txt" +cp "$ROOT/build/RecordingProbe.app/Info.plist" "$OUT/Info.plist" +/usr/libexec/PlistBuddy -c 'Set :CFBundleIdentifier com.dautov.NativeDriverProbe' "$OUT/Info.plist" +/usr/libexec/PlistBuddy -c 'Set :CFBundleExecutable NativeDriverProbe' "$OUT/Info.plist" +codesign -s - --force "$OUT" +echo "$OUT" diff --git a/SimulatorInjection/Tests/build-probe.sh b/SimulatorInjection/Tests/build-probe.sh new file mode 100644 index 0000000..2ffccce --- /dev/null +++ b/SimulatorInjection/Tests/build-probe.sh @@ -0,0 +1,27 @@ +#!/bin/bash +set -euo pipefail +export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +OUT="$ROOT/build/RecordingProbe.app" +mkdir -p "$OUT" +SDK="$(xcrun --sdk iphonesimulator --show-sdk-path)" +xcrun clang -fobjc-arc -fmodules -Wno-deprecated-declarations -Wno-incompatible-pointer-types \ + -isysroot "$SDK" -target arm64-apple-ios16.0-simulator \ + -framework UIKit -framework AVFoundation -framework CoreMedia \ + "$ROOT/SimulatorInjection/Tests/RecordingProbe.m" -o "$OUT/RecordingProbe" +cat > "$OUT/Info.plist" <<'PLIST' + + +CFBundleIdentifiercom.dautov.RecordingProbe +CFBundleExecutableRecordingProbe +CFBundleNameRecordingProbe +CFBundleVersion1 +CFBundlePackageTypeAPPL +MinimumOSVersion16.0 +NSCameraUsageDescriptionValidate simulator movie recording. +NSMicrophoneUsageDescriptionValidate microphone sound in simulator movies. +UILaunchScreen + +PLIST +codesign -s - --force "$OUT" +echo "$OUT" diff --git a/SimulatorInjection/Tests/check-fixture-asset.py b/SimulatorInjection/Tests/check-fixture-asset.py new file mode 100644 index 0000000..5d88aef --- /dev/null +++ b/SimulatorInjection/Tests/check-fixture-asset.py @@ -0,0 +1,22 @@ +"""Independent expectations for the public probe's 440 Hz / 90° mirrored fixture.""" +import json +import sys + +path = sys.argv[1] +with open(path) as source: + asset = json.load(source) +video, audio = asset["vide"], asset["soun"] +assert asset["pass"], asset +assert 1.5 < asset["duration"] < 5, asset +assert video["width"] == 1280 and video["height"] == 720, video +assert video["samples"] >= 15 and video["changedFrames"] >= 14, video +assert 0.12 < audio["rms"] < 0.16, audio +assert 435 < audio["positiveCrossingFrequency"] < 445, audio +assert abs(video["firstPTS"] - audio["firstPTS"]) < 0.15, asset +assert abs(video["endPTS"] - audio["endPTS"]) < 0.25, asset +# Mirroring is about the displayed frame's vertical axis, after rotation. +# A 90° clockwise rotation then horizontal mirror maps source (0,0)->(0,0), +# (1280,0)->(0,1280), and (0,720)->(720,0). +expected = [1, 0, 0, 1, 0, 0] if "success-2.mov" in path else [0, 1, 1, 0, 0, 0] +assert all(abs(a - b) < 0.001 for a, b in zip(video["transform"], expected)), video +print("PASS decoded fixture, transform, and track timing:", path) diff --git a/SimulatorInjection/Tests/publish-orientation-fixture.py b/SimulatorInjection/Tests/publish-orientation-fixture.py new file mode 100644 index 0000000..c3432b0 --- /dev/null +++ b/SimulatorInjection/Tests/publish-orientation-fixture.py @@ -0,0 +1,68 @@ +"""Upright asymmetric webcam-transport fixture; no camera/microphone capture. + +Pixels are authored in display/top-left coordinates, independently of encoder +transforms. Existing transport sourceKind=1 identifies Mac camera frames. +""" +import argparse +import mmap +import os +import signal +import struct +import time + +parser = argparse.ArgumentParser() +parser.add_argument("--seconds", type=float, default=60) +args = parser.parse_args() +def stop(signum, frame): + raise SystemExit(0) +signal.signal(signal.SIGTERM, stop) +path = "/private/tmp/SimulatorCamera/frame.bin" +os.makedirs(os.path.dirname(path), exist_ok=True) +fd = os.open(path, os.O_RDWR | os.O_CREAT | os.O_EXCL, 0o600) +inode = os.fstat(fd).st_ino +os.ftruncate(fd, 64 + 1920 * 1080 * 4) +store = mmap.mmap(fd, 0) +os.close(fd) +w, h = 640, 360 +colors = [bytes((0, 0, 230, 255)), bytes((0, 210, 0, 255)), + bytes((230, 0, 0, 255)), bytes((0, 220, 220, 255))] +pixels = bytearray().join((colors[0] * (w // 2) + colors[1] * (w // 2)) for _ in range(h // 2)) +pixels += bytearray().join((colors[2] * (w // 2) + colors[3] * (w // 2)) for _ in range(h // 2)) +glyphs = { + "T": [31, 4, 4, 4, 4, 4, 4], "O": [14, 17, 17, 17, 17, 17, 14], + "P": [30, 17, 17, 30, 16, 16, 16], "L": [16, 16, 16, 16, 16, 16, 31], + "R": [30, 17, 17, 30, 20, 18, 17], "B": [30, 17, 17, 30, 17, 17, 30], +} +def label(text, x, y): + for letter in text: + for row, bits in enumerate(glyphs[letter]): + for col in range(5): + if bits & (1 << (4 - col)): + for dy in range(5): + start = ((y + row * 5 + dy) * w + x + col * 5) * 4 + pixels[start:start + 20] = bytes((255, 255, 255, 255)) * 5 + x += 30 +label("TOP", 275, 10) +label("TL", 15, 15); label("TR", 565, 15) +label("BL", 15, 310); label("BR", 565, 310) +start = time.monotonic() +frame = 0 +try: + while time.monotonic() - start < args.seconds: + frame += 1 + data = bytearray(pixels) + # Moving mark at the center, clear of the independent corner checks. + x = 280 + frame % 80 + for y in range(140, 220): + offset = (y * w + x) * 4 + data[offset:offset + 16] = bytes((255, 255, 255, 255)) * 4 + struct.pack_into(" build/evidence/live-microphone-feed.log 2>&1 & + FEED=$! +fi +env "${EXTRA[@]}" SIMCTL_CHILD_SIMCAM_RECORDING_LOG=1 SIMCTL_CHILD_DYLD_INSERT_LIBRARIES="$LIB" \ + xcrun simctl launch --console-pty "$DEVICE" com.dautov.NativeDriverProbe > "build/evidence/native-$MODE.log" 2>&1 & +LAUNCH=$! +# Wait for app-observed states, not a fixed rendering delay. Capture only this app. +wait_for_state() { + python3 - "$1" <<'PY' +import pathlib, sys, time +path = pathlib.Path(sys.argv[1]) +deadline = time.monotonic() + 20 +while not path.exists(): + if time.monotonic() > deadline: + raise SystemExit("Timed out waiting for " + str(path)) + time.sleep(0.1) +PY +} +wait_for_state "$DATA/Documents/native-recording-ready" +xcrun simctl io "$DEVICE" screenshot "build/evidence/native-$MODE-recording.png" +wait_for_state "$DATA/Documents/native-replay-ready" +xcrun simctl io "$DEVICE" screenshot "build/evidence/native-$MODE-replay.png" +wait "$LAUNCH" +if [[ "$MODE" == live ]]; then wait "$FEED"; fi +cp "$DATA/Documents/native-result.txt" "build/evidence/native-$MODE-result.txt" +grep -q '^PASS' "build/evidence/native-$MODE-result.txt" +for TAKE in 1 2; do + cp "$DATA/Documents/native-$TAKE.mov" "build/evidence/native-$MODE-$TAKE.mov" + build/AnalyzeMovie "build/evidence/native-$MODE-$TAKE.mov" > "build/evidence/native-$MODE-$TAKE.json" +done +rm -f "$DATA/Documents/native-result.txt" +env "${EXTRA[@]}" SIMCTL_CHILD_NATIVE_CANCEL=1 SIMCTL_CHILD_DYLD_INSERT_LIBRARIES="$LIB" \ + xcrun simctl launch --console-pty "$DEVICE" com.dautov.NativeDriverProbe > "build/evidence/native-$MODE-cancel.log" 2>&1 +cp "$DATA/Documents/native-result.txt" "build/evidence/native-$MODE-cancel-result.txt" +grep -q '^PASS' "build/evidence/native-$MODE-cancel-result.txt" +echo "PASS exact native driver ($MODE): two recordings, playback, startup cancellation" diff --git a/SimulatorInjection/Tests/run-orientation-probe.sh b/SimulatorInjection/Tests/run-orientation-probe.sh new file mode 100644 index 0000000..ff5fa8a --- /dev/null +++ b/SimulatorInjection/Tests/run-orientation-probe.sh @@ -0,0 +1,39 @@ +#!/bin/bash +set -euo pipefail +export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$ROOT" +DEVICE="${1:?dedicated simulator ID required}" +EVIDENCE="${2:?new evidence directory required}" +mkdir -p "$EVIDENCE" +bash SimulatorInjection/Tests/build-probe.sh +SIMCAM_RECORDING_TESTS=1 zsh SimulatorInjection/build.sh "$ROOT/build/RecordingTestInjection" +xcrun clang -fobjc-arc -fmodules -Wno-deprecated-declarations -framework AppKit -framework AVFoundation -framework CoreMedia -framework ImageIO SimulatorInjection/Tests/CheckOrientation.m -o build/CheckOrientation +xcrun simctl install "$DEVICE" build/RecordingProbe.app +xcrun simctl privacy "$DEVICE" grant camera com.dautov.RecordingProbe +xcrun simctl privacy "$DEVICE" grant microphone com.dautov.RecordingProbe +DATA="$(xcrun simctl get_app_container "$DEVICE" com.dautov.RecordingProbe data)" +python3 SimulatorInjection/Tests/publish-orientation-fixture.py --seconds 60 > "$EVIDENCE/fixture.log" 2>&1 & +FEED=$! +# Gracefully interrupt our own fixture only, allowing its inode-checked cleanup. +trap 'kill -TERM "$FEED" 2>/dev/null || true; wait "$FEED" 2>/dev/null || true' EXIT +python3 - <<'PY' +import os,time +deadline=time.monotonic()+5 +while not os.path.exists('/private/tmp/SimulatorCamera/frame.bin'): + if time.monotonic()>deadline: raise SystemExit('fixture did not start') + time.sleep(.05) +PY +for ANGLE in ${ORIENTATION_ANGLES:-90 0 180 270}; do + for MIRROR in ${ORIENTATION_MIRRORS:-0 1}; do + CASE="orientation-$ANGLE-$MIRROR" + rm -f "$DATA/Documents/$CASE.json" "$DATA/Documents/$CASE-preview.png" + env SIMCTL_CHILD_SC_TEST_TONE=1 SIMCTL_CHILD_PROBE_CASE="$CASE" \ + SIMCTL_CHILD_PROBE_ORIENTATION_ANGLE="$ANGLE" SIMCTL_CHILD_PROBE_ORIENTATION_MIRROR="$MIRROR" \ + SIMCTL_CHILD_DYLD_INSERT_LIBRARIES="$ROOT/build/RecordingTestInjection/libSimCamInject.dylib" \ + xcrun simctl launch --console-pty "$DEVICE" com.dautov.RecordingProbe > "$EVIDENCE/$CASE.log" 2>&1 + cp "$DATA/Documents/$CASE.json" "$DATA/Documents/$CASE-preview.png" "$DATA/Documents/$CASE-1.mov" "$EVIDENCE/" + python3 -c 'import json,sys; assert json.load(open(sys.argv[1]))["pass"]' "$EVIDENCE/$CASE.json" + build/CheckOrientation "$EVIDENCE/$CASE-1.mov" "$ANGLE" "$MIRROR" "$EVIDENCE/$CASE-preview.png" "$EVIDENCE/$CASE-frame.png" | tee "$EVIDENCE/$CASE-check.log" + done +done diff --git a/SimulatorInjection/Tests/run-probe.sh b/SimulatorInjection/Tests/run-probe.sh new file mode 100644 index 0000000..ef89a61 --- /dev/null +++ b/SimulatorInjection/Tests/run-probe.sh @@ -0,0 +1,43 @@ +#!/bin/bash +# Reuses only the explicitly supplied dedicated device. Does not boot/erase user devices. +set -euo pipefail +export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" +ROOT="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$ROOT" +DEVICE="${1:?usage: run-probe.sh DEDICATED_SIMULATOR_UDID [case ...]}" +shift +CASES=("$@") +if ((${#CASES[@]} == 0)); then CASES=(success preview-lifecycle immediate not-running existing-file missing-directory session-stop remove-output drop-session remove-input interruption no-video no-audio denied-audio); fi +mkdir -p build/evidence +bash SimulatorInjection/Tests/build-probe.sh +SIMCAM_RECORDING_TESTS=1 zsh SimulatorInjection/build.sh "$ROOT/build/RecordingTestInjection" +xcrun clang -fobjc-arc -fmodules -Wno-deprecated-declarations -Wno-incompatible-pointer-types \ + -framework AVFoundation -framework CoreVideo -framework CoreMedia \ + SimulatorInjection/Tests/AnalyzeMovie.m -o build/AnalyzeMovie +xcrun simctl install "$DEVICE" build/RecordingProbe.app +xcrun simctl privacy "$DEVICE" grant camera com.dautov.RecordingProbe +xcrun simctl privacy "$DEVICE" grant microphone com.dautov.RecordingProbe +DATA="$(xcrun simctl get_app_container "$DEVICE" com.dautov.RecordingProbe data)" +for CASE in "${CASES[@]}"; do + EXTRA=(SIMCTL_CHILD_SC_TEST_TONE=1) + if [[ "$CASE" == no-video ]]; then EXTRA+=(SIMCTL_CHILD_SC_TEST_NO_VIDEO=1); fi + if [[ "$CASE" == no-audio ]]; then EXTRA+=(SIMCTL_CHILD_SC_TEST_NO_AUDIO=1); fi + if [[ "$CASE" == denied-audio ]]; then xcrun simctl privacy "$DEVICE" revoke microphone com.dautov.RecordingProbe; fi + rm -f "$DATA/Documents/$CASE.json" + env "${EXTRA[@]}" SIMCTL_CHILD_PROBE_CASE="$CASE" \ + SIMCTL_CHILD_DYLD_INSERT_LIBRARIES="$ROOT/build/RecordingTestInjection/libSimCamInject.dylib" \ + xcrun simctl launch --console-pty "$DEVICE" com.dautov.RecordingProbe > "build/evidence/$CASE.log" 2>&1 + if [[ "$CASE" == denied-audio ]]; then xcrun simctl privacy "$DEVICE" grant microphone com.dautov.RecordingProbe; fi + cp "$DATA/Documents/$CASE.json" "build/evidence/$CASE.json" + python3 -c 'import json,sys; r=json.load(open(sys.argv[1])); print(r); assert r["pass"]' "build/evidence/$CASE.json" + case "$CASE" in + success|preview-lifecycle|session-stop|remove-output|drop-session|remove-input|interruption) + for FILE in "$DATA/Documents/$CASE-"*.mov; do + NAME="$(basename "$FILE")" + cp "$FILE" "build/evidence/$NAME" + build/AnalyzeMovie "build/evidence/$NAME" > "build/evidence/$NAME.json" + python3 SimulatorInjection/Tests/check-fixture-asset.py "build/evidence/$NAME.json" + done;; + esac +done +echo 'PASS public movie-output probes and decoded audiovisual assets' diff --git a/SimulatorInjection/Tests/run-webcam-probe.py b/SimulatorInjection/Tests/run-webcam-probe.py new file mode 100644 index 0000000..c61540c --- /dev/null +++ b/SimulatorInjection/Tests/run-webcam-probe.py @@ -0,0 +1,122 @@ +"""SHORT authorized real webcam/microphone validation on an explicit isolated device. + +Build the production injection, original-caller probe, simcamctl and analysis +tools first. Never uses the test-tone dylib. Does not change audio defaults. +""" +import argparse +import json +import os +from pathlib import Path +import shutil +import struct +import subprocess +import time + +parser = argparse.ArgumentParser() +parser.add_argument("device") +parser.add_argument("evidence", type=Path) +args = parser.parse_args() +root = Path(__file__).resolve().parents[2] +evidence = args.evidence.resolve() +evidence.mkdir(parents=True, exist_ok=False) +env = dict(os.environ, DEVELOPER_DIR="/Applications/Xcode.app/Contents/Developer") +# Do not accidentally inherit synthetic sources into a physical-capture proof. +for key in list(env): + if key.startswith("SIMCTL_CHILD_SC_TEST_"): + del env[key] +def sim(*words, **kwargs): + return subprocess.run(["xcrun", "simctl", *words], env=env, check=True, **kwargs) +sim("install", args.device, str(root / "build/NativeDriverProbe.app")) +for permission in ("camera", "microphone"): + sim("privacy", args.device, "grant", permission, "com.dautov.NativeDriverProbe") +data = Path(sim("get_app_container", args.device, "com.dautov.NativeDriverProbe", "data", capture_output=True, text=True).stdout.strip()) / "Documents" +for name in ("native-result.txt", "native-recording-ready", "native-replay-ready"): + (data / name).unlink(missing_ok=True) +raw = Path("/private/tmp/SimulatorCamera/frame.bin") +if raw.exists(): + raise SystemExit("Existing shared frame file: refusing to overwrite another source") +processes = [] +logs = [] +raw_inode = None +launched = False +def start(command, name, environment=env): + log = (evidence / name).open("w") + logs.append(log) + process = subprocess.Popen(command, cwd=root, env=environment, stdout=log, stderr=subprocess.STDOUT) + processes.append(process) + return process +def wait_marker(name): + deadline = time.monotonic() + 25 + while not (data / name).exists(): + result = data / "native-result.txt" + if result.exists() and result.read_text().startswith("FAIL"): + raise RuntimeError(result.read_text()) + if time.monotonic() > deadline: + raise TimeoutError(name) + time.sleep(.1) +try: + camera = start([str(root / "build/WebcamVerification/Build/Products/Debug/simcamctl"), "feed", "--camera", "--seconds", "15"], "webcam-feed.log") + microphone = start([str(root / "build/HostAudioFeed"), "BlackHole 2ch", "15", "MacBook Air Microphone"], "microphone-feed.log") + deadline = time.monotonic() + 10 + previous_seq = None + while True: + if raw.exists(): + raw_inode = raw.stat().st_ino + with raw.open("rb") as source: + header = source.read(64) + if len(header) == 64: + magic, version, width, height, stride, kind, seq, stamp = struct.unpack_from(" 0 and not seq % 2 and time.time() - stamp < .75: + if previous_seq is not None and seq > previous_seq: + (evidence / "source-header.json").write_text(json.dumps(dict(width=width, height=height, sourceKind=kind, seq=seq, hostTime=stamp), indent=2)) + break + previous_seq = seq + if camera.poll() is not None or time.monotonic() > deadline: + raise RuntimeError("No fresh advancing camera-tagged frame header; see webcam-feed.log") + time.sleep(.1) + launch_env = dict(env, SIMCTL_CHILD_DYLD_INSERT_LIBRARIES=str(root / "build/SimulatorInjection/libSimCamInject.dylib"), SIMCTL_CHILD_SIMCAM_RECORDING_LOG="1") + app = start(["xcrun", "simctl", "launch", "--console-pty", args.device, "com.dautov.NativeDriverProbe"], "native-driver.log", launch_env) + launched = True + wait_marker("native-recording-ready") + sim("io", args.device, "screenshot", str(evidence / "recording.png")) + wait_marker("native-replay-ready") + sim("io", args.device, "screenshot", str(evidence / "replay.png")) + app.wait(timeout=15) + result = (data / "native-result.txt").read_text() + (evidence / "result.txt").write_text(result) + if not result.startswith("PASS"): + raise RuntimeError(result) + movies = [] + for take in (1, 2): + movie = evidence / f"webcam-microphone-{take}.mov" + shutil.copy2(data / f"native-{take}.mov", movie) + analysis = subprocess.run([str(root / "build/AnalyzeMovie"), str(movie)], env=env, check=True, capture_output=True, text=True) + measurement = json.loads(analysis.stdout) + movie.with_suffix(".json").write_text(analysis.stdout) + assert measurement["duration"] > 1.5 + # The original caller requests portrait 90°; upright host pixels must + # display upright at their original aspect, not merely pass track checks. + assert all(abs(a - b) < .001 for a, b in zip(measurement["vide"]["transform"], [1, 0, 0, 1, 0, 0])), measurement + subprocess.run([str(root / "build/ExtractMovieFrames"), str(movie), str(evidence / f"take-{take}-frame")], env=env, check=True) + movies.append(measurement) + (evidence / "summary.json").write_text(json.dumps(dict(result="pass", source="physical webcam + microphone", movies=movies, + visualInspection="Required separately: recording.png, replay.png and original-asset extracted frames", + physicalSync="Not measured; sample timing is not physical lip-sync proof"), indent=2)) +finally: + for process in processes: + if process.poll() is None: + try: + process.wait(timeout=20) + except subprocess.TimeoutExpired: + process.terminate() # Only this script's own process. + try: + process.wait(timeout=3) + except subprocess.TimeoutExpired: + process.kill() + process.wait() + if launched: + subprocess.run(["xcrun", "simctl", "terminate", args.device, "com.dautov.NativeDriverProbe"], env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + for log in logs: + log.close() + if raw_inode is not None and raw.exists() and raw.stat().st_ino == raw_inode: + raw.unlink() diff --git a/SimulatorInjection/build.sh b/SimulatorInjection/build.sh index 7846f9a..2be0771 100755 --- a/SimulatorInjection/build.sh +++ b/SimulatorInjection/build.sh @@ -8,11 +8,14 @@ HERE=${0:A:h} OUT=${1:-$HERE/../build/SimulatorInjection} mkdir -p "$OUT" SDK=$(xcrun --sdk iphonesimulator --show-sdk-path) +TEST_FLAGS=() +if [[ "${SIMCAM_RECORDING_TESTS:-0}" == 1 ]]; then TEST_FLAGS=(-DSIMCAM_RECORDING_TESTS=1); fi xcrun clang -dynamiclib -fobjc-arc -fmodules -O2 -Wall -Wno-unused-variable \ + "${TEST_FLAGS[@]}" \ -isysroot "$SDK" -arch arm64 -arch x86_64 -mios-simulator-version-min=16.0 \ -framework Foundation -framework AVFoundation -framework CoreImage -framework CoreMedia \ -framework CoreVideo -framework UIKit -framework QuartzCore \ -install_name @rpath/libSimCamInject.dylib \ - -o "$OUT/libSimCamInject.dylib" "$HERE/SimCamInject.m" + -o "$OUT/libSimCamInject.dylib" "$HERE/SimCamInject.m" "$HERE/SCMovieRecorder.m" codesign -s - --force "$OUT/libSimCamInject.dylib" >/dev/null echo "$OUT/libSimCamInject.dylib" diff --git a/simcamctl/FeedCommand.swift b/simcamctl/FeedCommand.swift index f570e78..c126362 100644 --- a/simcamctl/FeedCommand.swift +++ b/simcamctl/FeedCommand.swift @@ -26,8 +26,18 @@ final class FeedCommand: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate, private var tick = 0 private var session: AVCaptureSession? private var written = 0 + private var parentWatch: DispatchSourceRead? func run(_ source: Source, seconds: Double?) -> Never { + if ProcessInfo.processInfo.environment["SIMCAM_STOP_ON_STDIN_EOF"] == "1" { + let watch = DispatchSource.makeReadSource(fileDescriptor: STDIN_FILENO, queue: .global()) + watch.setEventHandler { [self] in + var byte: UInt8 = 0 + if read(STDIN_FILENO, &byte, 1) == 0 { finish() } + } + parentWatch = watch + watch.resume() + } switch source { case .pattern: break case .qr(let text): still = render(qr: text) @@ -66,12 +76,12 @@ final class FeedCommand: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate, exit(3) } - private func write(_ pb: CVPixelBuffer) { + private func write(_ pb: CVPixelBuffer, sourceKind: UInt32 = 0) { CVPixelBufferLockBaseAddress(pb, .readOnly) defer { CVPixelBufferUnlockBaseAddress(pb, .readOnly) } guard let base = CVPixelBufferGetBaseAddress(pb) else { return } store.write(width: CVPixelBufferGetWidth(pb), height: CVPixelBufferGetHeight(pb), - bytesPerRow: CVPixelBufferGetBytesPerRow(pb), bgra: base, sourceKind: 0, + bytesPerRow: CVPixelBufferGetBytesPerRow(pb), bgra: base, sourceKind: sourceKind, time: Date().timeIntervalSince1970) written += 1 } @@ -159,7 +169,7 @@ final class FeedCommand: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate, } func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { - if let pb = CMSampleBufferGetImageBuffer(sampleBuffer) { write(pb) } + if let pb = CMSampleBufferGetImageBuffer(sampleBuffer) { write(pb, sourceKind: UInt32(SimCamSourceKind.macCamera.rawValue)) } } } From 774b4fab7cb6dbbd9e7d280ac2223b42e61072fd Mon Sep 17 00:00:00 2001 From: khang <49549324+ntdkhang@users.noreply.github.com> Date: Sat, 17 Oct 2026 20:16:20 +0700 Subject: [PATCH 2/2] Refer to the integration caller as a downstream app --- CHANGELOG.md | 2 +- SimulatorInjection/Tests/NativeDriverProbe.swift | 6 +++--- SimulatorInjection/Tests/README.md | 4 ++-- SimulatorInjection/Tests/build-native-driver-probe.sh | 2 +- SimulatorInjection/Tests/run-native-driver-probe.sh | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e0756a9..d6734bf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm delegates/state, separate media connections, and rotation/mirroring transforms. - Isolated public-API recording probes decode finalized files and exercise repeat takes, cancellation, teardown, permission denial, and missing samples. An optional - integration harness compiles Small Hello's original Media sources unchanged. + integration harness compiles our downstream app's original Media sources unchanged. ### Changed - **Frame transport rewritten on CoreMediaIO sink streams.** The v1.0.0 design diff --git a/SimulatorInjection/Tests/NativeDriverProbe.swift b/SimulatorInjection/Tests/NativeDriverProbe.swift index bee367f..8ca1069 100644 --- a/SimulatorInjection/Tests/NativeDriverProbe.swift +++ b/SimulatorInjection/Tests/NativeDriverProbe.swift @@ -18,7 +18,7 @@ final class NativeDriverProbe: UIResponder, UIApplicationDelegate { label.frame = CGRect(x: 24, y: 70, width: 340, height: 90) label.textColor = .white label.numberOfLines = 0 - label.text = "Native Small Hello driver\nPreparing recording" + label.text = "Native downstream app driver\nPreparing recording" view.view.addSubview(label) let preview = AVCaptureVideoPreviewLayer() preview.frame = CGRect(x: 20, y: 180, width: 350, height: 450) @@ -61,7 +61,7 @@ final class NativeDriverProbe: UIResponder, UIApplicationDelegate { for take in 1...2 { let url = documents.appendingPathComponent("native-\(take).mov") try? FileManager.default.removeItem(at: url) - label.text = "Native Small Hello driver\nRecording take \(take)" + label.text = "Native downstream app driver\nRecording take \(take)" print("NATIVE requesting take \(take)") let completion = try await driver.beginRecording(to: url) print("NATIVE started take \(take)") @@ -80,7 +80,7 @@ final class NativeDriverProbe: UIResponder, UIApplicationDelegate { } print("NATIVE finalized \(url.path)") } - label.text = "Native Small Hello driver\nReplaying finalized video + audio" + label.text = "Native downstream app driver\nReplaying finalized video + audio" preview.removeFromSuperlayer() let playbackAudio = AudioSessionLease() try playbackAudio.activateForPlayback() diff --git a/SimulatorInjection/Tests/README.md b/SimulatorInjection/Tests/README.md index 547501b..0473805 100644 --- a/SimulatorInjection/Tests/README.md +++ b/SimulatorInjection/Tests/README.md @@ -52,7 +52,7 @@ After the public suite has built its tools: ```sh bash SimulatorInjection/Tests/run-native-driver-probe.sh "$DEVICE" \ - /path/to/small-hello/app/Sources/Media fixture + /path/to/downstream-app/app/Sources/Media fixture ``` This compiles the actual caller's Media sources in place, read-only, with Swift 6 @@ -73,7 +73,7 @@ process. It does not change global audio defaults or install a driver: ```sh bash SimulatorInjection/Tests/run-native-driver-probe.sh "$DEVICE" \ - /path/to/small-hello/app/Sources/Media live + /path/to/downstream-app/app/Sources/Media live ``` This machine-specific mode requires those two existing devices and already diff --git a/SimulatorInjection/Tests/build-native-driver-probe.sh b/SimulatorInjection/Tests/build-native-driver-probe.sh index 41a9167..9d741b9 100644 --- a/SimulatorInjection/Tests/build-native-driver-probe.sh +++ b/SimulatorInjection/Tests/build-native-driver-probe.sh @@ -2,7 +2,7 @@ set -euo pipefail export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Developer}" ROOT="$(cd "$(dirname "$0")/../.." && pwd)" -SOURCE="${1:?usage: build-native-driver-probe.sh /path/to/small-hello/app/Sources/Media}" +SOURCE="${1:?usage: build-native-driver-probe.sh /path/to/downstream-app/app/Sources/Media}" OUT="$ROOT/build/NativeDriverProbe.app" mkdir -p "$OUT" "$ROOT/build/evidence" SDK="$(xcrun --sdk iphonesimulator --show-sdk-path)" diff --git a/SimulatorInjection/Tests/run-native-driver-probe.sh b/SimulatorInjection/Tests/run-native-driver-probe.sh index 2f29bfb..21f2a3a 100644 --- a/SimulatorInjection/Tests/run-native-driver-probe.sh +++ b/SimulatorInjection/Tests/run-native-driver-probe.sh @@ -4,7 +4,7 @@ export DEVELOPER_DIR="${DEVELOPER_DIR:-/Applications/Xcode.app/Contents/Develope ROOT="$(cd "$(dirname "$0")/../.." && pwd)" cd "$ROOT" DEVICE="${1:?usage: run-native-driver-probe.sh DEDICATED_SIMULATOR_UDID MEDIA_SOURCE_PATH [fixture|live]}" -SOURCE="${2:?provide original Small Hello Media source path}" +SOURCE="${2:?provide original downstream app Media source path}" MODE="${3:-fixture}" bash SimulatorInjection/Tests/build-native-driver-probe.sh "$SOURCE" xcrun simctl install "$DEVICE" build/NativeDriverProbe.app