For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development or superpowers:executing-plans to implement task-by-task. Steps use checkbox syntax for tracking.
Goal: Move normal audio-frame MIDI submission and SysEx message allocation off the audio callback onto the existing MIDI sender worker, as the user requested.
Architecture: Add a bounded SPSC queue of owned variable-length byte packets alongside the existing short-message queue. Audio-side writers copy raw bytes into preallocated slots; the existing worker constructs JUCE messages and submits them. Keep LED writer state retryable on queue-full, service short-message timing before each bounded SysEx dispatch, and join the worker before output-handler destruction/LED shutdown clearing.
Tech Stack: C++17, existing CircularQueue, JUCE8.0.15, doctest, iOS Release build and authenticated Wi-Fi deployment.
The user asked whether sendMessageNow on the audio thread could be problematic and explicitly requested moving SysEx to the MIDI thread as a lasting improvement. We found both WB and Launchpad frame-driven SysEx submissions, plus Twister/K-Mix frame-driven short-message bypasses. Route those short messages through the existing worker too, so normal frame processing makes no direct CoreMIDI call. Connection handshakes and final LED clearing are control-thread operations and do not become additional producers of the SPSC queue. Serialize them through the output handler and stop the worker before final clearing. The current MIDI-worker-off exposure must finish and be archived on its existing binary before heavy builds or deployment. Source/light focused tests can proceed while it records. No repeated approval or new user task is required; no commits/landing of the surrounding dirty diagnostics are requested.
Alternatives considered: a queue of juce::MidiMessage would still allocate/copy heap-backed SysEx on the audio producer; enlarging all16384 basic-message slots wastes tens of MB; the existing CircularByteQueue spins when full and cannot be used on audio. A small separately owned packet queue avoids those costs. This is not evidence that the call caused either observed symptom: the next run must score both.
Files: Create private/src/MidiSysexQueue.hpp and private/test/unit/midi_sysex_queue.cpp. Modify JUCE/SmartGridOne/Source/MidiSender.hpp, NonagonWrapper.hpp and MainComponent.cpp only for routing/worker lifetime. Existing private/test/CMakeLists.txt automatically discovers new unit files; no new test framework or project required.
Queue interface: Implement SmartGrid::MidiSysexQueue<MaxMessageBytes, QueueSize> using CircularQueue<Packet, QueueSize> with public Packet fields int m_routeId, size_t m_size, and fixed uint8_t m_data[MaxMessageBytes]. Expose bool TryPush(const uint8_t* data, size_t size, int routeId), Packet* Peek(), void Pop(), and size_t Size() const. TryPush validates non-null data and0<size<=MaxMessageBytes before reserving a producer slot with NextToPush, writes exactly size bytes and route/length, then CompletePush. A failed push leaves queued packets intact. Peek/Pop keep the current consumer slot owned until Pop. Document one producer(audio) and one consumer(MIDI worker). Size is a documented approximate third-thread diagnostic snapshot clamped to QueueSize; it is never used for capacity decisions. Generic CircularQueue remains unchanged.
MidiSender integration: Add SendSysex(const uint8_t*, size_t, int) returning bool accepted for handling. A true return means queued or deliberately discarded by disabled mode; false means invalid/full and needs a writer retry. Disabled mode increments a separate relaxed discard counter and returns true before queue access. Reject invalid route/null/size with a diagnostic counter and suitable assertion for impossible producer inputs. Enabled successful enqueue increments a relaxed queued counter; full increments a separate overflow counter. In run, HandleMessage then HandleSysex then the existing sleep. HandleSysex reads Peek, constructs juce::MidiMessage only there, submits via the existing handler SendMessage under its lock, counts submission, then Pop. Validate stable route before dereference. Extend existing once-second LogDiagnostics with SysEx queued/enqueued/submitted/full/discarded counters; no per-packet log. Add compile-time size checks at the concrete producer integration if writer types are not available in MidiSender.
Routing: WB Process calls SendSysex with buffer.m_buffer/m_size/m_routeId and resets only its failed color writer; Launchpad Process does the equivalent and resets its writer on rejection. Give Launchpad, encoder and K-Mix handlers their owner MidiSender pointer using the existing initialization/constructor flow. Route their normal short messages through SendMessage, preserving routes and full MIDI bytes. Do not construct juce::MidiMessage in normal audio-frame output handlers. Reuse existing route allocations. Handshake stays on control thread but uses handler.SendMessage for serialization.
Lifetime: Make shutdown join the MIDI worker on the non-audio owner/control thread. In MainComponent destructor CloseAudioDevice before ClearLEDs; in NonagonWrapper destructor stop the MIDI worker before any route handler destruction, while retaining I/O shutdown. ClearLEDs must stop/join queued MIDI before direct final clearing; use the handler’s synchronized send. Do not force-kill a worker while it owns a queue slot or output lock. Repeated shutdown is harmless. No restart after this terminal shutdown is required.
Example required owned-data test (adapt doctest macro names exactly to repository conventions):
SmartGrid::MidiSysexQueue<16, 2> queue;
uint8_t bytes[] = {0xF0, 0x79, 0x01, 0xF7};
DOCTEST_REQUIRE(queue.TryPush(bytes, sizeof(bytes), 3));
bytes[1] = 0;
auto* message = queue.Peek();
DOCTEST_REQUIRE(message != nullptr);
DOCTEST_CHECK(message->m_routeId == 3);
DOCTEST_CHECK(message->m_size == 4);
DOCTEST_CHECK(message->m_data[1] == 0x79);
Cover queue-full rejection preserving both existing packets, FIFO route/payload order, producer wraparound, null/zero/oversize rejection, exact maximum payload including its last F7 byte, and the consumer holding a slot while a producer attempts to fill/reuse it. Include a bounded real producer/consumer test with varying route/payload patterns, not a mocked JUCE test. Test expectations must be independent literal/pattern fixtures. A realistic pointer-retention, truncation, early-pop or publication-order bug should fail these tests.
clang++ -std=c++17 -O2 -pthread -DDOCTEST_CONFIG_NO_SHORT_MACRO_NAMES=1 -Iprivate/test -Iprivate/src private/test/support/TestMain.cpp private/test/unit/midi_sysex_queue.cpp -o /private/tmp/smartgrid-midi-sysex-queue-tests
/private/tmp/smartgrid-midi-sysex-queue-tests
Files: Record protocol/results in docs/experiments/ipad-maya-audio and the existing checkpoint; source changes remain those reviewed above.