Audio & Acoustics · Edge & embedded
On-device audio AI moves keyword spotting, sound event detection and audio classification onto phones, microcontrollers, DSPs and dedicated controllers. We engineer the path from microphone and front-end features to a quantized streaming model, and evaluate it on the target hardware with real microphones.
Why edge audio is its own discipline
A model that scores well in a notebook can still fail on a device. On constrained hardware, feature extraction, buffering, runtime and power management share one small budget with the network, and each choice changes what the model sees.
Our project experience includes deploying machine-learning and deep-learning models on Android and iOS phones and on dedicated controllers for applications involving drones and earphones. Across platforms, the lesson is the same: keep training and on-device features identical, and test with the shipping microphones. The Audio & Acoustics practice covers the wider signal chain.
Always-on detection of a small vocabulary, typically as a cascade: a cheap first stage runs continuously and a larger verifier wakes only on candidates. The design trades false accepts per hour against false rejects and idle power.
Recognizing alarms, impacts, machinery states or other target sounds locally, without streaming audio off the device. Class definitions, onset handling and post-processing such as median filtering or hysteresis matter as much as the classifier. See the acoustic event detection guide.
Small networks that estimate noise-suppression masks or gains run on every frame, so their cost and algorithmic delay add directly to the audio path. They usually complement classical processing; see noise reduction & ANC.
Engineering the pipeline
Most on-device audio failures come from mismatches between stages, not from the network itself. These are the areas we work through with your team.
Log-mel or MFCC extraction means windowing, an FFT, a mel filterbank and a log on every hop. On a microcontroller this can cost as much as a small model. We size the FFT, mel bands and hop, use fixed-point or vendor DSP kernels, and verify on-device features match training numerically.
Frame length and hop set time resolution and minimum algorithmic delay. Convolutional models can stream by caching internal state, so each hop computes only new frames instead of re-running a full window. Buffering, inference time and decision smoothing go into one latency budget.
Post-training int8 quantization with a representative calibration set comes first; quantization-aware training follows if accuracy drops. We check per-channel scales, activation ranges after the log and operator support. On most small cores, structured pruning and architecture changes cut compute more reliably than unstructured sparsity.
TensorFlow Lite and TFLite Micro, Core ML, ONNX Runtime and vendor NPU toolchains support different operators and quantization schemes. We choose architectures that convert cleanly, compare converted outputs with the reference, and profile per-layer latency and peak memory on the device.
Continuous listening is dominated by what runs when nothing happens. We look at duty cycling, voice-activity or energy gating, a low-power first stage that wakes a larger core, and sample rate. Power is measured on the board in each operating state.
Data recorded on studio or laptop microphones rarely matches a MEMS microphone behind a port and grille. We characterize the device response and noise floor, then augment with measured responses and recorded device noise. Multi-microphone products can add a beamforming front-end.
What we measure
We agree on metrics before optimizing, and report them for the deployed build, with the real microphone and runtime.
For wake words, false rejects at a fixed false-accept rate per hour of realistic background audio, not accuracy on balanced clips. For sound events, event-based precision and recall with onset tolerance, and false alarms per day in the field. Results are split by noise type and distance.
Time from the acoustic event to the device acting on it: buffering, feature hop, inference, smoothing and any core wake-up. We measure it end to end with a reference signal and report the distribution, not an average.
Flash for weights and code, peak RAM including tensor arena and audio buffers, cycles or milliseconds per hop, and the CPU or DSP headroom left for the rest of the application.
Average current while idle-listening and when active, measured on the board. Robustness tests cover the shipped microphone and enclosure, playback through the device's own speaker, and drift between float, quantized and on-device outputs.
Achievable numbers depend on your hardware, environment and data. We set targets with you against representative recordings; we do not promise benchmark figures.
How an engagement runs
Engagements range from reviewing an existing on-device model to building the pipeline with your firmware or mobile team. A typical path:
Agree on target hardware, runtime, latency, memory and power envelopes, and the error measures that matter. Profile a reference model on the device early, so the budget rests on measurements.
Record audio through the product's microphone and enclosure, build a negative set of realistic background audio, and lock one front-end specification for training and firmware.
Select architectures that fit the operator set and memory, train with device-realistic augmentation, then quantize and compare float, quantized and on-device outputs.
Run the agreed tests on production-representative units, and document the pipeline, conversion steps and test harness so your team can retrain and redeploy.
Before we start
For well-scoped tasks such as keyword spotting or a small set of sound events, often yes. Published keyword-spotting models commonly fit in tens to a few hundred kilobytes after int8 quantization. Whether yours fits depends on the classes, acoustic variability and what the rest of the firmware needs, so we profile a candidate on your hardware early.
On-device inference avoids network latency, works offline and keeps raw audio on the device. The cloud allows larger models and easier updates. Many products split the work: a small on-device model detects or gates, and sends only relevant segments or results onward. We compare both against your latency, privacy, cost and update requirements.
It varies. Well-calibrated post-training quantization often loses little, but audio models can be sensitive where activation ranges are wide, for example right after the log in a log-mel front-end. We measure float, quantized and on-device outputs on your test set and use quantization-aware training when the gap matters.
Common causes are feature mismatches between Python and firmware (window, FFT scaling, mel filterbank or log floor), a microphone response and noise floor unlike the training data, and quantization or conversion differences. We trace the same audio through each stage on both sides and compare outputs numerically to find the divergence.
Usually some. Public datasets help with pre-training and negatives, but a modest set of recordings through your microphone and enclosure, in realistic conditions, often closes the gap between lab and field. We help design the recording protocol and use measured device responses to make existing data more representative.
Keep exploring
Audio & Acoustics · Edge Audio AI
Share the problem, the data you have and what success would look like. We’ll discuss a practical next step.
Please don’t include confidential datasets, credentials or patient information.