Acoustic Event Detection and Classification

Acoustic event detection (also called sound event detection, SED) is the task of identifying what sound occurred in an audio stream and when it occurred — a glass break, a dog bark, a bearing fault, a siren, a cough. It differs from audio classification (or tagging), which assigns labels to a whole clip without timestamps, and from speech-specific tasks such as speech recognition. The field is organized around benchmarks such as Google's AudioSet ontology and the annual DCASE (Detection and Classification of Acoustic Scenes and Events) challenges, which have standardized much of the terminology and evaluation practice.

Feature representations

Raw audio is a one-dimensional waveform sampled tens of thousands of times per second, and almost all practical systems first convert it into a time–frequency representation. The workhorse is the short-time Fourier transform (STFT), whose magnitude over time forms a spectrogram. Applying a mel-scaled filter bank and a logarithm yields the log-mel spectrogram, the de-facto standard input for neural audio models; adding a discrete cosine transform produces MFCCs (mel-frequency cepstral coefficients), a compact feature that dominated the field before deep learning and remains useful for lightweight or classical-ML systems.

Since the late 2010s, learned embeddings have become the third option: models such as VGGish, YAMNet, PANNs and, more recently, self-supervised transformers (e.g. BEATs) are pre-trained on very large audio corpora and expose a fixed-size vector per audio frame or clip. Training a small classifier on such embeddings is often the fastest way to a strong baseline when labeled data is scarce.

Classical DSP and ML hybrids

Production systems are rarely a single end-to-end network. A common pattern is a cheap, always-on DSP front end — an energy or spectral-flux detector, band-pass filtering around the frequencies of interest, noise suppression, onset detection — that gates a heavier learned classifier which runs only when something plausibly happened. This staging cuts power consumption dramatically on embedded devices and reduces false alarms, because the classical stage encodes physical knowledge (the band a glass break occupies, the harmonic structure of a siren) that a network would otherwise need data to rediscover. The general trade-offs of combining DSP priors with learning are covered in the companion article Signal Processing Meets Machine Learning and in MLAIA's blog post on classical DSP for ML.

Data and labeling challenges

Sound event data is hard to label well. Interesting events are often rare (a fault, an intrusion, a safety incident), so real recordings are dominated by background and class balance is extreme. Annotation comes in two strengths: strong labels with onset/offset timestamps, which are expensive and slow to produce, and weak labels that only say an event occurred somewhere in a clip — AudioSet, the largest public corpus, is weakly labeled. Much of modern SED research concerns learning temporal localization from weak labels.

Practitioners compensate with data augmentation and synthesis: mixing isolated event samples over recorded backgrounds at controlled signal-to-noise ratios, pitch and time shifting, SpecAugment-style spectrogram masking, and mixup. A further, frequently underestimated problem is domain shift: models trained on one microphone, room, or deployment site often degrade on another, so validation should always include recordings from unseen devices and environments.

Evaluation: precision and recall on rare events

Accuracy is meaningless when 99.9% of frames are background. Meaningful evaluation uses precision, recall, and F1 per event class, computed either segment-based (fixed time windows) or event-based (a detection counts if it overlaps a reference event within a tolerance). For always-on detectors, the operational metric is usually false alarms per hour at a fixed miss rate, since even a small false-positive rate becomes intolerable over days of continuous audio. DCASE has additionally standardized the polyphonic sound detection score (PSDS), which evaluates across operating points rather than at a single threshold. Whatever the metric, the evaluation set must contain enough true events — often requiring deliberate collection — for the estimates to be statistically stable.

Edge versus cloud inference

Where the model runs is an architectural decision with real consequences. Edge inference (on-device) offers low latency, operation without connectivity, and privacy — raw audio never leaves the device, which matters for anything recorded in homes, hospitals, or workplaces. It constrains the model to small quantized CNNs/CRNNs, often behind the DSP gate described above. Cloud inference permits large models and centralized updates but adds network latency, bandwidth cost, and a privacy surface that may trigger regulatory obligations. Many deployed systems are tiered: an on-device detector decides that something happened, and a cloud model decides what, receiving only short snippets or embeddings rather than continuous audio.

Frequently asked questions

What is the difference between sound event detection and audio classification?

Sound event detection outputs event labels with onset and offset times in a continuous stream; audio classification assigns one or more labels to an entire clip with no timing. Detection is the harder problem and requires either strongly labeled data or weak-label learning techniques.

Why are spectrograms used instead of raw waveforms?

Time–frequency representations make the structure of sound (harmonics, transients, formants) explicit, so models need far less data to learn it. Raw-waveform models exist and can match spectrogram models at scale, but at typical industrial dataset sizes the spectrogram prior almost always wins.

How much labeled data does an acoustic event detector need?

There is no universal number, but with pre-trained embeddings, useful detectors have been built from a few hundred labeled events per class, augmented synthetically. Training competitive models from scratch typically requires tens of thousands of examples or a large weakly labeled corpus.

MLAIA consults on acoustic event detection and audio AI systems — from feature design to edge deployment. See our signal & audio domain expertise or get in touch. Related reading: Audio AI in Production and Engineering Production Speech Systems.