Custom voice commands that run on the device.
List the commands your product should understand and give a few example phrases for each. A model trains in about a minute. You download one file, about 8 MB, that runs on the device with no network, turning a spoken phrase into one of your commands or unknown.
Written to be read cold. Each term is explained where it comes up, the build is laid out in full behind the panels, and a glossary closes the page.
A voice command model takes a short spoken phrase and returns one label from a fixed set you define, say take_photo or start_video, or unknown when the phrase is none of them. It does one job: map speech to your commands. It does not hold a conversation or write text.
On-device means it runs on the product's own chip. The audio is processed where it is spoken and nothing is sent to a server. That is what makes it private, fast, and able to work with no network.
01 the problem
Custom on-device voice commands are hard to build
Off-the-shelf assistants recognize their own vocabulary, not the commands specific to your product, and most answer by sending audio to the cloud. Building a command set that runs on the product itself is normally weeks of work: record audio across many speakers, train a model, then fit the result into the chip's power and memory budget.
An assistant that thinks in the cloud carries four costs into a shipped product. Privacy: raw audio leaves the device to be processed elsewhere. Latency: every command waits on a round trip to a server. Availability: no network means no voice. Cost: each request is a bill that grows with usage.
For a wearable, an appliance, a car, or a tool on a factory floor, those costs are the difference between a feature that ships and one that stays a demo.
The second wall is vocabulary. A closed assistant recognizes what its vendor trained it on. A camera that wants flip_camera, a thermostat that wants set_eco_mode, a headset that wants mute_mic, each needs a command set nobody else defines. Adding those to a general assistant is not something its API allows.
02 the product
List your commands. Train. Download.
Three steps in a browser at intent.unified-sciences.com. No audio to record, no ML to configure.
List your commands
Name each command and add a few ways people say it, one per line. A handful of phrases per command is enough to define it.
Train
Press train. The model is ready in about a minute, with an accuracy report measured on voices held out from the build.
Download and run
Download one bundle, about 8 MB. Drop it into your app; it runs on the device and returns one of your intents or unknown.
The bundle takes a spoken phrase and returns one label. Anything outside your set comes back as unknown, so your app can ignore it or re-prompt rather than guess.
Illustration of the input/output contract. Confidence values shown are illustrative, not benchmarks.
The download is self-contained. Unzip it, pass an audio clip, get a label and a confidence back. It runs on onnxruntime and numpy alone, with no deep-learning framework and no network.
python runtime.py clip.wav
→ {"intent": "take_photo", "confidence": 0.98}
Custom intents, and what "unknown" is for
An intent is a command you name: take_photo, next_track, call_home. You choose the set, so the model recognizes exactly the actions your product exposes and nothing else. Two products can ship completely different sets from the same builder.
Unknown is the safety valve. Real speech includes side conversation, background talk, and phrases you never trained. Rather than force every sound into the nearest command, the model returns unknown when a phrase does not match your set. Your app decides what that means, usually to do nothing or ask again, which keeps a stray sentence from triggering the wrong action.
03 how it works
How a model gets built
Every model is a fixed speech encoder with a small classifier trained on top. The encoder turns audio into a compact numeric summary; the classifier maps that summary to your commands. Only the classifier learns your set, which is why a build is quick and a bundle is small.
A speech encoder is a model that turns raw audio into numbers that capture what was said, largely independent of who said it or the background. An embedding is the fixed-length list of numbers that comes out, one per clip, positioned so that similar-sounding commands land near each other. Mean pooling is how a clip of any length becomes that single vector: the encoder emits one vector per short slice of audio, and their average is the embedding.
The encoder is frozen and shared by every project. Its weights never change during a build, and the same 8 MB file ships in every bundle. What differs per project is only the small classifier trained on top.
The speech encoder, frozen and shared
The encoder here is Moonshine-tiny (UsefulSensors/moonshine-tiny), run as a fixed feature extractor. Frozen means its weights are held constant: training changes nothing inside it. A general ability to hear speech is reused as-is, and only the mapping to your specific commands is learned.
Because the encoder is identical across projects, it is loaded once, shared for every build, and shipped as one int8 ONNX file of about 8 MB. Quantizing it to 8-bit keeps the embedding it produces almost unchanged, matching the full-precision encoder at a cosine similarity above 0.99, so the smaller file behaves like the original.
From a clip to one embedding, by mean pooling
The encoder reads audio at 16 kHz and emits a sequence of vectors, one per short slice, so a longer phrase produces a longer sequence. A classifier needs a fixed-size input, so the sequence is collapsed by mean pooling: average the vectors across time into a single vector, the embedding, the same size for a one-word command and a five-word one.
That pooled embedding is what the classifier reads. The shipped encoder ONNX does the pooling itself, so at run time one clip goes in and one embedding comes out.
Training only the head, and why a build is quick
The head is the classifier trained per project: a small two-layer network that takes the embedding, passes it through a 128-unit layer with a GELU nonlinearity and light dropout, and outputs one score per label. The labels are your intents plus one extra class, unknown.
Frozen versus trained: the encoder's weights stay fixed while only the head's weights change. Since the encoder does not move, every clip's embedding is computed once and cached, and training is just fitting the small head on those fixed vectors, 220 optimization steps that finish in about 15 seconds. The full build in the browser, which also synthesizes the training voices, is about a minute. A frozen encoder plus a tiny trained head is also why the per-project part of a bundle is small: the 8 MB is the shared encoder, and the head is a handful of numbers on top.
Where the training audio comes from, synthesized not recorded
No customer recordings are needed. Each example phrase is first expanded into several phrasings: synonym swaps (take/capture, photo/picture), carrier templates (can you…, …please), a capture-verb bridge so record a video also teaches take a video within the same media family, and light word-order edits.
Each phrasing is spoken by 5 text-to-speech voices spanning US, UK, Indian, and Australian accents (via Microsoft edge-tts), with randomized speaking rate (−12% to +12%) and pitch (±15 Hz). The resulting clips are augmented for varied rooms and microphones: speed change (0.9–1.1×), white or pink noise at 5–25 dB SNR, light reverb, and random gain.
One more pass is free. Because the encoder is frozen, the cached embeddings themselves are augmented with small Gaussian-noise copies and with mixup during training, adding variety without synthesizing more audio.
Teaching the model to say "unknown", and measuring it fairly
Alongside your commands, the build synthesizes a set of 12 generic phrases (what time is it, tell me a joke, never mind) and labels them unknown. That gives the classifier a class for speech that is none of your commands, so out-of-set phrases are rejected instead of forced into the nearest match.
Evaluation is kept apart from the build. The accuracy report uses the original phrasings, neutral prosody, and 2 voices held out from training, so the number reflects speakers the model was not built from rather than the exact clips it saw.
04 on-device & edge
Why it runs on the chip
Running on the device is the point, not a detail. It is what makes the model private, quick, and independent of a network. On a modern Qualcomm chip there is dedicated silicon for exactly this kind of work.
The edge is the device itself: the phone, earbud, headset, camera, or appliance in someone's hand, as opposed to a server it talks to. Running at the edge means the work happens on that hardware.
Modern Snapdragon chips carry an NPU, a neural processing unit built to run models at low power. A model earns a place on it by fitting a tight budget for memory and energy. An 8 MB bundle that answers in about a millisecond fits that budget, which is why voice can live on the chip instead of in the cloud.
per inference · Snapdragon 8 Elite
Privacy, the audio never leaves the device
The model runs where the phrase is spoken. No audio is uploaded, and no server sees it. What the rest of the system receives is a short label such as take_photo, a few bytes, not a recording.
For a product that lives in a pocket, on a face, or in a living room, that removes the largest privacy question a voice feature raises. There is no recording in transit to intercept and no audio sitting in a datacenter.
Latency, an answer in about a millisecond
A cloud command spends most of its time in the network: capture, upload, queue, process, download. Even a fast connection adds tens to hundreds of milliseconds, and a weak one adds seconds. On the device that round trip disappears. The measured figure is about 1.2 ms per inference on a Snapdragon 8 Elite.
That headroom matters for anything interactive. A voice command that responds in single-digit milliseconds feels instant, and it leaves room for the rest of the audio pipeline to run in the same frame.
Offline, no network and no cloud bill
Nothing about the bundle needs a connection at run time. It works on a plane, in a basement, on a factory floor, in a car out of coverage. The same command set behaves the same way whether or not there is signal.
There is also no per-request cost. A cloud voice API charges for every command for the life of the product; an on-device model is a one-time build. Usage can scale to every unit shipped without a running bill.
The model is already profiled on a Snapdragon 8 Elite through Qualcomm AI Hub, at about 1.2 ms per inference. Two steps take it further onto Qualcomm silicon.
int8 on the NPU: the encoder already ships as int8 at about 8 MB; compiling that quantized graph onto the NPU through AI Hub would cut memory roughly 4× more for the tightest budgets. Earbud-class profiling: AI Hub exposes mobile Hexagon today, so the current number is a phone-class proxy; measuring on a hearables NPU such as the S5 Gen 3 gives the figure that matters for that class of product.
In numbers
A model in about a minute
List your commands, press train, download the bundle. The head itself fits in about 15 s; the full build is about 1 min.
~8 MB, fully offline
The whole bundle is about 8 MB and runs on onnxruntime and numpy with no network at run time.
~1.2 ms per inference
On a Snapdragon 8 Elite via Qualcomm AI Hub (fp32): about 1.2 ms for one phrase.
Your intents, or unknown
Returns one label from your set, or unknown for anything else, so you can ignore or re-prompt.
Nothing leaves the device
Audio is processed where it is spoken. 0 bytes go to a server at run time.
int8 keeps the features
The 8-bit encoder matches full precision at cosine >0.99, so the small file behaves like the original.
05 glossary
Every term, defined
- Voice command model
- A model that takes a short spoken phrase and returns one command from a fixed set, or
unknown. It maps speech to actions; it does not transcribe or converse. - Intent
- A command you name and want recognized, such as
take_photoormute_mic. You choose the full set. - Unknown
- The extra class the model returns when a phrase matches none of your intents. Lets an app ignore stray speech or re-prompt instead of guessing.
- Utterance
- One spoken phrase handed to the model, the unit it acts on.
- Confidence
- A number from 0 to 1 returned with each label, how sure the model is. An app can set a threshold below which it treats a result as unknown.
- Speech encoder
- A model that turns raw audio into numbers capturing what was said, largely independent of the speaker. Here it is fixed and shared, not trained per project.
- Moonshine-tiny
- The specific open speech encoder used (
UsefulSensors/moonshine-tiny), run as a frozen feature extractor. - Embedding
- The fixed-length vector the encoder produces for a clip. Similar-sounding commands land near each other, which is what the classifier reads.
- Mean pooling
- Averaging the encoder's per-slice vectors over time into one embedding, so a clip of any length becomes a single fixed-size vector.
- Frozen weights
- Weights held constant during a build. The encoder is frozen; only the head's weights change.
- Head (classifier)
- The small two-layer network trained per project. It reads an embedding and outputs one score per label. The only part that learns your commands.
- Text-to-speech (edge-tts)
- Turning written phrases into spoken audio. The build uses Microsoft edge-tts across several voices to synthesize training clips, so no recordings are needed.
- Data augmentation
- Making varied copies of training data, here by changing speaking rate and pitch, adding noise and reverb, and perturbing the embeddings, so the model generalizes.
- Mixup
- An augmentation that trains on blends of two examples and their labels, which smooths the classifier's decisions.
- SNR
- Signal-to-noise ratio in decibels, how loud speech is against added noise. Training clips span 5–25 dB to cover quiet and noisy rooms.
- Held-out voices
- Speakers used only to measure a model, never to build it, so the reported accuracy reflects voices it has not seen.
- Model bundle
- The single download, about 8 MB: the shared int8 encoder, this project's head, labels, and a small runtime.
- ONNX / onnxruntime
- A portable model format and a light engine that runs it. The bundle runs on onnxruntime and numpy, with no deep-learning framework.
- int8 / quantization
- Storing a model in 8-bit numbers to shrink and speed it up. The encoder ships this way at cosine >0.99 to full precision; a further int8 build on the NPU would cut memory about 4×.
- On-device
- Running on the product's own chip rather than a server. Nothing is sent off the device to get an answer.
- Offline
- Working with no network connection. An on-device model behaves the same with or without signal.
- Edge
- The device where the work happens, as opposed to the cloud it might otherwise call.
- NPU
- Neural processing unit. Silicon on a modern chip built to run models at low power, where a small model like this one runs.
- Snapdragon / Hexagon
- Qualcomm's mobile chips (Snapdragon) and the processor family behind their on-device AI (Hexagon).
- Qualcomm AI Hub
- Qualcomm's service for compiling and profiling a model on real Snapdragon hardware. Source of the ~1.2 ms figure.
- S5 Gen 3
- Qualcomm's sound platform for hearables, which carries its own NPU, the target for an earbud-class latency figure.
- Latency
- The time from a spoken phrase to a returned label. Measured here at about 1.2 ms per inference.