# Machine learning on embedded systems: how TinyML runs models on microcontrollers

> Machine learning on embedded systems runs small int8 models on microcontrollers (TinyML): what fits, how models shrink, the 2026 tools and field updates.

- URL: https://computese.com/machine-learning-on-embedded-systems/
- Author: Duong Quan Nguyen, CEO, Computese
- Published: 2024-05-28
- Updated: 2026-09-25
- Topics: AI & automation

## In short
- Machine learning on embedded systems runs a trained model on the device that holds the sensor: a microcontroller with a few hundred kilobytes of memory (TinyML), or an embedded Linux board with gigabytes and an accelerator.
- On a microcontroller, flash holds the model and a small SRAM holds the working memory. A standard ResNet-50 overshoots that budget about 100 times, so models are designed small, pruned or distilled, and quantized to 8-bit integers.
- As of September 2026 the main toolchains are LiteRT for Microcontrollers (formerly TensorFlow Lite for Microcontrollers), PyTorch's ExecuTorch and Edge Impulse (a Qualcomm company), with Arm's CMSIS-NN kernels underneath on Cortex-M.
- MLPerf Tiny compares hardware on five fixed tasks and reports latency and energy per inference at a fixed accuracy target, which says more about a chip than a peak TOPS figure.
- The hard parts come after the demo: data from the real sensor, testing on the real board, signed updates that can roll back, and watching the model for drift.

Machine learning on embedded systems means running a trained model on the device that holds the sensor: a microcontroller with a few hundred kilobytes of memory, or an embedded Linux board with an accelerator. On microcontrollers the field is called TinyML. The model is trained on an ordinary computer, shrunk to 8-bit integers and compiled into the device's firmware.

This guide covers what counts as an embedded device, why you would run a model there, the limits you work within, how models are made to fit, the toolchains and chips available as of September 2026, the MLPerf Tiny benchmark and the problems that start after the first demo. The NPUs in phones and laptops are a different class of hardware, covered in [AI in everyday devices](https://computese.com/ai-in-everyday-devices-transforming-technology/). If machine learning itself is new to you, start with [a first machine learning project in Python](https://computese.com/artificial-intelligence-with-python/).

## What counts as an embedded system for machine learning

An embedded system is a computer built into a product to do one job: the controller in a thermostat, a motor drive, a hearing aid or a doorbell. For machine learning, two very different kinds of hardware share that name, and almost every decision in a project depends on which one you have.

|                       | Microcontroller (TinyML)                                            | Embedded Linux board with an accelerator                 |
| --------------------- | ------------------------------------------------------------------- | -------------------------------------------------------- |
| Example               | STM32F746 (Arm Cortex-M7, 216 MHz) or ESP32-S3 (dual-core, 240 MHz) | NVIDIA Jetson Orin Nano Super                            |
| Working memory        | 320 KB (STM32F746) or 512 KB (ESP32-S3) of SRAM                     | 8 GB of LPDDR5                                           |
| Model storage         | 1 MB of flash on the STM32F746                                      | SD card or NVMe drive                                    |
| Power while inferring | Under a milliwatt to a few milliwatts                               | 7 to 25 W                                                |
| Operating system      | None, or a small real-time OS                                       | Linux                                                    |
| What runs well        | Wake words, sensor anomalies, low-resolution person detection       | Vision transformers, vision-language and language models |

The microcontroller figures come from a [2023 survey in _IEEE Circuits and Systems Magazine_](https://arxiv.org/abs/2403.19076), which calls the STM32F746 a popular Cortex-M7 part, and from [Espressif's ESP32-S3 page](https://www.espressif.com/en/products/socs/esp32-s3). The board figures are from [NVIDIA's Jetson Orin Nano Super page](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit/), which rates it at 67 INT8 TOPS. The gap is not small: the same survey puts microcontrollers about three orders of magnitude below phones in memory and storage, and five to six below cloud GPUs.

MLCommons, which runs the main benchmark for these devices, [defines TinyML](https://mlcommons.org/2026/07/mlperf-tiny-v1-4-results/) as models small enough, typically under 2 million weights, to run on devices that draw from under a milliwatt to a few milliwatts. The rest of this guide is mostly about that end of the range, and notes where a Linux board changes the answer. On a Linux board you run the ordinary runtimes: Google's documentation suggests [standard LiteRT rather than the microcontroller runtime](https://developers.google.com/edge/litert/microcontrollers/overview) as easier to integrate on a device like a Raspberry Pi.

## Why run a model on the device instead of the cloud

Sending sensor data to a server and running the model there is simpler to build. Running it on the device wins when one of these matters:

- **Latency.** The decision is made next to the sensor, with no network round trip. Google's LiteRT documentation lists reliance on an internet connection, with its bandwidth limits and high latency, as the problem microcontroller inference removes.
- **Energy.** On battery devices the radio is the expensive part. The [paper that introduced MLPerf Tiny](https://arxiv.org/abs/2106.07597) notes that at this scale the energy cost of wireless communication is far higher than that of the computation. A sensor that sends a one-line alert instead of a continuous audio stream keeps its radio off most of the time.
- **Privacy.** Raw audio, images and vibration traces never leave the device. Only the result does, if you choose to send it.
- **Working offline.** A model compiled into the firmware keeps working in a basement, a field or a moving vehicle. [Edge Impulse's deployment documentation](https://docs.edgeimpulse.com/studio/projects/deployment) makes the same point for its C++ library: it runs without an internet connection.
- **Cost.** A [2022 review in _IEEE Sensors Journal_](https://arxiv.org/abs/2205.14550) put a Cortex-M4 class microcontroller at around 5 to 10 US dollars, able to run on a coin-cell battery for months or years.

The cloud is still the right place when the model needs more memory than any affordable device has, when events are rare and uploading them is cheap, or when the model changes daily. A common middle path is a cascade: a tiny always-on model listens for a wake word or an unusual vibration, then wakes a larger processor or sends a short clip to a bigger model. The MLPerf Tiny paper describes wake-word detection in exactly that role: a detector that runs continuously in order to wake a larger processor.

## The constraints: memory, flash, power and no operating system

A microcontroller has two kinds of memory, and a model uses them differently. Flash is non-volatile and, while the device runs, read-only: it holds the program and the model's weights. SRAM is the small read-write memory where the input, the intermediate results of each layer (the activations) and the runtime's bookkeeping live. The 2023 survey sums it up: SRAM limits the activations and flash limits the model size. On the STM32F746 that is 1 MB of flash and 320 KB of SRAM, with a clock the authors put at 10 to 20 times slower than a laptop's.

![A microphone feeds a waveform into a microcontroller chip that holds a tall flash block of stacked weight cards and a small orange SRAM block where intermediate results are written and erased.](https://computese.com/images/blog/machine-learning-on-embedded-systems/memory.dc7c1bed07-1536.webp)

*The model file lives in flash; the math runs in the much smaller SRAM, which is why peak working memory, not file size, decides what fits.*

That is why a model's parameter count is a poor guide to whether it fits. The survey finds that the real bottleneck is activation memory, not the number of parameters: MobileNetV2 has far fewer parameters than ResNet, yet needs more peak memory. Measured against a microcontroller's budget, ResNet-50 is about 100 times too large, MobileNetV2 about 20 times, and even an int8-quantized MobileNetV2 5.3 times.

The software environment is just as spare. [LiteRT for Microcontrollers](https://developers.google.com/edge/litert/microcontrollers/overview), Google's runtime for these devices, needs no operating system, no standard C or C++ library and no dynamic memory allocation, and its core fits in 16 KB on an Arm Cortex-M3. Instead of allocating memory as it goes, it reserves one contiguous block of SRAM, the arena (in code, the tensor arena), and packs every tensor into it; the 2022 review notes that this avoids memory fragmentation. You choose the arena's size when you build the firmware, so peak memory is something you measure before shipping, not something you discover in the field.

Two more limits shape the design:

- **Power.** TinyML workloads are designed to run always on at around a milliwatt or less, so the device sleeps between inferences and each inference has to finish quickly.
- **Training.** These devices run inference only. LiteRT for Microcontrollers does not support on-device training, and the 2023 survey measures full training at 6.9 times the memory of inference. Models are trained elsewhere and shipped to the device.

## How models are made to fit

Four techniques do most of the work, and they are usually combined: quantization almost always, an architecture built for the budget from the start, and pruning or distillation when you still need more.

### Quantize to 8-bit integers

Quantization stores each weight and activation as an 8-bit integer instead of a 32-bit float, with scale factors that map the real range of values onto 256 integer levels. Google's [post-training quantization guide](https://developers.google.com/edge/litert/conversion/tensorflow/quantization/post_training_quantization) lists full integer quantization as making a model four times smaller and three or more times faster, and as the option that runs on microcontrollers. To set the scales for activations, the converter needs a representative dataset: around 100 to 500 real input samples.

![A grid of finely shaded weight cells passes through an orange staircase ruler, snaps to its coarse steps, and comes out as a narrower grid of flat cells that slides into a small chip.](https://computese.com/images/blog/machine-learning-on-embedded-systems/quantize.0df0e4c7d3-1536.webp)

*Rounding every value onto 256 steps makes the model four times smaller and lets integer-only hardware run it.*

In TensorFlow, the integer-only conversion from that guide looks like this:

```python
import tensorflow as tf

def representative_dataset():
    for sample in calibration_samples:  # 100 to 500 real inputs
        yield [sample]

converter = tf.lite.TFLiteConverter.from_saved_model("saved_model_dir")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
open("model.tflite", "wb").write(converter.convert())
```

The last step in [Google's workflow](https://developers.google.com/edge/litert/microcontrollers/overview) turns the model file into a C byte array, stored in read-only program memory, for example with `xxd -i model.tflite > model_data.cc`. PyTorch users get the same result through ExecuTorch, whose [Ethos-U backend](https://docs.pytorch.org/executorch/stable/backends/arm-ethos-u/arm-ethos-u-overview.html) is integer-only and supports both post-training quantization and quantization-aware training, where the model learns during training to tolerate the rounding.

Quantization is rarely free. In the MLPerf Tiny anomaly-detection benchmark, the reference model scores an AUC of 0.88 in 32-bit floats and 0.86 after quantization, and the benchmark's pass mark was set just below that, at 0.85. Always test the quantized model, not the float one you trained.

### Prune what the model does not need

Pruning removes weights that contribute little, or whole channels and filters. The 2022 review compared both on its case studies: pruning compressed models by 13.6 times on average against 3.9 times for quantization, and 16 times when combined. But pruning saved less SRAM and less time, because a pruned layer still multiplies at the original precision, and unstructured pruning (zeroing individual weights) can even add overhead on a microcontroller. It also cost more accuracy: 4.9% on average against 0.4% for quantization. Prune to save flash; quantize to save SRAM and time.

### Distill a large model into a small one

Knowledge distillation trains a small student model to match the output probabilities of a larger, accurate teacher, not only the correct labels. [Hinton, Vinyals and Dean](https://arxiv.org/abs/1503.02531) called these probabilities soft targets: they carry information about which wrong answers are nearly right, which a small model cannot easily learn from labels alone. The teacher never ships; only the student goes on the device.

### Start from an architecture built for the budget

The biggest savings come from not starting with a server model at all. MLPerf Tiny's keyword spotter is a depthwise-separable CNN with 38.6 thousand parameters; its person detector is a MobileNetV1 shrunk to a quarter of its usual width, working on 96 by 96 pixel images. Neural architecture search takes this further: [MCUNet](https://arxiv.org/abs/2007.10319) (NeurIPS 2020) designed the network and its inference engine together and was the first to pass 70% top-1 accuracy on ImageNet on an off-the-shelf microcontroller.

Sometimes the right model is not a deep network. In one case study in the 2022 review, a small multilayer perceptron working on spectral features matched a CNN working on raw sensor data while using 2.5 times less flash, 18 times less SRAM and running 2.2 times faster. Good signal processing before the model is often the cheapest optimization there is.

## The toolchains in 2026: LiteRT, ExecuTorch, Edge Impulse and CMSIS-NN

Most projects take one of three routes from a trained model to firmware, and on Cortex-M chips two of them can run the same Arm kernel library underneath. Names changed recently, so dates matter here.

| Tool                        | Owner                                                   | What it does                                                                      | Status as of September 2026                                                          |
| --------------------------- | ------------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| LiteRT for Microcontrollers | Google (Google AI Edge)                                 | C++ 17 interpreter for 32-bit microcontrollers; runs converted `.tflite` models   | Formerly TensorFlow Lite for Microcontrollers (TFLM); documentation updated May 2026 |
| ExecuTorch                  | PyTorch, with Meta, Arm, Apple and Qualcomm as partners | Exports PyTorch models to run on phones down to microcontrollers                  | Documentation at version 1.5; Cortex-M backend in beta                               |
| Edge Impulse                | Qualcomm (acquired March 2025)                          | Web platform from data collection to training to a deployable library or firmware | Offers int8 or float32 builds and its own EON Compiler                               |
| CMSIS-NN                    | Arm (open source)                                       | Optimized neural network kernels for Cortex-M                                     | Available under both LiteRT and ExecuTorch on Cortex-M                               |

**LiteRT for Microcontrollers.** Google [renamed TensorFlow Lite to LiteRT](https://developers.googleblog.com/en/tensorflow-lite-is-now-litert/) on September 4, 2024, to reflect that it now runs models authored in PyTorch, JAX and Keras too. The microcontroller runtime is written in C++ 17, needs a 32-bit platform, has been tested extensively on Arm Cortex-M and ported to ESP32, and ships as an Arduino library. Its limits are clear in its own documentation: a subset of TensorFlow operations, a limited set of devices, a low-level C++ API with manual memory management, and no training. You will still see the old name, TFLM, in the source repository and in benchmark reports.

**ExecuTorch.** [ExecuTorch](https://docs.pytorch.org/executorch/stable/intro-overview.html) is PyTorch's own runtime for on-device inference, from high-end phones to constrained microcontrollers, built on PyTorch 2's export instead of the older TorchScript. For microcontrollers its [Cortex-M backend](https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html) replaces quantized operators with CMSIS-NN kernels and falls back to portable floating-point code for the rest; it is in beta and has been validated with MLPerf Tiny models and MobileNetV2. The Ethos-U backend hands the quantized graph to Arm's Vela compiler for the NPU.

**Edge Impulse.** A hosted platform that covers the whole loop, which the 2022 review describes as running from data collection through feature extraction and training to deployment. Its deployment page offers an int8 or float32 build and its EON Compiler, which the documentation says uses less RAM and flash than LiteRT for Microcontrollers at the same accuracy. Before you deploy, it estimates latency, flash and RAM for your target board. Edge Impulse [was acquired by Qualcomm Technologies in March 2025](https://www.edgeimpulse.com/about) and still deploys to other vendors' chips, including Arm Ethos-U.

**CMSIS-NN.** [Arm's kernel library](https://github.com/ARM-software/CMSIS-NN) is the layer underneath. It follows the int8 and int16 quantization specification of TensorFlow Lite for Microcontrollers, so its results are bit-exact with the reference kernels, and it has three versions of each kernel: plain C for cores like the Cortex-M0 and M3, DSP instructions for the Cortex-M4 and M33, and the Helium vector extension (MVE) for the Cortex-M55 and M85. ExecuTorch's Cortex-M backend calls it directly, and LiteRT for Microcontrollers [switches to it with one build flag](https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/micro/kernels/cmsis_nn) in place of its reference kernels.

Chip vendors add their own tools; the MLPerf Tiny v1.4 report lists STEdgeAI-Core, NXP eIQ and AndesAIRE among them. For how these runtimes relate to the frameworks you train in, see our [comparison of AI frameworks](https://computese.com/latest-ai-tools-and-frameworks-a-comparative-analysis/).

> [!TIP]
> Pick the toolchain by the framework your team trains in and the chip you are likely to ship on. A TensorFlow or Keras team starts with LiteRT; a PyTorch team starts with ExecuTorch; a team with more embedded than machine learning experience often gets to a working prototype fastest with Edge Impulse.

## Hardware: from plain Cortex-M to NPUs and Linux boards

Chips for embedded machine learning fall into four steps, and each step up buys speed with power, cost or both.

1. **A plain microcontroller core.** Every Cortex-M can run a small model on its scalar core. The DSP instructions in the Cortex-M4, M7 and M33 speed up the int8 math that CMSIS-NN uses.
2. **A core with vector instructions.** [Arm describes the Cortex-M55](https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u85) as the first Cortex-M with Helium, its vector extension, which the Cortex-M85 also has. Espressif's ESP32-S3 takes the same idea on a different architecture: a dual-core Xtensa LX7 at 240 MHz with 512 KB of SRAM and vector instructions for neural network and signal-processing work, reached through its ESP-NN and ESP-DSP libraries.
3. **A microcontroller with an NPU.** Arm's Ethos-U55, U65 and U85 NPU designs sit beside a Cortex-M core in other companies' chips; the U85 can also pair with a Cortex-A. The Ethos-U85 scales from 128 to 2,048 multiply-accumulate units, from 256 GOP/s up to 4 TOPS at 1 GHz, and adds native support for transformer networks. Other vendors ship their own: in the MLPerf Tiny v1.4 round, ST reported that a preview STM32H7P with its Neural-ART NPU cut image-classification inference time by up to 96% against the same chip's Cortex-M7 alone.
4. **An embedded Linux board.** When the model needs megabytes of activations, several models run at once, or the input is high-resolution video, move to a board such as the Jetson Orin Nano Super, and budget in watts. For chips that compute inside the memory array itself, a research direction rather than a product category, see our explainer on [memristor AI chips](https://computese.com/artifical-intelligence-chip-breakthrough/).

Choose a chip by its measured memory and energy on a model like yours, not by a peak TOPS number. An NPU accelerates only the operators it supports; the rest run on the CPU, and one unsupported layer in the middle of a network can erase much of the gain.

## What MLPerf Tiny measures, and how to read the results

[MLPerf Tiny](https://mlcommons.org/benchmarks/inference-tiny/) is the industry benchmark for this class of device, run by MLCommons with EEMBC. Each task fixes a dataset, a reference model and a quality target, so vendors cannot swap in an easier model to post better numbers:

| Task                                 | Dataset                               | Reference model                      | Quality target                                       |
| ------------------------------------ | ------------------------------------- | ------------------------------------ | ---------------------------------------------------- |
| Keyword spotting                     | Speech Commands                       | DS-CNN (52.5 KB)                     | 90% top-1 accuracy                                   |
| Person detection (visual wake words) | COCO, at 96 by 96 pixels              | MobileNetV1 (325 KB)                 | 80% top-1 accuracy                                   |
| Image classification                 | CIFAR-10                              | ResNet (96 KB)                       | 85% top-1 accuracy                                   |
| Anomaly detection                    | ToyADMOS toy car sounds               | Fully connected autoencoder (270 KB) | 0.85 AUC                                             |
| Streaming wake word                  | Speech Commands with background noise | 1D DS-CNN                            | No more than 8 false positives and 8 false negatives |

The model sizes are the 2021 reference versions from the MLPerf Tiny paper; the streaming task was added in v1.3 in 2025. Each result reports three things: whether the model met the quality target, latency (in the 2021 paper's method, the median of five runs of at least 10 seconds each) and, optionally, energy per inference in microjoules. The streaming wake-word task also measures the device while it sits idle and listens, which is how a wake-word system spends most of its life. The [v1.4 round](https://mlcommons.org/2026/07/mlperf-tiny-v1-4-results/), published in July 2026, had 25 system configurations from nine organizations. MLCommons' own illustration of what the energy figures mean: one accelerator detected a person using 22.2 microjoules per inference, low enough for a CR2032 coin cell to run one inference a second for more than three years.

Read the results with three caveats. Compare the closed division, where everyone runs the same model, when you are choosing hardware; the open division allows different models and shows what is possible, not what you will get. Check whether a submission measured energy at all, since it is optional. And remember that the benchmark times the model only: pre- and post-processing, such as audio feature extraction, are not timed in the original four tasks, so measure your whole pipeline on the device.

## What TinyML is used for

The MLPerf Tiny tasks were chosen because they are the jobs these devices actually do:

- **Keyword spotting and wake words.** The device listens all the time for a short list of words. The benchmark's version recognizes 10 words plus "unknown" and "silence", trained on Speech Commands v2, a set of 105,829 recordings from 2,618 speakers. The paper's examples are the wake words of voice assistants, which is where voice control on [smart home devices](https://computese.com/how-ai-driven-smart-home-devices/) starts.
- **Anomaly detection for predictive maintenance.** Failures are rare and varied, so there is little failure data to train on. The benchmark's answer, and a common one, is an autoencoder trained only on normal operation: it learns to reconstruct normal sound or vibration, and a high reconstruction error flags something new. The MLPerf paper lists sound, vibration, temperature and power as common inputs. The hard part is setting the alarm threshold, which is a business decision about false alarms as much as a technical one.
- **Simple vision.** Is there a person in this 96 by 96 pixel frame? The MLPerf paper ties the task to smart doorbells and occupancy sensing, where the answer is needed without storing or sending images.
- **Motion and gestures.** Accelerometer and gyroscope data from wearables and other devices, classified into activities or gestures, often with spectral features and a small model, as in the case study above.

## The hard parts: data, on-device testing, updates and drift

A model that works on a laptop is the start of the job. The rest is what separates a demo from a product.

### Collect data from the real sensor

Record with the production sensor, mounted where it will be, at the sample rate the firmware will use. The 2022 review lists what goes wrong with sensor data in the field: missing samples, timestamps that drift apart between channels and jitter in the windows the model sees, caused by scheduling delays, clock errors, failing sensors and power limits. Collect the negatives too: background noise, people who are not saying the wake word, a machine that is loud but healthy. Edge Impulse and similar platforms help with labelling and versioning, but no tool fixes data recorded on the wrong microphone.

### Test on the device, not only on the laptop

Run the quantized model on the target board against the same test set you used on the workstation, and measure latency, peak SRAM and energy there. The 2022 review warns that a small model can match its larger original's accuracy and still disagree with it on individual samples, so check which cases changed, not only the headline number. Test the whole pipeline, including feature extraction and the logic that acts on the output, on the real device. Estimates, such as the ones Edge Impulse gives before deployment, are useful for choosing a board; they are not a substitute for this measurement.

### Update models in the field securely

On a microcontroller the model is usually a byte array compiled into the firmware, so updating the model means updating the firmware. The IETF's [firmware update architecture for IoT (RFC 9019)](https://www.rfc-editor.org/rfc/rfc9019.html) sets out the rules: the image must be authenticated and integrity-protected so a modified or unknown image cannot be installed, and the bootloader should keep the previous image so the device can recover when a new one does not boot or work. Chips help: the ESP32-S3, for example, supports secure boot and flash encryption. Because LiteRT for Microcontrollers interprets the model at run time instead of compiling it into code, the 2022 review notes, it is also possible to design a separate flash area for the model and replace it on its own; that update needs the same signature check and rollback.

![A server sends a sealed package over Wi-Fi to a small device, whose bootloader checks the orange seal before writing the package into one of two memory slots while the other keeps the old version.](https://computese.com/images/blog/machine-learning-on-embedded-systems/update.54d72b9720-1536.webp)

*Check the signature before writing, keep the old image until the new one runs: a model update is a firmware update.*

There is now a legal reason to get this right as well. The EU's [Cyber Resilience Act](https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act) was written partly to address the lack of timely security updates in connected products. Its reporting obligations for actively exploited vulnerabilities apply from September 11, 2026, and its main obligations from December 11, 2027, for hardware and software products sold in the EU.

> [!IMPORTANT]
> Design the signing keys, the bootloader, the rollback path and a staged rollout before the first unit ships. An update mechanism added later has to be installed through the very channel it was meant to secure.

The signing key deserves the same care as any production secret; the habits in our [secure coding checklist](https://computese.com/best-practices-for-secure-coding/) apply to the build pipeline that signs firmware.

### Monitor for drift

A deployed model meets data it never saw: a new microphone revision, a machine that wears in, a building with different acoustics. The 2022 review notes that models in the field need periodic fine-tuning for these shifts, and that very small models are especially prone to failing on new data distributions. Since you usually cannot upload raw data, log compact signals instead: how often each class fires, the distribution of confidence scores, and simple statistics of the input such as signal level. When those move away from what you saw in testing, collect new data, retrain, test on the device again and ship through the update path.

## How to start a machine learning project on an embedded device

1. **Write down the decision.** What must the device decide, how fast, how often, and on what battery life? That sets the latency and energy budget.
2. **Record real data** on the production sensor in the real environment, including the negatives and the awkward cases.
3. **Build a small baseline**: good features and a small model. Check its accuracy on the workstation before you touch firmware.
4. **Pick the chip by measurement.** Choose by peak memory and energy on a model like yours. Estimates from your toolchain narrow the list; a development board settles it.
5. **Quantize and test on the device**, with the same test set, and compare the results sample by sample with the float model.
6. **Build the update and monitoring path** before launch: signed images, rollback, staged rollout and the few signals you will watch for drift.

The device is rarely the whole product. Its alerts and readings need an API to receive them, a back end to store them and a portal where people act on them, and that is the part our [custom software development](https://computese.com/services/custom-software-development/) team builds; the firmware and the hardware stay with your embedded engineers or device maker. If the AI you need sits in a business process rather than on a sensor, such as reading documents or sorting requests, our [AI and automation service](https://computese.com/services/ai-automation/) builds those workflows, with an evaluation set run before every model change.

## Key terms
- **TinyML**: Machine learning on microcontrollers and similar devices that draw from under a milliwatt to a few milliwatts, with models usually under 2 million weights.
- **Microcontroller (MCU)**: A single chip with a processor core, flash, SRAM and peripherals, built to control one product. Arm Cortex-M and Espressif ESP32 parts are common examples.
- **Flash and SRAM**: Flash is non-volatile storage that holds the program and the model's weights. SRAM is the fast read-write memory where inputs and intermediate results (activations) live during inference.
- **Quantization**: Storing a model's weights and doing its math at lower precision, usually 8-bit integers (int8) instead of 32-bit floats, so it is smaller, faster and runs on integer-only hardware.
- **Pruning**: Removing weights, channels or filters that contribute little to a model's output, to shrink its size and, when whole structures are removed, its compute.
- **Knowledge distillation**: Training a small student model to reproduce the output probabilities of a larger teacher model, so the small model keeps more of the large one's accuracy.
- **NPU (neural processing unit)**: A hardware block built for the multiply-accumulate math of neural networks. Arm's Ethos-U NPUs sit beside a Cortex-M or Cortex-A core in microcontroller-class chips.
- **Tensor arena**: The fixed block of SRAM that LiteRT for Microcontrollers reserves at start-up and packs every tensor into, instead of allocating memory while it runs.
- **MLPerf Tiny**: MLCommons' benchmark suite for ultra-low-power machine learning, which measures accuracy, latency and energy per inference on fixed tasks and models.
- **Data drift**: A change in the data a deployed model sees, such as a new microphone, a worn machine or a different room, that makes it less accurate than it was in testing.

## Common questions

### What is TinyML?

TinyML is machine learning on microcontrollers and similar devices that draw from under a milliwatt to a few milliwatts. Models are small, typically under 2 million weights, quantized to 8-bit integers and compiled into the device's firmware. Typical jobs are wake words, vibration or sound anomaly detection and low-resolution person detection.

### Can a microcontroller run a neural network?

Yes, if the network is designed for it. The MLPerf Tiny keyword-spotting model is 52.5 KB and its person-detection model 325 KB, and the LiteRT for Microcontrollers runtime fits in 16 KB on an Arm Cortex-M3. Models built for phones or servers do not fit: a standard ResNet-50 needs about 100 times a typical microcontroller's memory.

### Is TensorFlow Lite for Microcontrollers still supported?

Yes, under a new name. Google renamed TensorFlow Lite to LiteRT in September 2024, and its documentation now calls the microcontroller runtime LiteRT for Microcontrollers. The code and many benchmark reports still use the old name, TensorFlow Lite for Microcontrollers (TFLM).

### Can you train a model on a microcontroller?

Generally not. LiteRT for Microcontrollers does not support on-device training, and full training needs several times the memory of inference. Research methods can update a few layers on the device, but the practical pattern is to train on a workstation or in the cloud and ship the new model as an update.

### Should I use a microcontroller or an embedded Linux board?

Use a microcontroller when the model fits in a few hundred kilobytes and the device must run for months on a battery. Use an embedded Linux board, such as a Jetson or a Raspberry Pi, for higher-resolution video, several models at once or generative models, and budget for watts instead of milliwatts.

### How do you update a machine learning model on a deployed device?

On a microcontroller the model is usually compiled into the firmware, so updating the model means a firmware update. Sign the image, have the bootloader verify it before running it, and keep the previous image so the device can roll back, as the IETF's IoT firmware update architecture (RFC 9019) describes.

## Sources
1. [LiteRT for Microcontrollers](https://developers.google.com/edge/litert/microcontrollers/overview), Google AI Edge
2. [Tiny Machine Learning: Progress and Futures](https://arxiv.org/abs/2403.19076), IEEE Circuits and Systems Magazine (arXiv)
3. [Machine Learning for Microcontroller-Class Hardware: A Review](https://arxiv.org/abs/2205.14550), IEEE Sensors Journal (arXiv)
4. [MLPerf Tiny: Benchmarking AI at the Edge](https://mlcommons.org/2026/07/mlperf-tiny-v1-4-results/), MLCommons
5. [ESP32-S3](https://www.espressif.com/en/products/socs/esp32-s3), Espressif Systems
6. [Jetson Orin Nano Super Developer Kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit/), NVIDIA
7. [MLPerf Tiny Benchmark](https://arxiv.org/abs/2106.07597), arXiv (Banbury et al.)
8. [Deployment](https://docs.edgeimpulse.com/studio/projects/deployment), Edge Impulse Documentation
9. [Post-training quantization](https://developers.google.com/edge/litert/conversion/tensorflow/quantization/post_training_quantization), Google AI Edge
10. [Arm Ethos-U Backend](https://docs.pytorch.org/executorch/stable/backends/arm-ethos-u/arm-ethos-u-overview.html), ExecuTorch documentation
11. [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531), arXiv (Hinton, Vinyals and Dean)
12. [MCUNet: Tiny Deep Learning on IoT Devices](https://arxiv.org/abs/2007.10319), NeurIPS 2020 (arXiv)
13. [TensorFlow Lite is now LiteRT](https://developers.googleblog.com/en/tensorflow-lite-is-now-litert/), Google Developers Blog
14. [ExecuTorch Overview](https://docs.pytorch.org/executorch/stable/intro-overview.html), ExecuTorch documentation
15. [Arm Cortex-M Backend](https://docs.pytorch.org/executorch/stable/backends/arm-cortex-m/arm-cortex-m-overview.html), ExecuTorch documentation
16. [About Edge Impulse](https://www.edgeimpulse.com/about), Edge Impulse
17. [CMSIS-NN](https://github.com/ARM-software/CMSIS-NN), Arm (GitHub)
18. [TensorFlow Lite for Microcontrollers: CMSIS-NN kernels](https://github.com/tensorflow/tflite-micro/tree/main/tensorflow/lite/micro/kernels/cmsis_nn), TensorFlow (GitHub)
19. [Ethos-U85](https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u85), Arm
20. [MLPerf Inference: Tiny](https://mlcommons.org/benchmarks/inference-tiny/), MLCommons
21. [RFC 9019: A Firmware Update Architecture for Internet of Things](https://www.rfc-editor.org/rfc/rfc9019.html), IETF
22. [Cyber Resilience Act](https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act), European Commission