A vision transformer (ViT) splits an image into small patches, turns each patch into a token, and uses self-attention to relate every patch to every other patch. In 2026, ViTs are less a single model than the default backbone under vision foundation models such as DINOv3, SigLIP 2 and SAM 3. On edge devices, teams usually run distilled or hybrid versions rather than full-size ViTs.
Updated October 2026
What changed in 2026
- Meta's DINOv3 (Aug 2025) trained a 7B-parameter ViT on 1.7 billion unlabelled images, and released smaller distilled ViT and ConvNeXt versions for deployment.
- SAM 3 (Nov 2025) segments every instance of a concept from a text or example prompt, using a shared Perception Encoder ViT backbone.
- Open vision-language models now read images through ViT encoders: Gemma 3 uses a 400M SigLIP encoder and Qwen3-VL uses SigLIP 2.
- NVIDIA's C-RADIOv4 (Jan 2026) distils SigLIP 2, DINOv3 and SAM 3 into one backbone of 412M to 631M parameters.
- Edge-first designs mix convolutions and attention: MobileNetV4 reaches 87% ImageNet top-1 at 3.8 ms on a Pixel 8 EdgeTPU.
What is a vision transformer?
The ViT was introduced by Google researchers in "An Image is Worth 16x16 Words" (October 2020). It showed that a standard transformer, the architecture behind language models, could classify images without convolutions, once it was pretrained on enough data.
A ViT processes an image in five steps:
- Split the image into fixed-size patches, for example 16 x 16 pixels.
- Embed each patch as a vector, so the image becomes a sequence of tokens.
- Add position information, because attention alone does not know where a patch sits.
- Run transformer layers, where self-attention lets every patch weigh every other patch.
- Read out a result: a class label, or patch features that a detection, segmentation or language head uses.
The key difference from a CNN is step 4. A convolution looks at a small neighbourhood and builds up a wider view layer by layer. Self-attention sees the whole image from the first layer, which helps with context but costs more compute as image resolution grows.
Vision transformer vs CNN in 2026
The honest answer to "is ViT better than CNN?" is that it depends on data, hardware and task. Modern CNNs closed much of the gap: ConvNeXt reached 87.8% ImageNet top-1 using only convolutions. Many 2026 edge models are hybrids of both.
| Factor | ViT | CNN |
|---|---|---|
| Built-in assumptions | Few; learns spatial structure from data | Locality and translation built in |
| Data needs | High for training from scratch; low if you start from a pretrained backbone | Lower from scratch |
| Global context | Strong from the first layer | Builds up through depth |
| Compute at high resolution | Grows quickly with token count | Grows more gently |
| Edge accelerator support | Improving; attention ops vary by NPU | Mature on most NPUs and DSPs |
| Foundation model ecosystem | Most 2025 to 2026 backbones are ViTs | Smaller; often distilled from ViTs |
For most teams in 2026 the practical question is not which to train from scratch. It is which pretrained backbone to fine-tune or use frozen, and whether it fits the target device.
ViTs as the backbone of vision foundation models
In 2026 most new vision capability arrives as a large pretrained ViT that many tasks share. You attach a small head for your task, or use the features as they are.
| Model | Owner, date | What it gives you | Edge note |
|---|---|---|---|
| DINOv3 | Meta, Aug 2025 | Self-supervised features; a frozen backbone that Meta reports beats specialised models on detection, segmentation and depth | Distilled ViT and ConvNeXt variants released |
| SigLIP 2 | Google, Feb 2025 | Image-text encoder for zero-shot classification, search and VLMs; multilingual | Used as the vision encoder in several open VLMs |
| SAM 3 | Meta, Nov 2025 | Finds, segments and tracks all instances of a concept from text or example images | About 474M parameters; Meta reports 30 ms per image on an H200 GPU, so edge use needs a smaller encoder |
| C-RADIOv4 | NVIDIA, Jan 2026 | One backbone distilled from SigLIP 2, DINOv3 and SAM 3 | 412M and 631M sizes; a ViTDet mode speeds high-resolution input |
| Gemma 3, Qwen3-VL | Google 2025; Alibaba Dec 2025 | Vision-language models that answer questions about images | Both read images through SigLIP-family ViT encoders |
A concrete example from Meta: the World Resources Institute used DINOv3 to measure tree canopy height in Kenya and cut average error from 4.1 m to 1.2 m compared with DINOv2 (Meta).
Running vision transformers on edge devices
Edge devices such as cameras, gateways, robots and inspection stations (see our guide to computer vision solutions) have fixed power, memory and accelerator limits. Full-size foundation ViTs rarely fit. Teams use four approaches:
- Distil a large ViT into a small model. DINOv3 ships distilled ViT and ConvNeXt students. C-RADIO compresses several teachers into one backbone.
- Use a hybrid architecture. MobileNetV4 combines convolution blocks with a mobile-tuned attention block (Mobile MQA) that its authors report gives a 39% speed-up. Apple's FastVLM (CVPR 2025) uses a hybrid FastViTHD encoder that outputs fewer tokens. It reports up to 85x faster time-to-first-token than LLaVA-OneVision at high resolution, with a 3.4x smaller encoder.
- Cut tokens. Lower input resolution, larger patches or token pruning reduce attention cost, which grows with the square of the token count.
- Quantise and compile for the target chip. INT8 or lower precision and vendor compilers matter more for ViTs than CNNs, because attention and layer-norm support differs across NPUs.
A simple rule: if your accelerator handles attention poorly, a convolutional or hybrid student distilled from a ViT teacher often gives most of the accuracy at a fraction of the latency.
How to choose a vision backbone for an edge project
- Define the task and latency budget, for example "detect defects on a 30 fps line within 20 ms on a Jetson-class device."
- Start from a pretrained backbone that matches the task: DINOv3 or C-RADIO for dense features, SigLIP 2 for image-text search, SAM 3 for open-vocabulary segmentation.
- Benchmark on the real device, not a datacentre GPU. Measure latency, memory and power with your own images.
- Distil or switch to a hybrid if the full model misses the budget.
- Plan updates and monitoring. Backbones change every few months, and camera, lighting or product changes shift accuracy in the field.
Downsides of vision transformers
- They need large pretraining data; training from scratch on a small dataset usually underperforms a CNN.
- Attention cost rises quickly with resolution, which hurts high-resolution inspection and medical images.
- Operator support on edge NPUs is less uniform than for convolutions.
- Large backbones carry licence terms; check each model's licence before commercial use.
Where it's heading: 2026 → 2030
| When | What it looks like |
|---|---|
| Around 2020 | The ViT paper shows a plain transformer can match strong CNNs on image classification after large-scale pretraining. CNNs still dominate production. |
| 2026 (now) | ViTs are the default backbone of vision foundation models (DINOv3, SAM 3) and of VLM encoders. Edge teams deploy distilled or hybrid students (MobileNetV4, FastVLM). |
| By 2028 (expected) | Our view: one distilled multi-task backbone, of the kind C-RADIO points to, serves detection, segmentation and language queries on the same edge device, with small task heads swapped over the air. |
| By 2030 (expected) | Our view: most edge vision runs as a shared encoder feeding both perception and physical AI planning, and the hard work moves from model choice to fleet-wide updates, monitoring and governance. |
Run vision transformers across your edge fleet with NexaStack
NexaStack deploys, updates and monitors AI models, including ViT backbones and their distilled students, across edge sites from one control plane. Teams can roll a new backbone to a subset of cameras, compare latency and accuracy against the current model, and roll back if needed, with each model decision logged and governed.
Frequently asked questions
What is a vision transformer?
A vision transformer is a neural network that splits an image into patches, treats each patch as a token, and uses self-attention to relate patches across the whole image. It was introduced by Google researchers in 2020.
Is a vision transformer better than a CNN?
Not always. ViTs do well when pretrained on large data and when global context matters. CNNs and hybrids are often faster and easier to run on edge accelerators. Modern CNNs such as ConvNeXt reach similar ImageNet accuracy.
Can you give an example of a vision transformer?
The original ViT-B/16 is the classic example. In 2026, widely used ViT-based models include Meta's DINOv3 and SAM 3, Google's SigLIP 2, and the vision encoders inside Gemma 3 and Qwen3-VL.
What are the downsides of vision transformers?
They need large pretraining data, their compute grows quickly with image resolution, and edge NPUs support attention less uniformly than convolutions.
What are the latest vision transformer trends in 2026?
Self-supervised foundation backbones (DINOv3), open-vocabulary segmentation (SAM 3), agglomerative models that merge several teachers (C-RADIOv4), and hybrid or distilled students built for mobile and edge chips.
Test ViT backbones on your own edge hardware
See how NexaStack rolls out, benchmarks and governs ViT backbones and their distilled students on your own edge hardware. Book an edge vision model demo →
