View source for Conformer (neural network)

{{Short description|Neural network that combines self-attention with convolution, introduced for speech recognition in 2020}}
The '''Conformer''' is a neural network architecture that puts a convolution module and a self-attention module in the same block. It was introduced for automatic speech recognition by researchers at Google in 2020, in a paper titled "Conformer: Convolution-augmented Transformer for Speech Recognition".<ref name="gulati">{{cite conference |last1=Gulati |first1=A. |last2=et al. |title=Conformer: Convolution-augmented Transformer for Speech Recognition |conference=Interspeech 2020 |pages=5036-5040 |year=2020 |doi=10.21437/Interspeech.2020-3015}}</ref> The name is a contraction of ''convolution'' and ''transformer''. Its encoder has since been used on other one-dimensional signals, among them nanopore reads, electroencephalograms and, in the case of [[Molecular Reality Corporation]], the current through a single [[nanopore]].

== Background ==
Through self-attention, a transformer relates every position in a sequence to every other and so captures global context. A convolutional network looks at a small neighborhood at a time and captures local patterns well and cheaply. The authors say that transformers are good at "content-based global interactions" while convolutions "exploit local features effectively", and set out to combine the two "in a parameter-efficient way".<ref name="gulati-arxiv">{{cite journal |last1=Gulati |first1=A. |last2=et al. |title=Conformer: Convolution-augmented Transformer for Speech Recognition |journal=arXiv |year=2020 |doi=10.48550/arXiv.2005.08100}}</ref> Their conclusion is that including the convolution modules is critical to the performance of the model.

== Architecture ==
A Conformer block contains four modules in a fixed order. A feed-forward module comes first and a second feed-forward module comes last, each with its output added at half weight, a sandwich borrowed from an earlier design called Macaron-Net. Between them sit a multi-headed self-attention module that uses the relative sinusoidal positional encoding of Transformer-XL, and a convolution module. The convolution module has a pointwise convolution that doubles the channels, a gated linear unit, a depthwise one-dimensional convolution, batch normalization, a Swish activation and a second pointwise convolution. A layer normalization follows the block.<ref name="gulati-arxiv" />

The input is a spectrogram, with 80 filterbank channels computed over a 25 ms window every 10 ms. A convolutional layer subsamples it before the blocks. Three sizes were trained on the 970 hours of LibriSpeech, a corpus of read audiobooks.

{| class="wikitable"
|+ Conformer encoders in the original paper
! Model !! Parameters !! Layers !! Encoder width !! Attention heads !! Convolution kernel
|-
| Small || 10.3 million || 16 || 144 || 4 || 32
|-
| Medium || 30.7 million || 16 || 256 || 4 || 32
|-
| Large || 118.8 million || 17 || 512 || 8 || 32
|}

The large model reached word error rates of 2.1% on the standard test set and 4.3% on the harder "test-other" set without a language model, and 1.9% and 3.9% with an external one, which the paper calls a new state of the art, and a 15% relative improvement on test-other with an external language model over the best previous model, a Transformer Transducer. The small model reached 2.7% and 6.3%.<ref name="gulati-arxiv" />

== Other signals ==
Nanopore sequencing was an early destination. A basecaller of 2020 built on the convolution-augmented transformer decoded the raw current of an Oxford Nanopore device into bases, and a basecaller for nanopore RNA in 2024 combined a transformer with gated convolution.<ref name="catcaller">{{cite conference |last1=Lv |first1=X. |last2=et al. |title=An end-to-end Oxford Nanopore basecaller using convolution-augmented transformer |conference=2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) |year=2020 |doi=10.1109/BIBM49941.2020.9313290}}</ref><ref name="gcrt">{{cite journal |last1=Li |first1=Q. |last2=et al. |title=GCRTcall: a transformer based basecaller for nanopore RNA sequencing enhanced by gated convolution and relative position embedding via joint loss training |journal=Frontiers in Genetics |volume=15 |pages=1443532 |year=2024 |doi=10.3389/fgene.2024.1443532}}</ref> For EEG, a network that places convolutional layers ahead of a transformer was published under the name EEG Conformer.<ref name="eeg">{{cite journal |last1=Song |first1=Y. |last2=et al. |title=EEG Conformer: convolutional transformer for EEG decoding and visualization |journal=IEEE Transactions on Neural Systems and Rehabilitation Engineering |volume=31 |year=2023 |doi=10.1109/TNSRE.2022.3230250}}</ref>

A speech model takes one feature vector every 10 ms. The company's live stream records a value every 5 µs, which is 2,000 times faster, and at 100 kS/s an event of 185 µs is about 18 samples long.

== At Molecular Reality ==
In April 2026 the company trained a Conformer of three blocks on barcode events from a public set of [[DNA barcode (nanopore)|DNA barcodes]] read by a solid-state nanopore, and compared it with a retrained QuipuNet CNN on a split that held out whole experiments. The Conformer scored 85.4% against 81.0% for the CNN, a result that is unpublished.<ref name="aiexp">{{cite web |url=https://github.com/EpicQuestBio/ai-experiments |title=ai-experiments |website=GitHub |date=April 2026}}</ref> Its description reads "experiments replicating existing nanopore/speech applications of novel AI methods", and one of its steps reruns, in simplified form, the speech experiment of the Conformer paper.<ref name="aiexp" /> Company records call the model the first Conformer the company knows of on single-molecule sensing traces, a qualifier that matters, since basecallers had used the design before.<ref name="records">{{cite web |title=Company records |publisher=Molecular Reality Corporation |date=October 2026}}</ref>

In the company's plans the Conformer reads the current, a Perceiver fuses current, voltage and flow-cell video, and a [[joint embedding predictive architecture]] supplies a training objective that needs no labels, as laid out in [[The Nanopore World Model]].<ref name="records" /><ref name="wm">{{cite web |url=https://molecularreality.com/nanopore-world-model |title=Keltar's Critique v2: The Nanopore World Model (Hardened) |publisher=Molecular Reality Corporation |date=June 2026}}</ref> Its investor copy puts the case for the whole program in one line: "The machine learning that cracked speech now works on ionic current."<ref name="records" />

== See also ==
* [[Machine learning at Molecular Reality]]
* [[Machine learning on nanopore signals]]
* [[QuipuNet]]
* [[Joint embedding predictive architecture]]

== References ==
{{Reflist}}

[[Category:Neural network architectures]]
[[Category:Machine learning]]
[[Category:Speech recognition]]