lightsystemdarkgreen
@ e89b7c3raw

encoding_rs

Build Status crates.io docs.rs

encoding_rs an implementation of the (non-JavaScript parts of) the Encoding Standard written in Rust.

The Encoding Standard defines the Web-compatible set of character encodings, which means this crate can be used to decode Web content. encoding_rs is used in Gecko starting with Firefox 56. Due to the notable overlap between the legacy encodings on the Web and the legacy encodings used on Windows, this crate may be of use for non-Web-related situations as well; see below for links to adjacent crates.

Additionally, the mem module provides various operations for dealing with in-RAM text (as opposed to data that’s coming from or going to an IO boundary). The mem module is a module instead of a separate crate due to internal implementation detail efficiencies.

Functionality

Due to the Gecko use case, encoding_rs supports decoding to and encoding from UTF-16 in addition to supporting the usual Rust use case of decoding to and encoding from UTF-8. Additionally, the API has been designed to be FFI-friendly to accommodate the C++ side of Gecko.

Specifically, encoding_rs does the following:

Additionally, encoding_rs::mem does the following:

API surprises to watch out for

The encoder API encodes to the output encoding of the Encoding that methods are invoked on. That is, you can’t use this crate to produce output in UTF-16LE, UTF-16BE, or replacement, since the output encoding for those encodings is UTF-8! (Changing the API to make this more obvious from the API shape itself would now be a semver break.)

Performance

Primary performance attention is given to the x86_64, aarch64, and thumbv7neon targets with the simd-accel and std Cargo features enabled.

Speed is a non-goal when encoding to legacy encodings. By default, encoding to legacy encodings is optimized for binary size rather than speed, which was determined to not be user-visibly slow on Raspberry Pi 3 (which stood in for a slow phone for testing) in the Web-exposed encoder use cases.

See the Cargo features below for optionally making CJK legacy encode faster.

Various aspects of the code other than ASCII acceleration were optimized in 2018 on Haswell using then-current compiler. Alternative approaches might be faster today.

ASCII acceleration was completely rewritten in 2026 targeting the following scenarios (with simd-accel and std enabled):

If you are targeting x86_64, i686, aarch64, or thumbv7neon targets, it’s worthwhile to enable the simd-accel Cargo feature if you can tolerate the reliance on nightly Rust features.

If you are targeting x86_64 and you know that the binaries are only going to be deployed to x86-64-v3 or higher, it’s worthwhile both for run-time performance with simd-accel and for build times (see below) regardless of simd-accel to compile with RUSTFLAGS='-C target_cpu=x86-64-v3' (or higher).

A framework for measuring performance is available separatelygithub.

Rust Version Compatibility

It is a goal to support the latest stable Rust, the latest nightly Rust and the version of Rust that’s used for Firefox Nightly.

At this time, there is no firm commitment to support a version older than what’s required by Firefox. MSRV changes are not treated as semver-breaking.

Furthermore, enabling the simd-accel feature may further constrain the compatible Rust versions.

Currently, the MSRV is 1.88 with or without simd-accel, which is older than Firefox’s MSRV.

Integration with std::io

Notably, the above feature list doesn’t include the capability to wrap a std::io::Read, decode it into UTF-8 and presenting the result via std::io::Read. The encoding_rs_io crate provides that capability.

no_std Environment

The crate works in a no_std environment. By default, the alloc feature, which assumes that an allocator is present is enabled. For a no-allocator environment, the default features (i.e. alloc) can be turned off. This makes the part of the API that returns Vec/String/Cow unavailable.

Decoding Email

For decoding character encodings that occur in email, use the charset crate instead of using this one directly. (It wraps this crate and adds UTF-7 decoding.)

Windows Code Page Identifier Mappings

For mappings to and from Windows code page identifiers, use the codepage crate.

DOS Encodings

This crate does not support single-byte DOS encodings that aren’t required by the Web Platform, but the oem_cp crate does.

Preparing Text for the Encoders

Normalizing text into Unicode Normalization Form C prior to encoding text into a legacy encoding minimizes unmappable characters. Text can be normalized to Unicode Normalization Form C using the icu_normalizer crate.

The exception is windows-1258, which after normalizing to Unicode Normalization Form C requires tone marks to be decomposed in order to minimize unmappable characters. Vietnamese tone marks can be decomposed using the detone crate.

Licensing

TL;DR: (Apache-2.0 OR MIT) AND BSD-3-Clause for the code and data combination.

Please see the file named COPYRIGHTgithub.

The non-test code that isn’t generated from the WHATWG data in this crate is under Apache-2.0 OR MIT. Test code is under CC0.

This crate contains code/data generated from WHATWG-supplied data. The WHATWG upstream changed its license for portions of specs incorporated into source code from CC0 to BSD-3-Clause between the initial release of this crate and the present version of this crate. The in-source licensing legends have been updated for the parts of the generated code that have changed since the upstream license change.

Documentation

Generated API documentation is available online.

There is a long-form write-up about the design and internals of the crate.

C and C++ bindings

An FFI layer for encoding_rs is available as a separate crategithub. The crate comes with a demo C++ wrappergithub using the C++ standard library and GSLgithub types.

The bindings for the mem module are in the encoding_c_mem crategithub.

For the Gecko context, there’s a C++ wrapper using the MFBT/XPCOM types.

There’s a write-up about the C++ wrappers.

Sample programs

Multiversioning

When applicable, UTF-8 validation delegates to the simdutf8 crate. On x86 and x86_64, the UTF-8 validation function is multiversioned (unless the best version, currently AVX2, is known to be usable statically, e.g. by RUSTFLAGS='-C target_cpu=x86-64-v3'). This crate takes care of the dispatch using the core_detect crate without requiring std and without involving the simd-accel feature.

With the simd-accel feature, some other aspects of this crate are multiversioned on x86 and x86_64 in a way that does require std, because the multiversion crate uses std and not core_detect. See below.

Build times

On x86 and x86_64 targets (only), if the std Cargo feature is specified without the simd-accel Cargo feature, the usual proc macro dependencies end up in the dependency graph but aren’t actually used, so it doesn’t make sense to enable the std feature without also enabling the simd-accel feature.

If your binary is only going to be deployed to x86-64-v3 or higher, you can avoid the proc macro dependencies while still getting the performance benefits by enabling simd-accel (with or without std) and specifying RUSTFLAGS='-C target_cpu=x86-64-v3'.

This issue does not apply to non-x86/x86_64 targets.

Furthermore, multiversion 0.9.0 brings syn 3 into the dependency graph while multiversion 0.8.0 brings syn 2 into the dependency graph. If you don’t already have syn 3 but do have syn 2 in your dependency graph, you can downgrade multiversion to 0.8.0 to avoid two versions of syn.

Optional features

There are multiple optional features.

There are remarks about which features are used by Firefox to indicate what configuration is the one the crate author pays the most attention to. While not a Cargo feature, the panic configuration is relevant in this sense. Panic aborts the process in Firefox.

Historically, there have been bugs that have affected non-Firefox configurations but not the Firefox configuration.

alloc (enabled by default)

Enables the parts of the API that deal with String, Vec, and Cow.

Enabled but not actually used by Firefox.

std

When used together with simd-accel (see below), enables run-time detection of AVX2+BMI1 on x86 and x86_64 when the compilation target does not include these target features statically.

This feature has no effect on SIMD capabilities in other scenarios. This feature is not useful without the simd-accel feature.

This feature has the side effect of linking std, so this is not compatible with the no_std context. Unfortunately, even though CPU feature detection uses an instruction rather than syscalls on x86/x86_64, the CPU feature detection check requires std, because the corresponding operation on some other architectures relies on operating system support.

Only some operations are multiversioned: Decoding to UTF-16, encoding from UTF-16 to UTF-8, and various operations in the mem module. Notably, if you don’t use the mem module and don’t use UTF-16 as the in-memory Unicode representation, the std feature won’t be useful.

This feature does not affect multiversioning UTF-8 validation, which uses a separate mechanism.

Used by Firefox.

simd-accel

Enables SIMD acceleration using the nightly-dependent core::simd part of the standard library.

This is an opt-in feature, because enabling this feature opts out of Rust’s guarantees of future compilers compiling old code (aka. “stability story”).

The simd-accel feature has been tested with these target architectures:

With other little-endian target architectures, the code should compute correct results, but performance has not been tuned. With big-endian architectures, core::simd isn’t actually used for now.

On 32-bit ARM, be careful to either use a thumbv7neon target or to explicitly enable neon and also build the standard library with neon enabled. Enabling simd-accel will build without error but will not result in the intended performance when neon isn’t enabled for the compilation or when the stardard library has been built without neon. (The thumbv7neon targets exist to address this issue.)

When simd-accel is enabled, this crate benefits from AVX2+BMI1. When targeting x86_64, if you know that the binaries will only be run on x86-64-v3 or higher, it’s beneficial to compile with x86-64-v3 (or higher) statically enabled. If the binaries need to also run on x86-64-v2 or x86-64-v1 CPUs and linking std is OK, it’s beneficial to enable the std feature (see above) in addition to the simd-accel feature to enable run-time detection of AVX2+BMI1.

As of 2026, the crate is developed on Zen 3 and M3 Pro. The configurations that get the most attention are aarch64 build target running on M3 Pro, x86_64 build target with run-time AVX2+BMI1 detection enabled running on Zen 3, and -C target_cpu=x86-64-v3 running on Zen 3.

Occasional performance checks are made on Skylake. Performance may regress on x86-64-v1 and x86-64-v2.

i686 gets the code shape for x86_64 with minimal checks that it compiles and runs, but the testing is on x86_64 hardware.

For thumbv7neon, testing is Raspberry Pi 4 using the 32-bit version of Raspberry Pi OS.

If you use nightly Rust, you use targets whose first component is one of the above, and you are prepared to have to revise your configuration when updating Rust, you should enable this feature. Otherwise, please do not enable this feature.

Used by Firefox.

serde

Enables support for serializing and deserializing &'static Encoding-typed struct fields using Serde.

Not used by Firefox.

fast-legacy-encode

A catch-all option for enabling the fastest legacy encode options. Does not affect decode speed or UTF-8 encode speed.

At present, this option is equivalent to enabling the following options:

Adds 176 KB to the binary size.

Not used by Firefox.

fast-hangul-encode

Changes encoding precomposed Hangul syllables into EUC-KR from binary search over the decode-optimized tables to lookup by index making Korean plain-text encode about 4 times as fast as without this option.

Adds 20 KB to the binary size.

Does not affect decode speed.

Not used by Firefox.

fast-hanja-encode

Changes encoding of Hanja into EUC-KR from linear search over the decode-optimized table to lookup by index. Since Hanja is practically absent in modern Korean text, this option doesn’t affect perfomance in the common case and mainly makes sense if you want to make your application resilient agaist denial of service by someone intentionally feeding it a lot of Hanja to encode into EUC-KR.

Adds 40 KB to the binary size.

Does not affect decode speed.

Not used by Firefox.

fast-kanji-encode

Changes encoding of Kanji into Shift_JIS, EUC-JP and ISO-2022-JP from linear search over the decode-optimized tables to lookup by index making Japanese plain-text encode to legacy encodings 30 to 50 times as fast as without this option (about 2 times as fast as with less-slow-kanji-encode).

Takes precedence over less-slow-kanji-encode.

Adds 36 KB to the binary size (24 KB compared to less-slow-kanji-encode).

Does not affect decode speed.

Not used by Firefox.

less-slow-kanji-encode

Makes JIS X 0208 Level 1 Kanji (the most common Kanji in Shift_JIS, EUC-JP and ISO-2022-JP) encode less slow (binary search instead of linear search) making Japanese plain-text encode to legacy encodings 14 to 23 times as fast as without this option.

Adds 12 KB to the binary size.

Does not affect decode speed.

Not used by Firefox.

fast-gb-hanzi-encode

Changes encoding of Hanzi in the CJK Unified Ideographs block into GBK and gb18030 from linear search over a part the decode-optimized tables followed by a binary search over another part of the decode-optimized tables to lookup by index making Simplified Chinese plain-text encode to the legacy encodings 100 to 110 times as fast as without this option (about 2.5 times as fast as with less-slow-gb-hanzi-encode).

Takes precedence over less-slow-gb-hanzi-encode.

Adds 36 KB to the binary size (24 KB compared to less-slow-gb-hanzi-encode).

Does not affect decode speed.

Not used by Firefox.

less-slow-gb-hanzi-encode

Makes GB2312 Level 1 Hanzi (the most common Hanzi in gb18030 and GBK) encode less slow (binary search instead of linear search) making Simplified Chinese plain-text encode to the legacy encodings about 40 times as fast as without this option.

Adds 12 KB to the binary size.

Does not affect decode speed.

Not used by Firefox.

fast-big5-hanzi-encode

Changes encoding of Hanzi in the CJK Unified Ideographs block into Big5 from linear search over a part the decode-optimized tables to lookup by index making Traditional Chinese plain-text encode to Big5 105 to 125 times as fast as without this option (about 3 times as fast as with less-slow-big5-hanzi-encode).

Takes precedence over less-slow-big5-hanzi-encode.

Adds 40 KB to the binary size (20 KB compared to less-slow-big5-hanzi-encode).

Does not affect decode speed.

Not used by Firefox.

less-slow-big5-hanzi-encode

Makes Big5 Level 1 Hanzi (the most common Hanzi in Big5) encode less slow (binary search instead of linear search) making Traditional Chinese plain-text encode to Big5 about 36 times as fast as without this option.

Adds 20 KB to the binary size.

Does not affect decode speed.

Not used by Firefox.

Compatibility with rust-encoding

A compatibility layer that implements the rust-encoding API on top of encoding_rs is provided as a separate crategithub (cannot be uploaded to crates.io). The compatibility layer was originally written with the assumption that Firefox would need it, but it is not currently used in Firefox.

Regenerating Generated Code

To regenerate the generated code:

Roadmap

Release Notes

0.8.44

0.8.43

0.8.42

0.8.41

0.8.40

Thanks to everyone who reported bugs and contributed fixes to this release!

0.8.36 though 0.8.39

0.8.35

0.8.34

0.8.33

0.8.32

0.8.31

0.8.30

0.8.29

0.8.28

0.8.27

0.8.26

0.8.25

0.8.24

0.8.23

0.8.22

0.8.21

0.8.20

0.8.19

0.8.18

0.8.17

0.8.16

0.8.15

0.8.14

0.8.13

0.8.12

0.8.11

0.8.10

0.8.9

0.8.8

0.8.7

0.8.6

0.8.5

0.8.4

0.8.3

0.8.2

0.8.1

0.8.0

0.7.2

0.7.1

0.7.0

0.6.11

0.6.10

0.6.9

0.6.8

0.6.7

0.6.6

0.6.5

0.6.4

0.6.3

0.6.2

0.6.1

0.6.0

0.5.1

0.5.0

0.4.0

0.3.2

0.3.1

0.3

0.2.4

0.2.3

0.2.2

0.2.1

0.2.0

The initial release.