Skip to content

Commit cfe6d85

Browse files
committed
Update readme
1 parent 6a1f82d commit cfe6d85

1 file changed

Lines changed: 10 additions & 5 deletions

File tree

README.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,22 @@
11
Swift Tokenizers is a streamlined and optimized fork of Swift Transformers that focuses solely on tokenizer functionality. It has no dependency on the Hugging Face Hub: tokenizers are simply loaded from a directory, and downloading is handled separately.
22

3-
The package supports two backends behind the same Swift API:
3+
Refer to the [Benchmarks](#benchmarks) section to compare the performance of Swift Tokenizers and Swift Transformers.
44

5-
- `Swift` is the default backend. It is pure Swift and requires no Rust artifact.
6-
- `Rust` is an opt-in backend that uses a prebuilt Rust artifact for faster tokenization and decoding on supported Apple platforms.
5+
Two backends are available using Swift package traits:
76

8-
Refer to the [Benchmarks](#benchmarks) section to compare the performance of Swift Tokenizers and Swift Transformers.
7+
| | Swift (default) | Rust (opt-in) |
8+
|---|---|---|
9+
| Tokenization | Swift | [tokenizers](https://github.com/huggingface/tokenizers) |
10+
| Chat templates | [Swift Jinja](https://github.com/huggingface/swift-jinja) | [MiniJinja](https://github.com/mitsuhiko/minijinja) |
11+
| JSON parsing | [yyjson](https://github.com/ibireme/yyjson) (C) | [serde](https://github.com/serde-rs/serde) |
12+
13+
The opt-in `Rust` trait links a Rust binary and excludes the corresponding Swift implementations for even faster performance than the optimized Swift backend.
914

1015
## Package setup
1116

1217
Swift Tokenizers uses Swift package traits and requires Swift 6.1 or newer.
1318

14-
### Default pure Swift backend
19+
### Default Swift backend
1520

1621
If you add the package without specifying traits, the default `Swift` trait is enabled:
1722

0 commit comments

Comments
 (0)