You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+10-5Lines changed: 10 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,17 +1,22 @@
1
1
Swift Tokenizers is a streamlined and optimized fork of Swift Transformers that focuses solely on tokenizer functionality. It has no dependency on the Hugging Face Hub: tokenizers are simply loaded from a directory, and downloading is handled separately.
2
2
3
-
The package supports two backends behind the same Swift API:
3
+
Refer to the [Benchmarks](#benchmarks) section to compare the performance of Swift Tokenizers and Swift Transformers.
4
4
5
-
-`Swift` is the default backend. It is pure Swift and requires no Rust artifact.
6
-
-`Rust` is an opt-in backend that uses a prebuilt Rust artifact for faster tokenization and decoding on supported Apple platforms.
5
+
Two backends are available using Swift package traits:
7
6
8
-
Refer to the [Benchmarks](#benchmarks) section to compare the performance of Swift Tokenizers and Swift Transformers.
7
+
|| Swift (default) | Rust (opt-in) |
8
+
|---|---|---|
9
+
| Tokenization | Swift |[tokenizers](https://github.com/huggingface/tokenizers)|
The opt-in `Rust` trait links a Rust binary and excludes the corresponding Swift implementations for even faster performance than the optimized Swift backend.
9
14
10
15
## Package setup
11
16
12
17
Swift Tokenizers uses Swift package traits and requires Swift 6.1 or newer.
13
18
14
-
### Default pure Swift backend
19
+
### Default Swift backend
15
20
16
21
If you add the package without specifying traits, the default `Swift` trait is enabled:
0 commit comments