WARP format 1.0

- Update to flatbuffers `25.2.10`
- Add fuzzing targets for type and function `from_bytes`
- Update examples
- Simplify type spec
- Make constraints generic and remove specialized constraint lists
- Space optimizations for type and functions specs
- More tests with greater coverage
- Introduce the concept of a WARP `File` and `Chunk`s
- Make chunk compression configurable
- Make `Type` objects class field unboxed (decreases memory pressure)
- Use standard directory structure for Rust API
- Move tests to `tests` directory for more easy discovery
- Remove almost all uses of `unwrap` (needed for server-side parsing)
- Refactor `TypeMetadata`
- Add `mock` module for easy mocking in tests and examples
- Make `Symbol` space optimized
- Switch to using `.warp` extension to represent general analysis data instead of just signatures
- Add format version to `File` and `Chunk` (allow for breaking changes later)
- Make analysis data (signatures and types) copy on write (See `ChunkHandler` impl's)

This work is being done to allow for networked WARP information and generally to make the WARP format more usable in a wider set of scenarios. After this commit any breaking changes to the format will be held off for 2.0, if that ever becomes a thing.

Author: emesare

.gitignore

@@ -56,4 +56,10 @@ generated/
 **/out
 
 *.vsix
-*.deb
+*.deb
+
+# cargo-mutants
+mutants*
+
+*.warp
+*.sbin

Cargo.toml

@@ -1,28 +1,13 @@
-[package]
-name = "warp"
-version = "0.1.0"
-edition = "2021"
-license = "Apache-2.0"
+[workspace]
+resolver = "2"
+members = [
+    "rust",
+    "rust/fuzz",
+    "warp_cli"
+]
 
-[lib]
-path = "rust/lib.rs"
-
-[dependencies]
-flatbuffers = "24.3.25"
-bon = "2.3.0"
-uuid = { version = "1.11.0", features = ["v5"]}
-rand = "0.8.5"
-flate2 = "1.0.34"
-
-[features]
-default = []
-gen_flatbuffers = ["dep:flatbuffers-build"]
-
-[dev-dependencies]
-criterion = "0.5.1"
-
-[build-dependencies]
-flatbuffers-build = { git = "https://github.com/emesare/flatbuffers-build", features = ["vendored"], optional = true }
+[workspace.dependencies]
+warp = { path = "rust" }
 
 [profile.release]
 panic = "abort"
@@ -31,16 +16,3 @@ debug = "full"
 
 [profile.bench]
 lto = true
-
-[[example]]
-name = "simple"
-path = "rust/examples/simple.rs"
-
-[[example]]
-name = "random"
-path = "rust/examples/random.rs"
-
-[[bench]]
-name = "void"
-path = "rust/benches/void.rs"
-harness = false

LICENSE

@@ -1,4 +1,4 @@
-Copyright 2020-2024 Vector 35 Inc.
+Copyright 2020-2025 Vector 35 Inc.
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

README.md

@@ -68,7 +68,7 @@ The basic block GUID is the UUIDv5 of the byte sequence of the instructions (sor
 
 #### When are instructions that set a register to itself removed?
 
-To support hot-patching we must remove them as they can be injected by the compiler at the start of a function (see: [1] and [2]).
+To support hot-patching, we must remove them as they can be injected by the compiler at the start of a function (see: [1] and [2]).
 This does not affect the accuracy of the function GUID as they are only removed when the instruction is a NOP:
 
 - Register groups with no implicit extension will be removed (see: [3] (under 3.4.1.1))
@@ -85,15 +85,33 @@ For the `x86` architecture the instruction `e8b55b0100` (or `call 0x15bba`) woul
 
 The namespace for Basic Block GUID's is `0192a178-7a5f-7936-8653-3cbaa7d6afe7`.
 
-### Function Constraints
+### Constraints
 
-Function constraints allow us to further disambiguate between functions with the same GUID, when creating the functions we store information about the following:
+Constraints allow us to further disambiguate between functions with the same GUID; when creating the functions, we retrieve extra information
+that is consistent between versions of the same function, some examples are:
 
 - Called functions
 - Caller functions
 - Adjacent functions
 
-Each entry in the lists above is referred to as a "constraint" that can be used to further reduce the number of matches for a given function GUID.
+Each extra piece of information is referred to as a "constraint" that can be used to further reduce the number of matches for a given function GUID.
+
+#### Creating a Constraint
+
+Constraints are made up of a GUID and optionally, a matching offset. Adding a matching offset is preferred to give locality to the constraints,
+for example, if you have a function `A` which calls into function `B` that is one constraint, but if the function `B` is also adjacent to function `A`
+without a matching offset the two constraints may be merged into a single one, reducing the number of matching constraints.
+
+- The adjacent function `B` as a constraint: `(9F188A12-3EA1-477D-B368-361936EEA213, -30)`
+- The call to function `B` as a constraint: `(9F188A12-3EA1-477D-B368-361936EEA213, 48)`
+
+#### Creating a Constraint GUID
+
+The constraint GUID is the UUIDv5 of the relevant bytes that would be computable at creation time and lookup time.
+
+##### What is the UUIDv5 namespace?
+
+The namespace for Constraint GUID's is `019701f3-e89c-7afa-9181-371a5e98a576`.
 
 ##### Why don't we require matching on constraints for trivial functions?
 

about.toml

@@ -3,6 +3,7 @@ accepted = [
     "Apache-2.0",
     "MIT",
     "Unicode-DFS-2016",
+    "Unicode-3.0",
     "OFL-1.1",
     "BSL-1.0",
     "BSD-3-Clause",
@@ -11,5 +12,6 @@ accepted = [
     "NOASSERTION",
     "ISC",
     "Zlib",
-    "OpenSSL"
+    "OpenSSL",
+    "NCSA"
 ]

rust/Cargo.toml

@@ -0,0 +1,36 @@
+[package]
+name = "warp"
+version = "1.0.0"
+edition = "2021"
+license = "Apache-2.0"
+
+[dependencies]
+flatbuffers = "25.2.10"
+bon = "3.6.3"
+uuid = { version = "1.11.0", features = ["v5"]}
+flate2 = "1.0.34"
+itertools = "0.14"
+
+[features]
+default = []
+gen_flatbuffers = ["dep:flatbuffers-build"]
+
+[dev-dependencies]
+criterion = "0.6.0"
+
+[build-dependencies]
+flatbuffers-build = { git = "https://github.com/emesare/flatbuffers-build", rev = "44410b9", features = ["vendored"], optional = true }
+
+[[example]]
+name = "type_builder"
+
+[[example]]
+name = "dumper"
+
+[[bench]]
+name = "type"
+harness = false
+
+[[bench]]
+name = "chunk"
+harness = false

rust/benches/chunk.rs

@@ -0,0 +1,40 @@
+use criterion::{criterion_group, criterion_main, Criterion};
+use warp::chunk::{Chunk, ChunkKind, CompressionType};
+use warp::mock::{mock_function, mock_function_type_class, mock_type};
+use warp::r#type::chunk::TypeChunk;
+use warp::signature::chunk::SignatureChunk;
+use warp::{WarpFile, WarpFileHeader};
+
+pub fn chunk_benchmark(c: &mut Criterion) {
+    let count = 10000;
+    // Fill out a signature chunk with functions.
+    let mut functions = Vec::new();
+    for i in 0..count {
+        functions.push(mock_function(&format!("function_{}", i)));
+    }
+    let _signature_chunk = SignatureChunk::new(&functions).expect("Failed to create chunk");
+    let signature_chunk = Chunk::new(
+        ChunkKind::Signature(_signature_chunk),
+        CompressionType::None,
+    );
+
+    // Fill out a type chunk with types.
+    let mut types = Vec::new();
+    for i in 0..count {
+        types.push(mock_type(
+            &format!("type_{}", i),
+            mock_function_type_class(),
+        ));
+    }
+    let _type_chunk = TypeChunk::new(&types).expect("Failed to create chunk");
+    let type_chunk = Chunk::new(ChunkKind::Type(_type_chunk), CompressionType::Zstd);
+    let file = WarpFile::new(WarpFileHeader::new(), vec![signature_chunk, type_chunk]);
+    c.bench_function("file to bytes", |b| {
+        b.iter(|| {
+            file.to_bytes();
+        })
+    });
+}
+
+criterion_group!(benches, chunk_benchmark);
+criterion_main!(benches);

rust/benches/type.rs

@@ -0,0 +1,45 @@
+use criterion::{criterion_group, criterion_main, Criterion};
+use warp::mock::{mock_int_type_class, mock_type};
+use warp::r#type::class::{StructureClass, StructureMember, TypeClass};
+use warp::r#type::guid::TypeGUID;
+use warp::r#type::Type;
+
+pub fn void_benchmark(c: &mut Criterion) {
+    let void_type = Type::builder()
+        .name("my_void".to_owned())
+        .class(TypeClass::Void)
+        .build();
+
+    c.bench_function("uuid void", |b| {
+        b.iter(|| {
+            let _ = TypeGUID::from(&void_type);
+        })
+    });
+
+    c.bench_function("computed void", |b| b.iter(|| void_type.to_bytes()));
+}
+
+pub fn struct_benchmark(c: &mut Criterion) {
+    let int_type = mock_type("my_int", mock_int_type_class(None, false));
+    let structure_member = StructureMember::builder()
+        .name("member")
+        .ty(int_type)
+        .offset(0)
+        .build();
+    let struct_class = StructureClass::new(vec![structure_member]);
+    let struct_type = Type::builder()
+        .name("my_struct".to_owned())
+        .class(TypeClass::Structure(struct_class))
+        .build();
+
+    c.bench_function("uuid struct", |b| {
+        b.iter(|| {
+            let _ = TypeGUID::from(&struct_type);
+        })
+    });
+
+    c.bench_function("computed struct", |b| b.iter(|| struct_type.to_bytes()));
+}
+
+criterion_group!(benches, void_benchmark, struct_benchmark);
+criterion_main!(benches);

rust/benches/void.rs

@@ -10,11 +10,13 @@ pub fn main() {
     let _ = std::fs::remove_dir_all("rust/gen_flatbuffers");
     let workspace_dir: PathBuf = std::env::var("CARGO_MANIFEST_DIR").unwrap().into();
     BuilderOptions::new_with_files([
-        workspace_dir.join("type.fbs"),
-        workspace_dir.join("symbol.fbs"),
-        workspace_dir.join("signature.fbs"),
+        workspace_dir.join("../type.fbs"),
+        workspace_dir.join("../symbol.fbs"),
+        workspace_dir.join("../signature.fbs"),
+        workspace_dir.join("../target.fbs"),
+        workspace_dir.join("../warp.fbs"),
     ])
-    .set_output_path("rust/gen_flatbuffers")
+    .set_output_path("src/gen_flatbuffers")
     .compile()
     .expect("flatbuffer compilation failed");
 }