0.0.22

Summary

This version clarifies the concept of a Scope, which represents the context of a stream function execution, and adds support for diagnostics to it. Stream functions can now report diagnostics that can be consumed by the runtime and rendered in the CLI, as well as other potential consumers. Moreover, stream function implementations have been updated and made much more flexible by expecting the return of any value that implements IntoResult. This makes it easier to implement infallible stream functions, as well as to use the ? operator with those that are fallible.

Please note that this version includes fundamental breaking changes to the API, which were necessary in order to give module authors maximum flexibility, yet abstracting away the complexity of diagnostics and error handling.

Changelog

Breaking changes

• 5a90a74 zrx-scheduler, zrx-stream – add Diagnostic support to Scope
• 85e4507 zrx-stream – change FlatMapFn to take mutable reference to Scope
• 165c600 zrx-stream – change FilterFn to take mutable reference to Scope
• f78174a zrx-stream – change InspectFn to take mutable reference to Scope
• ef584b9 zrx-stream – change GetFn to take mutable reference to Scope
• 656941f zrx-stream – change DefaultFn to take mutable reference to Scope
• e3e983e zrx-stream – change FilterMapFn to take mutable reference to Scope
• b1fd9fb zrx-diagnostic – remove Report type and impls
• 1f84d32 zrx-scheduler, zrx-stream – remove Deref impl for Scope
• e78d244 zrx-stream – change MapFn impls to return IntoResult
• a1050a5 zrx-stream – change MapFn to take mutable reference to Scope
• a2d5bd5 zrx-scheduler, zrx-stream – improve flexibility of Key method impls
• cefdbc2 zrx-scheduler, zrx-stream – rename Scope to Key and Scoped to Scope

Features

• fc4165a zrx-stream – switch tracing to use Scope::key in stream functions
• 331d81e zrx-scheduler – add IntoResult conversion trait for step Result
• 1625f36 zrx-scheduler – add Scope::key impl
• 2e88a17 zrx-scheduler – add IntoError conversion trait and impls for step Error
• 204d582 zrx-graph – add Graph::groups to find nodes per key function
• 11ff68a zrx-graph – allow references in Graph::group_sinks and Graph::group_sources
• 8439ace zrx-graph – add Graph::group_sources to find sources per key function
• c66b29b zrx-graph – add Graph::group_sinks to find sinks per key function
• 366d7e9 zrx-graph – add Graph::is_acyclic impl (Kahn's algorithm)
• 30fd302 zrx-id – add Clone impl for Matcher

Bug fixes

• 7798ee3 zrx-scheduler – mitigate cycles in inter-module subscriptions

Refactorings

• dd837b3 zrx-scheduler – add () as default for T in IntoResult
• 16c576c zrx-stream – remove unused Clone trait bounds

0.0.21

Summary

This version fixes several bugs that caused barriers to not be fulfilled correctly, which surfaced during the integration of the new runtime into Zensical, including a race condition in the WorkSharing strategy of the Executor.

Changelog

Features

• ce29f93 zrx-stream – switch Barriers::insert to accept Into<Scope<I>>
• ddcaba2 zrx-stream – add Barriers::len and Barriers::is_empty impls
• 161e35d zrx-stream – add Debug impl for Advance
• 4cfa644 zrx-stream – add Stream::inspect operator

Bug fixes

• c860754 zrx-stream – contained Scope instances in Barrier reset on every Event
• 538d026 zrx-stream – early return from Barries::notify on existing Scope
• 6fa7285 zrx-stream – add values to emitted items of Stream::select
• d3f2e56 zrx-stream – unnecessary must_use attributes on Stream operator functions
• 35a7178 zrx-scheduler – Actions dependency graph can contain self-loops
• ad6b3ad zrx-executor – race condition in WorkSharing strategy pending task computation

Refactorings

• 664504d zrx-scheduler – queue Event submission and drain on Schedule::submit

0.0.20

Summary

This version adds an efficient Barrier implementation to zrx-stream, which is a building block for complex operators that allow to perform fine-grained synchronization between streams, like Stream::select. Additionally, function traits for use in Stream operators have been refactored to be more flexible and easier to use, removing the need for newtype wrappers by leveraging specialized trait implementations for common function signatures.

Highlights

• Add Barrier implementation to zrx-stream for efficient synchronization
• Allow an Action to express Interest for certain Event types
• Remove Arc from zrx-id, since Scope already uses an Arc internally
• Fix race condition where a Frontier could be terminated prematurely

Changelog

Breaking changes

• 47f3f65 zrx-stream – increase flexibility of operator functions

Features

• 2444eb2 zrx-stream – add Stream::select operator impl
• e2f9337 zrx-stream – add Stream::product operator impl
• 501713e zrx-stream – add Barrier and Barriers impls
• dbb8c6f zrx-scheduler – allow Action to register Interest for Scope lifecycle
• 1bff0be zrx-stream – add KeyFn trait and impls for common signatures
• ae48b18 zrx-stream – add FlatMapFn trait and impls for common signatures
• 05fba90 zrx-stream – add DefaultFn trait and impls for common signatures
• 606184d zrx-id – add Value impl for Id
• 3e5574f zrx-scheduler – add Error conversions for Scope
• ba5a668 zrx-scheduler – add Scope::try_as_id for use in new function shapes

Bug fixes

• f454d16 zrx-scheduler – Frontier terminated prematurely, leading to race condition

Refactorings

• 086db9a zrx-stream – rename KeyFn into GetFn
• a1e23ff zrx-store – split Iter impls from Slots
• 5e70b79 zrx-scheduler – add IntoIterator impl for Scope
• bfb74b8 zrx-scheduler – simplify Scope by always using an inner Arc
• 6dcc682 zrx-id – remove Arc in Id and Selector to reduce indirections

0.0.19

Summary

This version ships an extensive refactoring of zrx-store, which is fundamental for efficient and optimal barrier management. The most notable breaking change is the removal of the StoreMut::insert_if_changed method and all of its implementations, which has been merged with StoreMut::insert to only return the prior value if it changed, simplifying the API and improving performance.

Stash has been optimized to iterate over its underlying Slab for optimal performance, and the Items::insert and Items::remove methods now both return whether the operation resulted in a change. The logic of Matches::has_any and Matches::has_all has been inverted for clarity, and Matches::insert has been renamed to Matches::add to better reflect its functionality.

Changelog

Breaking changes

• 13ffd28 zrx-store – remove StoreMut::insert_if_changed, merge with StoreMut::insert
• f26f1d4 zrx-store – change Stash to iterate over Slab for performance
• a74a05b zrx-store – add return flag to Items::insert and Items::remove
• 884a75a zrx-id – invert logic of Matches::has_any and Matches::has_all for clarity
• 0f3f350 zrx-id – rename Matches::insert to Matches::add

Features

• 36b05ff zrx-store – add Drain iterator impl for Items
• 4aa08a3 zrx-store – derive Debug for IntoIter impl for Items
• 04a5ace zrx-store – derive Clone for Stash
• 5c4e93f zrx-store – derive Debug for Iter impls for Store impls
• e952fa0 zrx-id – derive Debug for IntoIter impl for Matches
• 3fa4432 zrx-store – add optimized StoreMut::insert_if_changed impl for Slab
• d3c921c zrx-store – add Stash::key impl for index-to-key lookup
• a10c59a zrx-store – add Stash::slots and Stash::slots_mut iterator impls
• bfbf831 zrx-store – add non-consuming Iter impl for Items
• f0d79a6 zrx-store – add IntoIterator impl for mutable reference to Stash
• 9dc2b85 zrx-store – add IntoIterator impl for mutable reference to Queue

Bug fixes

• 61cf1be zrx-store – Stash::index_mut can't return key, switch to value-only

Refactorings

• dad70cb zrx-store – simplify Drain impl for Items
• b1fe0ff zrx-store – remove unnecessary trait bounds

0.0.18

Summary

This version introduces a Stash data structure, which is fundamentally a Slab with key-to-index mapping. It's essential for implementing efficient barrier and frontier management, allowing for quick insertions and removals. The Stash is designed to be a versatile building block for various use cases, particularly those requiring temporal storage. Moreover, missing iterator implementations for some Store implementors have been added.

Changelog

Breaking changes

• ffb60c9 zrx-store – switch Stash to store (K, V) for key-to-index mapping
• ddb1774 zrx-store – remove superseded StoreDelta trait

Features

• 676a6b6 zrx-store – add StoreRange impl for Stash for range iteration
• 0b30af7 zrx-store – add Stash::get and Stash::remove method impls
• b192ce1 zrx-store – add Items impl to manage sets of Stash items
• 73303a7 zrx-scheduler, zrx-store – add top-level export for Queue and Stash
• 847273f zrx-store – add Stash store impls for temporary storage
• da481ea zrx-store – add ExactSizeIterator impls for Ordered iterators
• 4dbcfa8 zrx-store – add ExactSizeIterator impls for Slab iterator adapters
• 6c8fc69 zrx-store – add IntoIterator impl for Queue
• 4b5cec7 zrx-storage – add Storages::new impl for consistency

Refactorings

• fd4de90 zrx-executor, zrx-id, zrx-scheduler, zrx-storage – rename files containing set impls to set.rs

0.0.17

Summary

This version ships the initial version of the module system and includes a ground up rewrite of zrx-scheduler and zrx-stream to support it. Together with the fundamental changes made to zrx-id, zrx-storage and zrx-graph in prior versions, this is a major milestone in the development of Zensical and the foundation for future work on the system.

Note

Please note that this version does not yet include a public module API. The module system is still in its early stages and its API is subject to change. With the initial release of the module system, we've completed phase 2 of our phased transition strategy, and are entering phase 3 in which we will be porting functionality from MkDocs plugins to Zensical modules.

Module system

The initial version of the module system ships with this release, and represents the conceptual core of Zensical's long-term architecture. The fundamental premise is straightforward: all specific functionality is provided as modules, resulting in a system that is inherently flexible and composable. Users can pick and choose modules according to their requirements, with no unnecessary overhead from functionality they don't use.

Example

How module implementations look in practice:

use zrx::id::selector;
use zrx::module::{Module, Context, Result};

/// Optimize module.
pub struct Optimize {
    optimize_png: bool,
}

impl Module for Optimize {
    /// Initializes the optimize module.
    fn setup(&self, ctx: &mut Context) -> Result {
        if self.optimize_png {
            let stream = ctx.add_stream(selector!(location = "**/*.png"));
            stream.map(|path| optimize_png(path));
        }
    }
}

A few things worth noting about this example:

• Business logic stays pure. The optimize_png function doesn't know anything about streams, scheduling, or other modules – it just takes a path and does its job. The module system handles all the plumbing. This makes it much easier to test and maintain business logic, and also makes it reusable outside of Zensical if needed.

• Configuration is just a struct. Module behavior is controlled through plain fields, with no registration callbacks or lifecycle hooks to implement. This makes it easy to understand and modify module behavior without diving into complex initialization logic. All interaction with the runtime is through the context object, which provides a stable and evolvable API.

• Conditional setup is trivial. Entire parts of the workflow can be enabled or disabled at initialization time, without any special support from the system. When optimize_png is false, the module simply doesn't add any streams to the workflow, and the scheduler won't execute any of its logic.

• The API surface is minimal. Implementing a module means implementing a single method, setup. Everything else follows from the stream and selector abstractions. This makes the API easy to learn and reason about, and also makes it easier for us to evolve it without breaking existing modules.

Architecture

The module system is tightly integrated with the zrx-scheduler, zrx-stream and zrx-id crates. Each module is built from a dedicated workflow, representing a set of stream transformations. Streams can be combined and transformed using operators like joins and maps, a concept borrowed from reactive programming.

Modules can be attached and detached at runtime. The scheduler automatically recomputes the execution plan whenever modules are added or removed, so users can freely add and remove functionality without manual intervention.

Modules can cooperate through typed subscriptions. A module can subscribe to streams from other modules, filtered by type – this is how modules communicate and share work, without shared state or brittle callbacks. Each module runs in isolation from other modules, yet can subscribe to them. Communication is implemented with channels.

Scheduler

The scheduler has been rewritten from the ground up to provide a more flexible and powerful execution model.

Performance

Several fundamental changes were made to improve performance:

• Batch processing. Actions - the work units the scheduler executes, with each stream operator compiled into one - now process items in batches rather than individually, reducing per-item overhead significantly.

• Type erasure at the data structure level. Previously, type erasure was performed on a value level, requiring every value to be individually boxed. The new scheduler erases types at the data structure level instead, improving cache coherence and locality. This applies to inter-module channels as well, where the channels themselves are boxed rather than the values they carry.

• Strategic downcasting. Downcasts are now placed so that the number of downcasts per action execution is constant, rather than linear in the number of inputs or outputs.

Architecture

Each action now has a dedicated store it can manage. Different store shapes are supported - unordered, ordered, and indexed - which is essential for certain types of operators and lays the groundwork for transparent caching, planned for a future release.

Currently, module execution happens on the main thread, with parallelizable work in operators offloaded to a thread pool. Because inter-module communication is implemented with channels and modules are fully self-contained, the architecture is already structured to support offloading modules onto their own threads - a further performance improvement we plan to pursue.

This only scratches the surface of the changes that went into this rewrite, which represents roughly three months of work. The new design is still an early version and will continue to evolve, but the fundamental architecture is in place. We're confident it strikes close to the right balance between performance, ease of use and flexibility.

What's next

We understand the interest in the module system and are working hard to make it available as soon as possible. Yet we want to be confident that the API is stable and well-designed before we release it, so we are taking the time to iterate on it internally before making it public. We will be sharing more details about the module system and its API in the coming weeks, and we are excited to see what the community builds with it once it's available.

Early previews of the module API will be available through Zensical Spark.

Changelog

Breaking changes

• fab1c11 zrx-scheduler, zrx-stream – redesign architecture for performance and modularity
• 75af4dd zrx-graph – remove Graph::to_edge_graph (deprecated in 0.0.6)
• 6179105 zrx-id – move selector module out of matcher one level up
• a8e422e zrx-id – move expression module out of filter one level up

Features

• ff5cab6 zrx, zrx-module – add Module trait and related impls

0.0.16

Summary

This version introduces zrx-storage, a crate that extends zrx-store with a higher-level API for managing collections of data with synchronization between multiple stores. It's intended for storing intermediate state in the scheduler and executor, and for sharing data between different modules and workflows. Moreover, it's the foundation to implement action-level caching. A set of accessors for join- and set-operations on stores is provided as well.

Additionally, several improvements and bug fixes have been made to the zrx-id crate, including better handling of specificity and ordering of expressions.

Highlights

• The Storage and Storages types allow access of multiple stores at once.
• The Specified type adds Specificity ordering to arbitrary values.
• The Executor can now be shared between threads.

Changelog

Breaking changes

• 42e3c02 zrx-store – change Collection::downcast_ref/mut target to Collection

Features

• 1348e83 zrx, zrx-storage – add Storage and Storages types for data store synchronization
• 9e5af24 zrx-executor – switch Strategy in Executor to Arc to share between threads
• 11b62f1 zrx-id – add Default impl for Specified
• ba22db9 zrx-id – add Default impl for Expression
• 499d66b zrx-id – add PartialEq and Eq impls to Expression and Operand
• a817004 zrx-id – add Specified for ordering values by Specificity

Bug fixes

• e092ce4 zrx-id – always order all-zero Specificity first
• 1fd37da zrx-id – Specificity is consumed by Specified::specificity accessor

Refactorings

• aa9c937 zrx-id – replace u32 cast with explicit conversion
• 2522c17 zrx-id – store Character class as raw string slices
• d2ac737 zrx-id – move convert exports to containing modules

0.0.15

Summary

This version introduces Specificity — an ordering and tie-breaking concept for selectors and expressions that allows our runtime to determine which module wins when multiple modules compete for a resource. The idea is borrowed from CSS, where more specific selectors take precedence, and adapted here for globs, selectors, and expressions in ZRX.

Background

ZRX uses selectors to query and subscribe to resources. Selectors can be combined into expressions using ANY, ALL, and NOT operators. Each selector has 6 components, each of which can be a glob pattern:

zrs:<provider>:<resource>:<variant>:<context>:<location>:<fragment>

How Specificity works

Each component contributes to a 4-tuple (A, B, C, L):

• A — literal segments (most specific)
• B — single-wildcard segments (*, ?)
• C — double-wildcard segments (**)
• L — literal character count (tiebreaker)

Tuples are compared lexicographically — higher is more specific. Expressions combine specificities according to their operator: ALL sums them (constraints stack), ANY takes the minimum (only as specific as the broadest arm), and NOT contributes nothing (it filters, but doesn't select).

Examples

zrs:{git,file}:::{docs}:index.md:  # (3, 0, 0, 15)
zrs::::docs:{index,about}.md:      # (2, 0, 0, 12)
zrs:::::index.{md,rst}:            # (1, 0, 0, 8)
zrs:::::{*}:                       # (0, 1, 0, 0)

This is an early alpha — documentation on using specificity in the router and other modules will follow. This release includes the core implementation for selectors and expressions.

Changelog

Breaking changes

• a69615d zrx-graph – replace Graph::with with Graph::adjacent accessor

Features

• b451d42 zrx-id – add ToSpecificity impl for Term and Expression
• a3ea5eb zrx-id – add ToSpecificity impl for Id and Selector
• 38b3ccd zrx-id – add AsRef impl to Id and Selector to obtain Format
• a4b1fd3 zrx-id – add Specificity impl for computing glob ordering
• dd4e3f9 zrx-graph – add Traversal::initial impl to obtain initial nodes

Bug fixes

• 9e6eaa2 zrx-id – computed Specificity for Expression::any not aligned
• 41b3e49 zrx-id – comparison of Id and Selector susceptible to hash collisions

Performance improvements

• dad5ed4 zrx-id – provide dedicated PartialEq impl for Selector
• b2870f5 zrx-id – remove branching in PartialEq impl for Id

Refactorings

• c6fd98e zrx-id – change Specificity computation to non-consuming

0.0.14

Summary

This version includes several improvements to the graph traversal logic, including convergence of traversals, which allows to combine multiple traversals into a single one, as well as several refactorings to improve code clarity and maintainability. Combining traversals is essential for multi-source modules, where traversals are created for each source and then combined and synchronized for execution.

Changelog

Breaking changes

• 6e2a178 zrx-graph – move Graph::empty impl into Graph::default
• 2b23a19 zrx-id – replace IntoExpression with Into<Expression>
• 53cbd20 zrx-executor, zrx-scheduler – make Executor::capacity and Strategy::capacity non-optional
• 12e626f zrx-id – remove Builder::set_* APIs in favor of consuming methods
• 0fb0e31 zrx-graph – rename visitor module into iter

Features

• 8350ec7 zrx-graph – add Traversal::converge impl
• b529775 zrx-graph – add Eq impl for Topology
• fbb9130 zrx-graph – add AsRef impl for Graph
• a84bf3f zrx-graph – add Graph::with and Graph::with_mut operator impls

Bug fixes

• 3679f97 zrx-graph – duplicate initial nodes break Traversal::converge

Performance improvements

• 7f13011 zrx-graph – replace BTreeSet with HashSet in convergence computation

Refactorings

• 565b498 zrx-graph – use CommonDescendants iterator in Traversal::converge
• 3015667 zrx-graph – move Traversal errors into dedicated Error
• 3335e0e zrx-graph – switch to named struct for Topology wrapper
• d23dc3b zrx-id – move Builder construction for Format into Default

0.0.13

Summary

This version fixes a race condition in the computation of running tasks in the executor, which could lead to premature termination of the execution loop. Additionally, it includes various improvements and refactorings across the codebase, such as renaming traits for better clarity, adding missing methods, and optimizing hash computations.

Changelog

Breaking changes

• fb3c836 zrx-id – rename non-consuming TryInto* traits into TryTo*

Features

• 099a9fb zrx-graph – add IntoIterator impl for graph Builder
• 55e9740 zrx-store – add missing StoreMut methods to Collection

Bug fixes

• ac6f0d5 zrx-executor – race condition in task count computation in both strategies
• 2623a30 zrx-id – missing prefix for Default impls of id and selector Builder

Performance improvements

• 0915c4a zrx-id – use ahash for faster precomputing of hashes

Refactorings

• f1c28a7 zrx-executor – simplify Immediate construction methods
• 229fee6 zrx-graph – move Builder instantiation to Default
• d702beb zrx-id, zrx-store – remove elided lifetime on Formatter argument
• 96d4e9a zrx-store – switch Ordered to use BTreeSet for value ordering
• 08c8593 zrx-id – expose Operand and Operator, as they're part of the public API