v2.8.1: Inline syntax & dynamic offset fix

Changes in this release introduce a new inline syntax that can be used to unpack structs (does not include annotated struct classes) from bytes or a stream:

stream = BytesIO(b"\xff\xff@\xe2\x01\x00")

# To affect the endianess, either use 
O_DEFAULT_ENDIAN.value = LittleEndian
# or creating a constant field that represents the typed struct
uint32le = LittleEndian + uint32

# to read/unpack values from a stream/buffer, just use the left shift operator
value1: int = uint32le << stream 
value2: int = uint16 << stream 
value3: str = String(10) << stream 

# Instead of using the special operator, all default struct classes provide
# wrapper functions for packing and unpacking:
value = uint8.from_bytes(stream)
assert uint8.to_bytes(value) == 0xFF

Note

To implement the from_bytes and to_bytes methods for annotated struct/bitfield/union classes, use the struct_factory.mixin type.

---

Added

• New inline syntax using the << operator to automatically unpack structs from data
• All default struct types now implement from_bytes, from_file and to_bytes for direct packing or unpacking
• Add PackMixin and UnpackMixin to automatically implement the methods mentioned above for struct types

Changes

• pack, unpack and sizeof were moved into their own module (no changed visibe when importing from caterpillar.models)

Removed

• Drop support for 'x' format character in PyStructFormattedField

Fixes

• Pre-computed conditions in the Field class incorrectly evaluated to true or false unconditionally (in case of a context-lambda). This issue has now been fixed in all places.
• Fix some missing or wrong typing annotations
• #57: allow dynamic offsets to be used again

---

New Contributors

• @tsheinen made their first contribution in #57

Full Changelog: v2.8.0...v2.8.1

v2.8.0: Extended Syntax & typing fixes

This release introduces a new way of defining structs while also being typing compliant. Changes made in this release introduce a new syntax named "extended syntax". It allows specifying fields alongside their Python type without breaking the core concept of caterpillar.

The newly introduced type aliases apply a specific naming scheme and always end with _t, so they can be distinguished from normal struct types.

from caterpillar.types import uint32_t # new module
from caterpillar.py import struct, uint32 # struct types

from caterpillar.py import f # extended syntax field specifier
from caterpillar.py import Invisible # field marker to hide them in the cosntructor

@struct
class Format:
    # Annotating fields is the same as before. This time, is is important to note that we can 
    # either use the type directly or specify it manually
    value: uint32_t
    # manual definition
    value: f[int, uint32] # NOTE: no '_t' here as we have to use the struct here
    # to mark fields as invisible to the constructor, use "Invisible"
    end_marker: f[bytes, b"CONSTANT"] = Invisible()

# packing and unpacking is still the same...

Added

• New concept: extended syntax featuring typing compliance; introduces shortcut f to annotate fields and Invisible() to hide fields from the constructor
• struct_factory.mixin and bitfield_factory.mixin both provide default function wrappers for packing and unpacking data as well as typing fixes for operating on types directly.
• New parentctx context path
• Dynamic endian is now used by default (i. e. can be changed via order parameter in pack() or unpack())
• New generic Timestamp class
• A number of new test cases
• New module: caterpillar.types: defines default types that can be used as annotations within struct definitions
• O_DEFAULT_ENDIAN as a global option to set a global default byteorder
• O_DEFAULT_ARCH same concept for arch-like objects
• new 'strict' option to Enum struct
• ATTR_PACK and ATTR_UNPACK to caterpillar.shared
• 'order' and 'arch' options to pack() and unpack(), which temporarily change the global endianess or arch (compatible with Dynamic byteorder)

Changes

• Merge all stub files with their corresponding Python files. All typing is now inline.
• padding struct now has its own dedicated class
• sizeof() now always returns an integer
• ContextPath: dropped support for the call action
• Field.get_name() now always returns a string
• Rename ssize and size to pssize and psize in caterpillar.fields exports
• options.get_flags() always returns a list

Fixes

• IntFlag support for Enum structs
• Fix incorrect Sha1 Digest length
• Add missing global exports in caterpillar.py
• Bitfield, Struct and Sequence now respect fields with already configured byteorder
• Prefixed struct now does not require as_field=True when calling pack() or unpack()
• #56

---

Full Changelog: v2.7.0...v2.8.0

v2.7.0: Dynamic Endian

This release adds a new feature/concept: dynamic byte order

Supported concepts:

• Root context resolution using CTX_ORDER
• Context key lookup using a string or callable
• Runtime resolution using a function
• Per-field byte order overrides using addition syntax

Example:

@struct(order=Dynamic)
class Format:
    a: uint16
    b: uint32

pack(obj, **{CTX_ORDER: BigEndian})

Please refer to the advanced tutorial in the documentation for more information.

Fixes

• This pull request also fixed parsing of MAC addresses and mixed endian configuration within a struct definition

---

What's Changed

• [DEV] Dynamic Endian by @MatrixEditor in #52

Full Changelog: v2.6.3...v2.7.0

v2.6.3: Stub file fixes for type hints

What's Changed

• [FIX] Various stub fixes by @MatrixEditor in #51

Full Changelog: v2.6.2...v2.6.3

v2.6.2: Python 3.10+ support and 1-length array fix

• Fix 1-length sized arrays defined with Field objects
• Add backwards compatibility with Python 3.10
• CAPI is only compatible with Python >=3.12

---

What's Changed

• [FIX] Fix field behavior on 1-length arrays and add compatibility with Python 3.10 by @MatrixEditor in #50

Full Changelog: v2.6.1...v2.6.2

v2.6.1: Fix AsLengthRef and switch in Field

[2.6.1] - Hot-Fix Release

Fixed

• caterpillar.fields._base

  • Fix switch functionality within the Field class for ContextPath objects

• caterpillar.fields.common

  • fix % operator for the AsLengthRef class.

---

What's Changed

• [FIX] Fix AsLengthRef and switch functionality in Field by @MatrixEditor in #49

Full Changelog: v2.6.0...v2.6.1

v2.6.0: AsLengthRef and CAPI compatibility changes

This release targets a complete revamp of the CAPI making it compatible with the existing Python code base. Additionally, a new struct has been implemented:

AsLengthRef

In some situations the length of a payload/string is stored either in front of that string or with some other fields in-between. Let's consider the following example:

@struct
class Format:
     length  : uint16
     flags   : uint32
     payload : Bytes(this.length)

This could be represented by a AsLengthRef field:

@struct
class Format:
    _length : uint16 % AsLengthRef(name="_length", target="payload") = 0
    flags   : uint32
    payload : Bytes(this._length)

This way, the length is populated automatically in the current context.

Changelog

Added

• caterpillar.fields._base

  • Add support for non-context lambda switch fields.

• caterpillar.fields.common

  • Add compatibility support for CAPI atoms in Int, UInt and PyStructFormattedField.

• caterpillar.options

  • Add custom Flag.__eq__ implementation to support equality check with c_Option objects.

• caterpillar.abc

  • Add new typing-only _OptionLike protocol.
  • Add missing _SupportsType protocol to the stubs file.
  • Add new method get to the _ContextLike protocol.

• caterpillar.context

  • Add new option O_CONTEXT_FACTORY that controls the global context type.
    Value must be a method or another type implementing the _ContextLike protocol.
  • Add new global context path: root (exported as G in shortcuts).

• caterpillar.shortcuts

  • Add new shortcuts C for ctx, P for parent and G for the root context as ContextPath objects.

• CAPI

  • New index assignment system when generating CAPI code — a running number is now applied instead of a hard-coded index.
  • Add complete Context implementation in C (c_Context) that conforms to the _ContextLike protocol.
  • Add Atom for C-based struct-like classes (previously known as catom).
  • Add native support for __bits__ in Atom.
  • Add special class LengthInfo for packing or unpacking multiple objects.
  • New builtin atoms (CAPI): Repeated, Conditional and Switch.
  • Add new shared objects and exception types to the native implementation (Cp_ContextFactory, Cp_ArrayFactory, CpExc_Stop and Cp_DefaultOption).

---

Changed

• caterpillar.fields._base
  • Rework Field implementation to reduce overhead when packing and unpacking elements.
  • Use pre-computed states instead of calculating everything on the fly; states will be adjusted automatically when setting new values (via @property attributes).

---

Fixed

• caterpillar.fields.common
  • Fix issue in Prefixed that occurred when the internal struct packs a sequence of elements.

---

Removed

• CAPI
  • Remove old CAPI and completely revamp the CAPI concept to make it compatible with the Python equivalent.

---

Related Issues and Pull requests

• [DEV] (2.6.0-rc) CAPI - Endian and Arch by @MatrixEditor in #41
• [DEV] (2.6.0-rc) CAPI - c_Option (Flag counterpart) by @MatrixEditor in #43
• [DEV] (2.6.0-rc) CAPI Context by @MatrixEditor in #44
• [DEV] (2.6.0-rc) CAPI Atom and LengthInfo by @MatrixEditor in #45
• [DEV] (2.6.0-rc) CAPI Builtin-Atoms by @MatrixEditor in #47
• [DEV] (2.6.0-rc) New Struct: LengthFor / AsLengthRef by @MatrixEditor in #46
• [DEV] (2.6.0-rc) Documentation updates by @MatrixEditor in #48
• [DEV] (2.6.0-rc) CAPI Compatibility by @MatrixEditor in #42

Full Changelog: v2.5.1...v2.6.0

v2.5.1: Hotfix for packing and unpacking Bitfields

What's Changed

• [HOTFIX] Bitfield packing and unpacking by @MatrixEditor in #39

Full Changelog: v2.5.0...v2.5.1

v2.5.0: New Bitfield concept and Stub files

What's Changed

The code of all modules were refactored to include as little type hints as possible now. All typing related information should be taken from .pyi files.

caterpillar.abc

• Removed _Action and split into two separate Protocols _ActionLike := _SupportsActionUnpack | _SupportsActionPack
• Renamed _Switch to _SwitchLike
• Removed _EnumLike
• Added two new protocols: '_SupportsBits' and '_ContainsBits'
• The following attributes and methods were moved into caterpillar.shared: STRUCT_FIELD -> ATTR_STRUCT, hasstruct, getstruct and typeof
• Removed unused '_ContextPathStr'
• Removed __type__() requirement from '_StructLike'
• Added new protocol: _SupportsType

caterpillar.byteorder

• Moved 'BYTEORDER_FIELD' to caterpillar.shared as ATTR_BYTEORDER

caterpillar.shortcuts

• Shortcuts now include typeof, to_struct, hasstruct, getstruct and sizeof

caterpillar.shared

• New constants moved from other modules: ATTR_BYTEORDER, ATTR_TYPE, ATTR_BITS, ATTR_SIGNED, ATTR_TEMPLATE

caterpillar.context

• New context attribute: '_root' can be set to point to the root context instance. Internally, instead of a for-loop that iterates through parent context instances, a simple self.get(...) call is made. (see #35)

caterpillar.model._base

• Fixed an issue when parsing union objects with an unbound stream object
• Fixed an issue where field options defined in Sequences were not populated when creating fields.

caterpillar.model._struct

• sizeof now checks if the provided object implements the _SupportsSize protocol

caterpillar.model._bitfield

• Completely reworked the bitfield mechanism to make it even more powerful. For details refer to #34.
• Fixed an issue where field options defined in BitFields were not populated when creating fields.
• Moved BITS_ATTR and SIGNED_ATTR into caterpillar.shared
• Removed unused getformat function

caterpillar.fields.common

• Transformer: removed __fmt__ method

caterpillar.fields.compression

• Updated public compression methods to use lazy imports

Documentation

• Updated library documentation to reflect changes made to function signatures
• Updated the reference to explicitly state the Protocols defined in caterpillar.abc
• Created a change log to reflect changes made up to v2.5.0

---

Pull Requests

• [DEV] (2.5.0 rc) Python type hints by @MatrixEditor in #30
• [DEV] (2.5.0 rc) Bitfield revamp by @MatrixEditor in #34
• [DEV] (2.5.0-rc) Updates to resolving the root-context by @MatrixEditor in #35
• [DEV] (2.5.0-rc) Documentation Updates by @MatrixEditor in #36
• [DEV] (2.5.0-rc) Cleanup and Improvements by @MatrixEditor in #37
• [DEV] (2.5.0) Stub files and various improvements by @MatrixEditor in #32

Full Changelog: v2.4.5...v2.5.0

v2.4.5: Digest alternative for Python 3.14+

Example use:

@struct
class Format:
    # regular fields
    a: uint32
    b: uint16
    
    # hash start action with target name
    _hash_begin: DigestField.begin("hash", Md5_Algo)

    # hash-wrapped fields
    user_data: Bytes(10)

    # hash field (make it optional to avoid errors in instantiation)
    hash: DigestField("hash", Bytes(16))
    # you can also use the Md5_Field directly
    # hash: Md5_Field("hash")

Changes

• Updated documentation to include Digest
• New classes: DigestField and
• Added new parameter "both" to Action class
• Fixed protocol class of _SupportsUnpack

---

Full Changelog: v2.4.3...v2.4.5