Add C-API Capsule

[Vizonex]

Warning

This pull request is a still a being worked on and is based off previous work I did for a different library based off #958

I have not added a whole lot yet in order to be sure that this is what other maintainers had in mind otherwise I will simply wait for input. It should be noted that this concept I had was based off unfinished work I had previously done for another library but works nevertheless.

Important

It should also be noted that getting this implemented will take time and planning on what the best approach would be to take and to do it all slow and steady. I have not implemented more objects besides Fields and Factories because I wanted to ensure that what gets added makes sense and is in the best interest of the maintainers in order to not make the process anymore stressful than it may already be to maintain.

Edit: Here is the objects I am planning to add to the C-API

Checklist

• Field
• Factory
• Ext types
• Raw Types
• Attempting to Subclass StructMetaType Safely in C (Crashes for me upon attempting to bind at PyType_Type.tp_new(...))
• StructMetaTypes (I'm planning to subclass SturctMeta to implement an SQLModel inspired class and Instrument attributes via subclassing Field Types (I'll make a PR for subclassing fields again since I closed the previous one))
• Struct Types
• Encoder/Decoder Types (Tools for extra customization would be nice)
• 1:1 Msgspec C Structures but renamed as Msgspec... Example: Field is named MsgspecFieldObject in msgspec.h so when someone compiles the code something Generic like Field (which could theoretically be used in a different C library or another Dev's implementation) it isn't being mixed and causing developers headaches and compilation issues.
• Cython Bindings

fixes #958

[Vizonex]

I'm going to try working on encoders today for a bit as well as planning out how users should subclass the encoder objects possibly using the C Structure and build around it as well as the writing in of bytes & bytearray types.

[jacopoabramo]

It would be nice if you could provide some documentation on how to use this feature. I'm surely curious myself

[Vizonex]

It would be nice if you could provide some documentation on how to use this feature. I'm surely curious myself

@jacopoabramo I got around to making the _testcapi module it should however be disabled on compiling for distribution It's based on my previous work from multidict but with a few improvements from it (mainly macros for adding the functions to the test module).

[Vizonex]

I'm going to fix test_field_new when I get back home from work today. But this is looking good so far...

[Vizonex]

@jacopoabramo I added Ext and StructMeta StructMeta Still needs testing but the main functions I wanted to add in are now complete should be obvious as to why I am including it but it all boils down to other SQL ORM libraries being slow and I have need to make my own using this library. I'll try coming up with a test example sometime later today when I get time off work to show how a user could go about using this new api in the _testcapi module.

[Vizonex]

I'm a little bit stumped on figuring how how to properly subclass StructMeta , everytime I've been attempting to bind it. It decides to crash as soon as it gets to PyType_Type.tp_new(...). A quick rundown of a pure Python Version which would be the following that I am trying to get to work with C for testing.

class TableMeta(StructMeta):
    __struct_fields__: ClassVar[tuple[str, ...]]
    __struct_config__: ClassVar[StructConfig]
    __match_args__: ClassVar[tuple[str, ...]]

    # Idea I had after working with SQLModel for a very long time...
    __table__: ClassVar[str | None]

    def __new__(cls, name: str, bases: tuple[type, ...], namespace: dict[str, Any], /, **kwds) -> None:
        table = kwds.pop("table", None)
        new_cls = StructMeta.__new__(cls, name, bases, namespace, **kwds)
        if table is not None:
            if isinstance(table, str):
                new_cls.__table__ = table
            elif isinstance(table, bool):
                if table is True:
                    new_cls.__table__ = new_cls.__name__.lower()
                else:
                    new_cls.__table__ = None # ignore as table is simply Abstract...
        else:
            new_cls.__table__ = None
        return new_cls

class Table(metaclass=TableMeta):
    pass

class MyTable(Table, table=True):
    pass

class NamedTable(Table, table="My Named Table"):
    pass

[Vizonex]

Today I will attempt to resume this PR and I think I have an idea about how to approach adding new variables to StructMeta without it crashing involving the subclass of StructMixin perhaps?

[ofek]

Did you manage to make it work?

[Vizonex]

No I am having difficulties with subclassing the metaclass and making something with it in C without it crashing on me. If someone could figure out how to subclass StructMeta without the code crashing feel free to let me know. I have been mostly stumped at this point in time.

[Vizonex]

@ofek Good News I found a suitable solution for my problem

EDIT: (Bad news: Segfaults on linux on Windows on the otherhand it works)

from msgspec import _testcapi
from msgspec import StructMeta, Struct

class Table(Struct, _testcapi.TableMixin, metaclass=_testcapi.TableMeta):
    pass

class XY(Table, table=True):
    x:int
    y:str

t = XY(4, 'x')
print(t.__table_name__)
print(t.__abstract_table__)

This code now runs successfully.

[jacopoabramo]

@Vizonex could you run locally the pre-commit hooks? it would probably re-trigger the CI so it can evaluate if tests pass or not.

it's a really could feature to have in the future this one

[Vizonex]

@Vizonex could you run locally the pre-commit hooks? it would probably re-trigger the CI so it can evaluate if tests pass or not.

it's a really could feature to have in the future this one

@jacopoabramo
Sure, ^^ I recently wiped my computer of vscode related bulk which allowed me to gain 20GB of space so yeah I'll use pre-commit next time you see an update from this PR. I'm going to start working on my msgspec-sqlite3 wrapper so that I know what else I need to add in and implement on this end.

After that I'll document all the functions I've added and update the documentation if I get any bit of extra downtime.