CSV import

[Cantido]

The main way that I get transactions into my hledger system is using the CSV import feature, since everything exports CSVs, and that’s the one thing I need in Tackler in order to make the switch. Do you have any plans to add a feature like this someday?

[35VLG84]

Hi, nice to hear that you have investigated and considered tackler!

I would love to hear your comments about what are the most important aspects of CSV import functionality to your use cases.

So far there have been discussions about few different aspects of that feature:

1. Converting a CSV row into transaction
2. Detecting duplicates
   a) data level duplicate (same row)
   b) semantically duplicate entry

The first part (1)

This is probably not part of the initial implementation for the tackler core. As there are many tools already to handle CSV files, and every single CSV is semantically different. Also it's super simple to emit valid tackler transactions, as the syntax is very regular and concise. For this reason, there are no ambivalences with tackler's syntax, for example with hledger: decimal marks .

So it's really trivial to emit valid tackler transactions from the conversion program. The complexity is mapping CVS entries to correct accounts, and that's almost always custom logic code, or something where AI/LLM could be utilized.

The output could be organized pretty freely by utilizing transaction sharding.

The second part (2)

To detect data level duplicates (2. a.) (e.g. same CSV row is imported twice) I have had some plans to introduce new metadata field, e.g. data-id, which would have unique constraint over the transaction set. It would be then up to the import logic how this would be unique, for example it could be checksum over the CVS row data.

For 2.b, there isn't any plans to detect semantically duplicate entries at the moment.

Summary

The CVS conversion to accounts and transactions is something which will be outside tackler, but I would be very happy to have contributions for various conversion scripts and host those in tackler repository.

The second part, detecting duplicate import entries in transaction stream is something which will be implemented with high probability.

And as said, I would love to hear your comments and thoughts about this feature.

[Cantido]

I totally understand, it would be a major undertaking since even in hledger it’s a pretty complex feature.

I think the metadata field is a great idea. All the CSVs I would want to import contain a transaction ID.

[Cantido]

Oops I accidentally closed the issue, sorry about that! It won’t let me re-open.

[35VLG84]

I think the metadata field is a great idea. All the CSVs I would want to import contain a transaction ID.

Good to hear, this should be fairly straightforward feature. If you don't mind, I will ping you once I get TEP done (it could take while until it's done).

The TEP is probably just about:

1. ref-id / data-id :: non-empty, single line, newline terminated, trimmed (begin, end) string
2. configuration option to enforce uniqueness

The only complication is that I'm really unhappy how I did the initial refactoring from ANTLR to winnow for metadata. So ideally there is a change to refactor the parser at the same time.

Note: there are couple of different benchmarks for parser refactoring, which can be run with just, all of these will stress also the metadata parsing:

just:
    bench            # Run all benchmarks
    parser-bench     # Run parser benchmarks
    git-bench        # Run git benchmark and tests

[35VLG84]

For the CSV conversion, transaction sharding should be really handy feature. There are pretty much no restrictions how to organize the output (transaction files). You have almost full freedom to organize the transactions and file structure as you wish.