DB merging

[kshpytsya]

I am looking for a simple KV storage crate to use as a backend for persistent cache for a CLI tool I am currently developing. The cache is per user stored in location defined by cache_dir. Specifically, I need to cache some data (e.g. content hash) extracted from files, keyed by FileId. Extraction is performed in a multithreaded fashion assisted by rayon. Entries are either added or modified (in case of mtime or size change).

Mostly, the tool will not be executed concurrently, however I do not to preclude that possibility and want to make it reasonably efficient. To avoid blocking the second process for long time I envision having a process first open two DBs: the main one open for reading and a temporary one open for writing that will receive results for cache misses. After the scan, a quick merge of the temporary DB into the main one would be performed. In the unlikely case that the two processes would be processing overlapping set of files, they would just do a double work.

What would be an efficient way to merge one fjall DB into another? Would it make sense to use keyspaces for temporary DB instead of a separate DB?

[marvin-j97]

If you need concurrent access to a database, you can not simply open a database from multiple processes. That is explicitly not supported because allowing multi process makes many other things virtually impossible. Also opening a database again and again is wasteful, it's mostly tailored towards a stateful process that you could expose via RPC, Unix Domain Socket or similar.

SQLite and LMDB do support multi process access with numbers of gotchas.