Support for realtime web

[JohannesLichtenberger]

Meaning in a first step we should stream the data changes for a resource to interested clients (we could just stream the JSON we generate, which stores a change track).

In a second step we can support queries, which only query one resource. Thus, naively we'd probably have to execute the query/queries asynchronous once a trx has been flushed to disk and check that only result nodes in the curre transaction intent log are streamed to interested clients. Maybe we can store the compiled query at least up until indexes are added/dropped.

Better ideas are of course welcome. A simpler much more efficient approach would be to check for updates on certain paths in the resource.

[JohannesLichtenberger]

How does RethinkDB implement this?

[mosheduminer]

I wonder if looking at https://github.com/pubkey/event-reduce would be helpful.

[JohannesLichtenberger]

Thanks Moshe, do yohh understand the basic algorithm they are using? Maybe it's too late currently, but I don't get how they determine if a new record from a change event is a query result or not.

[mosheduminer]

I don't know, I haven't really looked at the implementation. So I only know what it says in the readme.

[ElenaSkep]

Hello! I am interested in working on this issue. Would it be ok? I was thinking of using webSockets to stream the json. Also could you specify in which path I should focus first?

[JohannesLichtenberger]

You can start by streaming updates to subscribed clients:

sirix/bundles/sirix-core/src/main/java/io/sirix/access/trx/node/AbstractNodeTrxImpl.java

Line 297 in 982a346

serializeUpdateDiffs(preCommitRevision);

Maybe we can add a post commit hook to listen for changes.

[JohannesLichtenberger]

So, I think it would be great if clients can subscribe to a database/resource update stream, thus that they receive what has been changed as a JSON change stream (we already write these changes to JSON files on disk).

So we need to have a simple pub/sub mechanism, checking for read-access rights and to write the changes to the websocket.

[ElenaSkep]

Hello again! @FayKounara and I have made some progress. We have created a pub/sub mechanism using Apache Kafka but it would be helpful if you could specify who is considered an interested client. Also we are using WebSockets and Nginx if its ok.

[JohannesLichtenberger]

@ElenaSkep oh wow, I think for our use case we should however keep it simple and use a non distributed pub/sub mechanism (maybe there' already a solution using Vert.x). Interested clients would for instance be a browser (IMHO it would be great to have a web based GUI which has views to either query or show the diffs between revisions). However, in the general case a Kafka based backend would be nice (but I think I'd implement a new storage solution in the io-package for that (analogous to the FileChannel or MMStorage)).

[JohannesLichtenberger]

So, I think a web client (for instance the TypeScript based client) would subscribe via a WebSocket to a database / resource in the database and subsequently it would receive all changes in the current JSON format. It may be used in the future by a new web frontend to display the differences.

[JohannesLichtenberger]

You'd probably use a simple thread safe blocking queue to handle the subscribers...

[JohannesLichtenberger]

So, in general it should be part of the sirix-rest-api bundle.