Proposal: support basic update operations without fetching documents first

[marshall007]

There are a few common cases where I think the query language should be extended to support partial updates on documents without having to fetch their existing values first.

Currently such operations require fetching all relevant documents, assigning the new field(s) client-side, and then applying all the updates to the server via .replace(...).

---

.assign([<field>, <value>] | <object>)

Apply updates across multiple documents.

collection.assign(<field>, <value>)
collection.assign({ field: <value> })
collection.findAll(...).assign({ field: <value> })

---

.increment(<field>, <by=1>)

.decrement(<field>, <by=1>)

Increment or decrement values on one or many documents without knowledge of the existing values.

collection.increment(<field>)
collection.decrement(<field>, 3)
collection.find(...).increment(<field>, 2)

[dalanmiller]

I think these should be a definitely included for 2.0.

[deontologician]

Should probably not add them directly on collections though, that's basically asking for people to do something crazy

[mlucy]

Adding it directly on collections would be pretty useful if people change their schema and want to add a field to every document (or remove one).

On assign:

• Is there a way to use assign to remove a field instead of setting it to null?
• Will assign recurse down into objects if you have nested field? (For example, will .assign({foo: {bar: 2}}) transform {foo: {bar: 1, baz: 1}} to {foo: {bar: 2, baz: 1}} or {foo: {bar: 2}}?)
  • My guess would be that recursing down (i.e. semantics like RethinkDB's merge has by default) is usually more useful.
  • Regardless, it might be useful to have some way to switch between the two behaviors (something like r.literal).

On increment/decrement:

• Is there a way to specify that you want to increment/decrement a nested field?

---

This proposal seems a tiny bit worrying, because if you do this over and over again for 4 years you might end up with a language that's pretty close to ReQL in complexity. If it's so annoying to do these operations that you want to add special sugar, that might be an indication that we're missing a primitive.

What if instead we had a command called something like modify, which takes a function and applies it? So something like:

collection.findAll(...).modify(function(doc) {
  doc['foo'] = 'bar';
  doc['baz'] += 1;
  return doc;
})

And then the client would fetch each document, call the function, and store the new value.

If we added a two-phase versioned commit thingy in the client, we could even apply the function atomically by only committing it on the server if the version matches. (Or we could have a separate atomicModify command.)

[deontologician]

A big part of this is performance though, so fetching to the client and running it on every doc is defeating the purpose. I think I agree in principle that it would be nice to have a better primitive, but the most general kind of update is an arbitrary function on the old document. That's too general, since we probably can't enforce permissions effectively on arbitrary functions coming back (mlucy's proposal would be enforcing permissions on the output of the function, which is fine, but requires it to fetch all the documents and run on the client).

So, I think the question is whether there is a better primitive with the constraints:

1. be analyzeable by the permissions system
2. can be converted into a reql function that runs entirely in the database

[marshall007]

Is there a way to specify that you want to increment/decrement a nested field?

@mlucy we could include a single object argument overload to these terms that would let you do that:

collection.find(...).increment({ foo: { bar: 1 } })

---

If it's so annoying to do these operations that you want to add special sugar, that might be an indication that we're missing a primitive.

I think that missing primitive is r.row, in other words the ability to arbitrarily reference other fields in an update. However, if such a term is added I think you inevitably start down the "ReQL complexity" path (math and logic operators would be necessary too, etc) as you suggest. The terms proposed here intentionally side-step that complexity and remain easily analyzable for permissions. I think they us a lot closer to the "handle the first 90% of cases before resorting to custom endpoints" mark.

[danielmewes]

can be converted into a reql function that runs entirely in the database

I think that's impossible with the Horizon permission system as soon as there's any validation function. The round-trip that we could save is the one between the Horizon server and the Horizon client.

I'm a little concerned that this is a premature optimization though.

Another problem with introducing server-side updates is that they are not going to work well with optimistic updates / latency compensation, since the client will not know the final result of the update when issuing it.

[deontologician]

I think that's impossible with the Horizon permission system as soon as there's any validation function

Would rethinkdb/rethinkdb#5813 allow us to do validation in the db?

[danielmewes]

Would rethinkdb/rethinkdb#5813 allow us to do validation in the db?

We discussed this outside of Github, but to mention the gist of it here: That would be difficult right now because validators are JavaScript code and the RethinkDB-side validation functions must be deterministic which r.js in general is not.

[mlucy]

Wait, so is the goal of this proposal to make these operations easier or to improve performance?

If it's the former, I think we should seriously consider something general like modify instead. The performance would be on par with any write operation you can currently do, since as far as I know the only way to do write operations right now is to issue a read followed by a write, so it wouldn't be a step down in that regard.

If it's the latter, you'd ultimately like to just run a ReQL query. Since people usually don't hit performance problems until they're already a decent way into a project, it might be fine to require them to do something slightly complicated like register a chunk of ReQL in their schema that clients could ask for by a particular name.

[marshall007]

Wait, so is the goal of this proposal to make these operations easier or to improve performance?

Both, really. The goal of this proposal was to avoid having to read documents before performing a write (mostly for simplicity, but also for performance on larger operations). The use-cases are the same as those for .update(...) in ReQL. If as a result of validation/permissions checking we're forced to iterate over each document (a la @mlucy's modify), I think we should at least try to perform that iteration on the server rather than round-tripping to the client for each row.