Frozen field is exporting as {}, instead of null

[absurdfarce]

Note: This issue is identical to DAT-648 from the older DataStax OSS JIRA. Moving to Github Issues in an attempt to consolidate all work here.

Description from the original ticket follows. This issue was originally reported by Steve Lacerda.

We have an issue where the customer has data like this:

The properties_frozen field is null. But, when it’s unloaded via dsbulk, it turns into {}:

tenant_name,driver_id,creation_time,description,properties,properties_frozen,state,tenant_id,update_time

ota-cst-lab,1,2023-04-05T22:01:20.473Z,SMPP Channel 1 to sms-router-0,"{"sms.channel.command.retry.number":"1","sms.channel.command.retry.period":"50","sms.channel.destination.host":"sms-router-0.sms-router","sms.channel.destination.port":"8181","sms.channel.login.name":"otacs","sms.channel.login.password":"otacs","sms.channel.protocol":"SMPP","sms.channel.stack.size":"5000","sms.channel.timeout.delay":"2000","sms.channel.window.size":"30"}",{},ENABLED,NULL,2023-04-05T22:01:20.473Z

I’ve tried unloading with several different params:

--codec.nullStrings "NULL, NIL" --schema.nullToUnset false --schema.allowMissingFields true

Everything I tried still pushed {} into the frozen field, instead of null.

The load is fine if the value is null, it’s just the unload.

I asked if we could just modify the csv file, but that’s not an option because they need to load hundreds of tables.

Here’s the schema for the table:

tenant_name ascii,
id ascii,
app_structure_changed boolean,
app_structure_delta ascii,
creation_date timestamp,
eid ascii,
file_structure_changed boolean,
files_delta ascii,
gp_state ascii,
org_groups frozen<set>,
other_ids map<ascii, ascii>,
other_ids_frozen frozen<map<ascii, ascii>>,
previous_state ascii,
profile_name ascii,
properties map<ascii, ascii>,
properties_frozen frozen<map<ascii, ascii>>,
subscription_state ascii,
subscription_state_change_date timestamp,
tenant_id ascii,
update_date timestamp,
PRIMARY KEY ((tenant_name, id))

Additional follow-up from Steve:

CQLSH COPY TO seems to work and preserves the null instead of forcing {}.

[absurdfarce]

Huh, that’s kinda interesting. This against C* 4.0.2:

CREATE TABLE foo.baz (
i int PRIMARY KEY,
j text,
k map<ascii,ascii>,
l frozen<map<ascii,ascii>>
);
insert into foo.baz (i,j,k,l) values (1,'one',{},{});
insert into foo.baz (i,j,k,l) values (2,'two',null,null);

$ ccm node1 cqlsh
Connected to cass-402-1 at 127.0.0.1:9042
[cqlsh 6.0.0 | Cassandra 4.0.2 | CQL spec 3.4.5 | Native protocol v5]
Use HELP for help.
cqlsh> select * from foo.baz;

i | j | k | l
---+-----+------+------
1 | one | null | {}
2 | two | null | null

(2 rows)

[absurdfarce]

Yup, successfully repro’d. All of this against C* 4.0.2 with dsbulk 1.11.0:

CREATE TABLE foo.baz (
i int PRIMARY KEY,
j text,
k map<ascii,ascii>,
l frozen<map<ascii,ascii>>
);

insert into foo.baz (i,j,k,l) values (1,'one',{},{});
insert into foo.baz (i,j,k,l) values (2,'two',null,null);
insert into foo.baz (i,j,k,l) values (3,'three',{'3': 'three'},{'3': 'three'});

$ bin/dsbulk unload -k foo -t baz 2> /dev/null
i,j,k,l
3,three,"{"3":"three"}","{"3":"three"}"
1,one,{},{}
2,two,{},{}
$ bin/dsbulk unload -k foo -t baz -c csv 2> /dev/null
i,j,k,l
1,one,{},{}
2,two,{},{}
3,three,"{"3":"three"}","{"3":"three"}"
$ bin/dsbulk unload -k foo -t baz -c csv --codec.nullStrings="NULL" 2> /dev/null
i,j,k,l
3,three,"{"3":"three"}","{"3":"three"}"
1,one,{},{}
2,two,{},{}

[absurdfarce]

So, that translation is happening at the Java driver:

package com.datastax.driver;

import com.datastax.oss.driver.api.core.CqlSession;
import com.datastax.oss.driver.api.core.cql.ResultSet;
import com.datastax.oss.driver.api.core.cql.Row;

import java.net.InetAddress;
import java.net.InetSocketAddress;
import java.net.UnknownHostException;

public class TestApp {

public static void main(String[] args) {

    InetAddress localhost = null;
    try { localhost = InetAddress.getByName("127.0.0.1"); }
    catch (UnknownHostException uhe) {

        System.out.println("Couldn't find localhost!");
    }

    try (CqlSession session = CqlSession
            .builder()
            .addContactPoint(new InetSocketAddress(localhost, 9042))
            .withLocalDatacenter("datacenter1")
            .build()) {

        ResultSet rs = session.execute("select i,j,k,l from foo.baz");
        for (Row r : rs.all()) {
            System.out.println(
                    String.format(
                            "i: %s, j: %s, k: %s, l: %s",
                            r.getInt("i"),
                            r.getString("j"),
                            r.getMap("k", String.class, String.class),
                            r.getMap("k", String.class, String.class)));
        }
    }
}

}

i: 1, j: one, k: {}, l: {}
i: 2, j: two, k: {}, l: {}
i: 3, j: three, k: {3=three}, l: {3=three}

Matches up to the corresponding Java driver test as well.

[absurdfarce]

The difficulty here is that we need a custom MapCodec that is basically the same as what’s in the Java driver distribution with the exception of some different behaviour here. We can certainly implement such a codec in dsbulk but the problem then becomes how we get it into the configuration. The converting codecs implemented in dsbulk rely on the base Java driver codecs for converting ByteBuffers to Java types, so hypothetically if we could get a custom MapCodec into the registry used there we’d be okay. Problem is that the setup in, say, the unload step is very constrained; it uses only the DefaultCodecRegistry here (by way of this code).

We can very likely add this behaviour to CodecSettings and add an additional codec config val to expose it… that’s probably the cleanest way to make this happen.

[absurdfarce]

Worth mentioning that all the collection codecs (ListCodec, SetCodec and MapCodec) behave in the same way.