Config params to allow extra, empty fields not respected since 1.8.0

[absurdfarce]

This commit (for DAT-627) introduced a change in the ordering of the load workflow. After this change the ConvertingCodecFactory is created before the SchemaSettings.init() occurs. The problem is that the call to CodecSettings.createCodecFactory() which creates the ConvertingCodecFactory includes two params indicating whether extra fields or missing fields should be allowed. These values are extracted from the SchemaSettings instance but note that SchemaSettings hasn't been initialized yet... and SchemaSettings.init() is where values are extracted from the active Lightbend Config and stored in SchemaSettings state.

Upshot here is that SchemaSettings supplied to createCodecFactory() will always have default values... which means the two params will always be false. If a user supplies settings at runtime to allow either extra or missing fields those settings will be included in the active Config but not in the build ConvertingCodecFactory.

This change came in with dsbulk 1.8.0 so this behaviour appears to have been present since that release.

[absurdfarce]

The underlying behaviour can be reproduced easily:

CREATE KEYSPACE ks1
  WITH REPLICATION = { 
   'class' : 'SimpleStrategy', 
   'replication_factor' : 1 
  };
CREATE TYPE ks1.someudt (
   field1 int,
   field2 float
);
CREATE TABLE ks1.table1 (
   col1 int primary key,
   col2 someudt 
);
insert into ks1.table1 (col1, col2) values (1, { field1: 1, field2: 1.0});
insert into ks1.table1 (col1, col2) values (2, { field1: 2, field2: 2.0});
insert into ks1.table1 (col1, col2) values (3, { field1: 3, field2: 3.0});

bin/dsbulk unload -k ks1 -t table1 -c json
# Manually create individual JSON files from the unload output
bin/dsbulk load -url "./../dat646/test_data_json" -k ks1 -t table1 -c json

alter type ks1.someudt add field3 double;

Once you have this setup in place you can fix the underlying issue with the following simple change:

diff --git a/workflow/load/src/main/java/com/datastax/oss/dsbulk/workflow/load/LoadWorkflow.java b/workflow/load/src/main/java/com/datastax/oss/dsbulk/workflow/load/LoadWorkflow.java
index 91200ddd..5bb7bd9d 100644
--- a/workflow/load/src/main/java/com/datastax/oss/dsbulk/workflow/load/LoadWorkflow.java
+++ b/workflow/load/src/main/java/com/datastax/oss/dsbulk/workflow/load/LoadWorkflow.java
@@ -110,7 +110,11 @@ public class LoadWorkflow implements Workflow {
   private Function<Flux<WriteResult>, Flux<WriteResult>> queryWarningsHandler;
   private Function<Flux<Resource>, Flux<Flux<Record>>> checkpointHandler;
 
+  private Config config;
+
   LoadWorkflow(Config config) {
+
+	this.config = config;
 	settingsManager = new SettingsManager(config);
   }
 
@@ -140,7 +144,8 @@ public class LoadWorkflow implements Workflow {
 	engineSettings.init();
 	ConvertingCodecFactory codecFactory =
     	codecSettings.createCodecFactory(
-        	schemaSettings.isAllowExtraFields(), schemaSettings.isAllowMissingFields());
+        	this.config.getBoolean("dsbulk.schema.allowExtraFields"),
+        	this.config.getBoolean("dsbulk.schema.allowMissingFields"));
 	session =
     	driverSettings.newSession(
         	executionId, codecFactory.getCodecRegistry(), monitoringSettings.getRegistry());

Note that this breaks the cyclic dependency discussed in the problem description above.

[khamdocs]

Just encountered this exact issue. In my use case I need to import vast amounts of data, with sparsely populated UDT's. The data size on disk is massively increased if we have to explicitly set nulls for all these values. Would be keen to see this fixed ASAP.

[absurdfarce]

PR up now to address this. I decided to go a slightly different way from what was outlined above for a couple reasons. First, the fix outlined above requires knowledge of the config path literals ("dsbulk.schema.allowExtraFields" etc.) within the workflow def itself... and I don't love that. SchemaSettings is pretty obviously intended to function as a facade and using this literal in this way completely circumvents that facade.

Second, it looks like all the workflows suffer from a similar problem... or at least unload and count do. Fixing this in SchemaSettings should address the issue for all workflows in one shot.