Support validation for unspecified dtype

[ehennestad]

Motivation

This is currently allowed: types.core.TimeSeries('data', types.core.TimeSeries())

This PR adds validation for datasets where dtype is omitted in the schema and MatNWB therefore allows any valid dtype. The main goal is to support the HDMF/NWB spec behavior for unspecified dtype while still validating that provided values are valid basic, reference, soft link, or compound values.

It also removes the earlier ad hoc use of dtype as a generator control flag for included typed datasets. Instead, included typed datasets now carry an internal skip_dtype_validation marker so duplicate dtype validation is not generated when the including dataset omits dtype.

How to test the behavior?

types.core.TimeSeries('data', types.core.TimeSeries())

Before:

ans = 

  TimeSeries with properties:

     starting_time_unit: 'seconds'
    timestamps_interval: 1
        timestamps_unit: 'seconds'
                   data: [1×1 types.core.TimeSeries]
              data_unit: ''
               comments: 'no comments'
                control: []
    control_description: ''
        data_continuity: ''
        data_conversion: 1
            data_offset: 0
        data_resolution: -1
            description: 'no description'
          starting_time: []
     starting_time_rate: []
             timestamps: []

After:

Error using types.util.checkDtype>validateAnyType (line 477)
Value was of type "types.core.TimeSeries" but must be one of the following types:
  char
  datetime
  double
  int16
  int32
  int64
  int8
  logical
  numeric
  single
  uint16
  uint32
  uint64
  uint8
  string
  cellstr
  types.untyped.ObjectView
  types.untyped.RegionView
  types.untyped.ExternalLink
  types.untyped.SoftLink
  struct
  table
  containers.Map

Error in types.util.checkDtype (line 26)
        value = validateAnyType(value);
                ^^^^^^^^^^^^^^^^^^^^^^
Error in types.core.TimeSeries/validate_data (line 213)
        val = types.util.checkDtype('data', 'any', val);
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in types.core.TimeSeries/set.data (line 133)
        obj.data = obj.validate_data(val);
                   ^^^^^^^^^^^^^^^^^^^^^^
Error in types.core.TimeSeries (line 101)
        obj.data = p.Results.data;
        ^^^^^^^^

Relevant unittests

nwbtest('Name','tests.unit.types.validators.CheckDtypeTest');

nwbtest('Name','tests.unit.schema.IncludedTypedDatasetValidationTest');

nwbtest('Name','tests.unit.schema.DataTypesTest');

Checklist

• Have you ensured the PR description clearly describes the problem and solutions?
• Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
• If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Codex

[Copilot]

Pull request overview

Adds proper validation for datasets whose schema omits dtype (HDMF/NWB “any dtype” behavior), while preventing duplicate dtype validation generation for included typed datasets.

Changes:

• Introduces “any dtype” runtime validation in types.util.checkDtype and wires it into several generated validate_data methods.
• Refactors dtype mapping to a centralized basic-dtype map (spec.getBasicDTypeMap) and updates codegen to use a skip_dtype_validation flag rather than ad hoc dtype handling.
• Updates/extends unit tests to cover “any dtype”, included typed dataset handling, and nested unwrap/rewrap behavior.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
+types/+util/unwrapValue.m	Adds nested unwrapping behavior for types.untyped.Anon wrappers.
+types/+util/rewrapValue.m	Adds nested rewrapping behavior for types.untyped.Anon wrappers.
+types/+util/checkDtype.m	Adds “dtype=any” validation logic, including compound/reference/link handling.
+types/+hdmf_experimental/HERD.m	Updates expected index dtypes to uint8 in validators.
+types/+hdmf_common/VectorData.m	Adds dtype='any' validation for data.
+types/+hdmf_common/Data.m	Adds dtype='any' validation for data.
+types/+hdmf_common/CSRMatrix.m	Adds dtype='any' validation for data; updates index dtypes to uint8.
+types/+core/TimeSeries.m	Adds dtype='any' validation for data.
+types/+core/ScratchData.m	Adds dtype='any' validation for data.
+types/+core/NWBData.m	Adds dtype='any' validation for data.
+types/+core/BaseImage.m	Adds dtype='any' validation for data.
+tests/+unit/dynamicTableTest.m	Adjusts DynamicTable-related test inputs (shape/type changes).
+tests/+unit/+types/FunctionTests.m	Adds unit test for nested Anon→DatasetClass unwrap/rewrap.
+tests/+unit/+types/+validators/CheckDtypeTest.m	Adds unit tests for dtype='any' accept/reject behavior.
+tests/+unit/+schema/IncludedTypedDatasetValidationTest.m	Adds tests for skip_dtype_validation behavior in schema expansion and Dataset parsing.
+tests/+system/DynamicTableTest.m	Adjusts system test inputs; minor comment wording update.
+spec/getBasicDTypeMap.m	New centralized mapping from HDMF basic dtypes to MATLAB types.
+spec/+internal/expandFieldsInheritedByInclusion.m	Replaces defaulting dtype to 'any' with skip_dtype_validation marker.
+file/mapType.m	Refactors dtype mapping to use spec.getBasicDTypeMap and improves error messaging/docs.
+file/fillValidators.m	Makes dtype validation generation conditional on skipDtypeValidation; removes prior “any means skip” special-case.
+file/Dataset.m	Adds and populates skipDtypeValidation from schema (skip_dtype_validation).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov]

Codecov Report

❌ Patch coverage is 95.31250% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.41%. Comparing base (19239e8) to head (7fa716a).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
+types/+util/checkDtype.m	97.16%	3 Missing ⚠️
+types/+util/rewrapValue.m	77.77%	2 Missing ⚠️
+types/+util/unwrapValue.m	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #816      +/-   ##
==========================================
- Coverage   95.44%   95.41%   -0.03%     
==========================================
  Files         203      203              
  Lines        7261     7368     +107     
==========================================
+ Hits         6930     7030     +100     
- Misses        331      338       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.