Skip to content

fix: handle out-of-range in mongo when normalization is false#892

Merged
vikaxsh merged 2 commits intostagingfrom
fix/mongo-year-limit
Mar 26, 2026
Merged

fix: handle out-of-range in mongo when normalization is false#892
vikaxsh merged 2 commits intostagingfrom
fix/mongo-year-limit

Conversation

@vaibhav-datazip
Copy link
Collaborator

Description

MongoDB stores dates as signed 64-bit integers (milliseconds since epoch), which supports years far beyond the standard [0, 9999] range (e.g., negative years for BC or 10000+ for the far-future). Go's time.Time.MarshalJSON strictly requires years to be within [0, 9999], and encountering any value outside this range causes a fatal json: error calling MarshalJSON crash during sync.

failed to send message to writer: failed to write records: failed to create raw data column buffer: failed to marshal data in normalization: json: error calling MarshalJSON for type primitive.DateTime: json: error calling MarshalJSON for type time.Time: Time.MarshalJSON: year outside of range [0,9999]"}

Why this occurred only during normalization: false:

  • When normalization is true, dates are processed by the shared ReformatDate utility which already includes safety bounds.
  • When normalization is false, the MongoDB driver uses a lightweight filterMongoObject function. This function was directly converting primitive.DateTime to time.Time without any guards, allowing unbounded years to hit the JSON marshaler.

The Fix:
Added a safety clamp in drivers/mongodb/internal/mon.go to ensure consistent date handling:

  • Years < 1: Falling back to 1970-01-01 00:00:00 UTC. To keep data consistent as normalization true will make year <1 as epoch start.
  • Years > 9999: Clamped to year 9999 while preserving the month, day, and time components.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Verified by injecting records with extreme dates into a local MongoDB instance and running the sync command:

  • Ancient Underflow: Tested with Year -1 (Date offset -62167219200001 ms).
  • Near-Zero/Spark Limit: Tested with Year 0000 (Date offset -62167219200000 ms).
  • Future Overflow: Tested with Year 10000 (Date offset 253402300800000 ms).
  • Normal values: values which lie between year 0 to 9999

Both normalization: true and normalization: false modes were tested to ensure the sync command completes without any error

Screenshots or Recordings

N/A

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

N/A

@vikaxsh vikaxsh merged commit 1f3553c into staging Mar 26, 2026
11 checks passed
@vikaxsh vikaxsh deleted the fix/mongo-year-limit branch March 26, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants