Fix DLQWriteTransform: DLQ file overwrites by adding window token to shard template

[darshan-sj]

This PR fixes a bug in DLQWriteTransform

The Problem

DLQWriteTransform applies a 1-minute FixedWindows before writing to GCS. However, the getShardTemplate() method previously returned a shard template (-SSSSS-of-NNNNN or -P-SSSSS-of-NNNNN) that lacked a window token.

Because the window token was missing, files generated in different 1-minute time windows but assigned to the same shard number were given identical filenames. This resulted in newer files silently overwriting older files in the GCS bucket, leading to a discrepancy between the expected and actual number of DLQ events.

The Solution

This PR adds the -W (window) token to the getShardTemplate() method.

By including the -W token, the DefaultFilenamePolicy now correctly resolves the window start and end times (in ISO-8601 format) and injects them into the filename. This ensures that every DLQ file generated across different time windows has a globally unique name, preventing any overwrites.

Example of the new filename format:
[prefix]-2026-03-10T06:47:00.000Z-2026-03-10T06:48:00.000Z-00005-of-00020.json

GCS Quota Considerations

The addition of the -W token adds exactly 49 bytes to the filename.

• In a Hierarchical Namespace bucket, the base name limit is 512 bytes. Our base name uses ~169 bytes (~33%).
• In a Flat Namespace bucket, the total object name limit is 1024 bytes. Even with a deeply nested directory structure or dynamic streaming paths, the total size remains well under 300 bytes (< 30%).

While this decreases the available filepath/filename length for customers by 49 characters, it is highly unlikely to cause Invalid Object Name errors and is a necessary trade-off to ensure data integrity.

Testing

• Unit Tests: Added testFilesAreWrittenWithWindowToken to DLQWriteTransformTest to verify that the generated filenames.
• Integration Tests: MySQLAllDataTypesBulkAndLiveIT

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 51.80%. Comparing base (36935dc) to head (01f1d8d).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3465      +/-   ##
============================================
- Coverage     54.06%   51.80%   -2.27%     
- Complexity     1937     5893    +3956     
============================================
  Files           536     1029     +493     
  Lines         30502    62342   +31840     
  Branches       3269     6836    +3567     
============================================
+ Hits          16492    32297   +15805     
- Misses        13027    27816   +14789     
- Partials        983     2229    +1246     

Components	Coverage Δ
spanner-templates	71.63% <ø> (-1.12%)	⬇️
spanner-import-export	68.91% <ø> (∅)
spanner-live-forward-migration	80.48% <ø> (+0.12%)	⬆️
spanner-live-reverse-replication	77.87% <ø> (+0.10%)	⬆️
spanner-bulk-migration	88.51% <ø> (+0.07%)	⬆️
gcs-spanner-dv	85.95% <ø> (-0.28%)	⬇️

Files with missing lines	Coverage Δ
...loud/teleport/v2/transforms/DLQWriteTransform.java	97.29% <100.00%> (ø)

... and 517 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical data integrity issue within the DLQWriteTransform by ensuring that Dead Letter Queue (DLQ) files written to Google Cloud Storage (GCS) have globally unique filenames. Previously, files from different processing windows could share the same name, leading to silent overwrites and potential data loss. The changes introduce a window token into the filename generation, guaranteeing that each DLQ file is uniquely identified by its processing window, thereby preserving all DLQ events.

Highlights

• Bug Fix: Resolved an issue in DLQWriteTransform where DLQ files were being silently overwritten in GCS due to identical filenames generated across different 1-minute time windows.
• Filename Uniqueness: Implemented the addition of a -W (window) token to the getShardTemplate() method, ensuring that filenames for DLQ events now include window start and end times, making them globally unique.
• Data Integrity: Enhanced data integrity by preventing silent overwrites of DLQ files, thereby ensuring that all expected DLQ events are retained and accounted for.
• Testing: Introduced a new unit test, testFilesAreWrittenWithWindowToken, to validate the correct generation of filenames with the newly added window token.

Changelog

• v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/DLQWriteTransform.java
  • Modified the getShardTemplate() method to prepend the -W (window) token to the shard template string.
• v2/datastream-common/src/test/java/com/google/cloud/teleport/v2/transforms/DLQWriteTransformTest.java
  • Added a new test method, testFilesAreWrittenWithWindowToken, to verify that DLQ filenames correctly include the window token and match the expected pattern.

Activity

• No human activity has occurred on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩