Add -f codepage flag for input/output encoding #638

dlevy-msft-sql · 2026-01-26T00:12:41Z

Summary

Implements the -f flag from ODBC sqlcmd for specifying input and output file code pages.

Changes

cmd/sqlcmd/sqlcmd.go: Added CodePage and ListCodePages fields, -f/--code-page flag, validation, and runtime application
pkg/sqlcmd/codepage.go: Comprehensive codepage support with parsing and encoding conversion
pkg/sqlcmd/codepage_test.go: Unit tests for codepage parsing and supported encodings
pkg/sqlcmd/commands.go: Updated :R command to respect codepage settings when reading files
pkg/sqlcmd/sqlcmd.go: Added CodePage field to Sqlcmd struct
cmd/sqlcmd/sqlcmd_test.go: Command line argument tests for valid and invalid codepage values
README.md: Documentation with format examples

Usage

# Single codepage for both input and output
sqlcmd -S server -f 65001 -i script.sql -o results.txt

# Different codepages for input and output
sqlcmd -S server -f i:1252,o:65001 -i windows_script.sql -o utf8_results.txt

# List all supported codepages
sqlcmd --list-codepages

Supported Codepages

Unicode: 65001 (UTF-8), 1200 (UTF-16LE), 1201 (UTF-16BE)
Windows: 874, 1250-1258
OEM/DOS: 437, 850, etc.
ISO-8859: 28591-28606
CJK: 932 (Shift-JIS), 936 (GB2312), 949 (Korean), 950 (Big5)
EBCDIC: 37, 1047, 1140

Testing

All existing tests pass
Added command line argument parsing tests
Added unit tests for codepage parsing and validation

Improves ODBC sqlcmd compatibility.

- Add -f/--code-page flag with ODBC-compatible format parsing - Support 50+ codepages: Unicode, Windows, OEM/DOS, ISO-8859, CJK, EBCDIC, Macintosh - Apply input codepage in IncludeFile() for :r command - Apply output codepage in outCommand() for :OUT file writes - Add --list-codepages flag to display all supported codepages - Add comprehensive unit tests for parsing and encoding lookup

- Fix nil encoding panic in errorCommand when OutputCodePage is 65001 (UTF-8) - Close file handle in outCommand when GetEncoding returns an error - Handle close error properly in errorCommand - Apply UTF-8 BOM stripping when input codepage is 65001 - Fix test subtest names to use strconv.Itoa instead of string(rune)

- Use localizer.Errorf for all user-facing error messages - Fix UTF-16 BOM handling using ExpectBOM for input decoding - Add transformWriteCloser to properly close underlying file handles - Use transformWriteCloser in outCommand and errorCommand for both UnicodeOutputFile and CodePage transforms to prevent file handle leaks - Add integration tests for output/error codepage encoding

Use strconv.Itoa instead of %d to avoid locale-specific thousands separators in error message.

- Create codepageRegistry map as single source of truth for codepages - GetEncoding() now uses the registry instead of switch statement - SupportedCodePages() now generates list from registry - Removes duplicate codepage definitions between the two functions - Sort SupportedCodePages result by codepage number for consistency

- Verify all returned codepages are valid in GetEncoding - Ensure results are sorted by codepage number - Check well-known codepages are present

- Fix IncludeFile to strip BOM from UTF-16 encoded files using BOMOverride - Add Windows API fallback (MultiByteToWideChar/WideCharToMultiByte) for codepages not in built-in registry (e.g., Japanese EBCDIC 20290) - Add helpful error on non-Windows when codepage not in registry - Add TestIncludeFileWithInputCodePage for Windows-1252 and UTF-16 LE/BE - Add TestGetEncodingWindowsFallback to verify Windows API fallback - Update README.md to document Windows codepage availability - Update code comments to document cross-platform vs Windows-only support

- Change UTF-16 BOM handling from IgnoreBOM to UseBOM for codepages 1200 (UTF-16 LE) and 1201 (UTF-16 BE) to properly strip BOMs on decode (pkg/sqlcmd/codepage.go) - Eliminate redundant CodePage parsing by storing parsed settings in codePageSettings field after validation in Validate(), then reusing in run() (cmd/sqlcmd/sqlcmd.go) - Add comprehensive Code Page Support documentation section to README.md with format guide, common codepages table, practical examples, and notes on default behavior Note: Integration test for IncludeFile with non-UTF8 input already exists in TestIncludeFileWithInputCodePage (pkg/sqlcmd/sqlcmd_test.go)

- Add validation to error when -f arg is non-empty but no codepage parsed (e.g., ',' or whitespace-only input) - Add unit tests for comma-only, whitespace-only, and multiple-comma inputs - Fix misleading BOM comments to accurately describe BOMOverride behavior - Remove unused skipOnEncError field from test table

dlevy-msft-sql · 2026-01-26T00:13:51Z

Supersedes #627

Copilot

Pull request overview

This PR adds configurable input/output code page handling to the legacy sqlcmd CLI, improving compatibility with ODBC sqlcmd and enabling legacy-encoding workflows.

Changes:

Introduces a central code page registry and parsing logic (-f / --code-page) with support for many Windows, OEM, ISO-8859, CJK, and EBCDIC encodings and a --list-codepages listing mode.
Wires code page settings through the legacy CLI into the core Sqlcmd engine so :R, :OUT, and :ERROR commands correctly decode input files and encode output/error files according to the configured code pages.
Adds comprehensive unit tests for code page parsing/lookup and CLI argument validation, plus README documentation and examples for the new functionality.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`pkg/sqlcmd/sqlcmd_test.go`	Adds tests to verify `IncludeFile` decodes input files correctly under various input code pages (Windows-1252, UTF-16LE/BE with and without BOM).
`pkg/sqlcmd/sqlcmd.go`	Extends `Sqlcmd` with a `CodePage` field and updates `IncludeFile` to select an appropriate `io.Reader` based on configured input code page or BOM auto-detection.
`pkg/sqlcmd/commands_test.go`	Adds tests confirming `:OUT` and `:ERROR` commands honor the configured output code page when writing to files.
`pkg/sqlcmd/commands.go`	Introduces a `transformWriteCloser` helper and updates `:OUT`/`:ERROR` commands to encode output using the selected code page while ensuring underlying files are closed correctly.
`pkg/sqlcmd/codepage_windows.go`	Implements Windows-specific `encoding.Encoding` support backed by `MultiByteToWideChar`/`WideCharToMultiByte` for system-installed code pages not in the built-in registry.
`pkg/sqlcmd/codepage_test.go`	Adds tests for `ParseCodePage`, `GetEncoding`, `SupportedCodePages`, and Windows-specific fallback behavior, including round-trip checks for a Windows-only EBCDIC code page.
`pkg/sqlcmd/codepage_other.go`	Provides a non-Windows implementation of `getSystemCodePageEncoding` that surfaces a clear error for unsupported system code pages.
`pkg/sqlcmd/codepage.go`	Defines the cross-platform code page registry, `CodePageSettings`, parsing/validation logic for `-f`, and a `SupportedCodePages` API used by the CLI.
`cmd/sqlcmd/sqlcmd_test.go`	Extends CLI argument tests to cover `-f` / `--code-page`, `--list-codepages`, and various invalid code page argument combinations.
`cmd/sqlcmd/sqlcmd.go`	Adds `CodePage` and `ListCodePages` arguments, validates and stores parsed `CodePageSettings`, applies them to the `Sqlcmd` instance, and implements `--list-codepages` output formatting.
`README.md`	Documents the new `-f` flag, describes supported code pages, provides usage examples, and explains `--list-codepages` and default BOM auto-detection behavior.

pkg/sqlcmd/codepage_windows.go

Use strconv.Itoa instead of %d format verb to avoid locale-based number formatting that adds thousands separators (99,999 vs 99999). This ensures consistent error messages across all platforms.

Address Copilot review: The windowsDecoder and windowsEncoder now properly handle the atEOF parameter and buffer incomplete sequences between Transform calls. This ensures correct behavior when transform.Reader/Writer splits multibyte sequences across chunks. Changes: - Add buffer fields to windowsDecoder and windowsEncoder structs - Use MB_ERR_INVALID_CHARS to detect incomplete sequences in decoder - Use utf8.Valid to detect incomplete UTF-8 sequences in encoder - Return transform.ErrShortSrc when more input is needed - Return error for incomplete sequences at EOF - Add TestWindowsEncodingStreaming test for streaming behavior

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

pkg/sqlcmd/sqlcmd.go

README.md

pkg/sqlcmd/codepage_windows.go

pkg/sqlcmd/codepage_test.go

- Fix getSystemCodePageEncoding comment to describe error return behavior - Clarify test comment in TestWindowsEncodingStreaming to match actual coverage - Improve comments in IncludeFile explaining BOMOverride behavior - Update README notes to clarify that --list-codepages shows built-in set only and that Windows may have additional codepages via OS API

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

dlevy-msft-sql added 11 commits January 24, 2026 20:13

Merge branch 'main' into round-out-part2

5309eda

Fix ineffectual assignment lint error

faa945c

Fix locale-specific number formatting in codepage error

2a14b75

Use strconv.Itoa instead of %d to avoid locale-specific thousands separators in error message.

test: add TestSupportedCodePages to verify registry consistency

065b1f4

- Verify all returned codepages are valid in GetEncoding - Ensure results are sorted by codepage number - Check well-known codepages are present

dlevy-msft-sql requested a review from Copilot January 26, 2026 00:12

Copilot started reviewing on behalf of dlevy-msft-sql January 26, 2026 00:13 View session

dlevy-msft-sql mentioned this pull request Jan 26, 2026

Add -f codepage flag for input/output encoding #627

Closed

dlevy-msft-sql self-assigned this Jan 26, 2026

dlevy-msft-sql added sqlcmd switch switch in existing sqlcmd Size: S Small issue (less than one week effort) labels Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

pkg/sqlcmd/codepage_windows.go Show resolved Hide resolved

dlevy-msft-sql added 2 commits January 25, 2026 18:26

Fix codepage error message for locale-independent formatting

b96e228

Use strconv.Itoa instead of %d format verb to avoid locale-based number formatting that adds thousands separators (99,999 vs 99999). This ensures consistent error messages across all platforms.

dlevy-msft-sql requested a review from Copilot January 26, 2026 00:41

Copilot started reviewing on behalf of dlevy-msft-sql January 26, 2026 00:42 View session

dlevy-msft-sql added this to the ODBC SQLCMD Interchangeability milestone Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

pkg/sqlcmd/sqlcmd.go Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

pkg/sqlcmd/codepage_windows.go Outdated Show resolved Hide resolved

pkg/sqlcmd/codepage_test.go Outdated Show resolved Hide resolved

dlevy-msft-sql requested a review from Copilot January 26, 2026 00:52

Copilot started reviewing on behalf of dlevy-msft-sql January 26, 2026 00:53 View session

Copilot AI reviewed Jan 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add -f codepage flag for input/output encoding #638

Add -f codepage flag for input/output encoding #638

Uh oh!

dlevy-msft-sql commented Jan 26, 2026

Uh oh!

dlevy-msft-sql commented Jan 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Add -f codepage flag for input/output encoding #638

Are you sure you want to change the base?

Add -f codepage flag for input/output encoding #638

Uh oh!

Conversation

dlevy-msft-sql commented Jan 26, 2026

Summary

Changes

Usage

Supported Codepages

Testing

Uh oh!

dlevy-msft-sql commented Jan 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant