Skip to content

FIX: Warn when setencoding settings cannot be applied (#825) - #828

Merged
Jahnvi Thakkar (jahnvi480) merged 4 commits into
mainfrom
jahnvi/issue-825-setencoding-ctype-sql-char-is-silently-i-a81bf8
Oct 5, 2026
Merged

Jahnvi Thakkar (jahnvi480) merged 4 commits into
mainfrom
jahnvi/issue-825-setencoding-ctype-sql-char-is-silently-i-a81bf8

Conversation

@jahnvi480

@jahnvi480 Jahnvi Thakkar (jahnvi480) commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Work Item / Issue Reference

AB#48882


Summary

Make unsupported setencoding() requests non-silent while preserving the existing UTF-16LE / SQL_C_WCHAR parameter binding.

  • Emit a caller-visible UserWarning for valid requests that cannot be applied, including explicit or automatic SQL_CHAR and UTF-16BE. Keep supported defaults warning-free and retain existing validation errors.
  • Preserve getencoding() compatibility and leave settings unchanged when warnings are treated as errors.
  • Align native execute() and executemany() encoding checks.
  • Clarify requested versus effective settings in README/docstrings and document the legacy internal C-type alias without changing its value.
  • Replace permissive encoding tests with strict warning and data-preservation assertions across execute, executemany, setinputsizes, and streaming.

This is a warning-based fix for ignored configuration, not an implementation of configurable narrow binding. The public wiki has not been edited and still needs corresponding clarification if this approach is adopted.

Validation

  • Reproduced the silent no-op on Windows x64 / Python 3.13.15 against live SQL Server LocalDB, including native binding diagnostics. New warning assertions fail on the baseline.
  • Rebuilt the native extension; 408 focused encoding, CP1252-boundary, and execute-parity tests passed.
  • Black passed for all 101 Python files.
  • Full non-stress suite: 5295 passed, 173 skipped, 42 deselected, 2 failures. Both failures also reproduce with the unchanged HEAD connection implementation: a logging test assumes a password exists on a passwordless LocalDB connection, and a long-path test exceeds Windows MAX_PATH.
  • Linux/macOS validation remains for CI.

Preserve wide-character binding while warning about ignored encoding requests. Align native encoding gates and document and test the effective contract.

Refs #825; AB#48882

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings September 30, 2026 12:18
@jahnvi480 Jahnvi Thakkar (jahnvi480) added bug Something isn't working area: data-types Type conversion and encoding: VARCHAR/NVARCHAR, UTF-8, decimal, datetime, UUID, binary, JSON. labels Sep 30, 2026
@github-actions github-actions Bot added the pr-size: medium Moderate update size label Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

PR Performance Report

✅ No regression detected

No consistent slowdowns detected across all 2 environments.

0 IMPROVEMENTS 0 SLOWDOWNS 2/2 ENVIRONMENTS

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task Before After Paired change Result
Connection opening 11.480 ms 11.103 ms -3.0% no signal
SELECT queries 1.129 ms 1.177 ms +4.0% no signal
Row insertion 35.859 ms 35.602 ms -0.1% no signal
Executemany inserts 162.406 ms 161.398 ms -0.5% no signal
Fetch-all queries 126.816 ms 126.605 ms +0.3% no signal
Row-by-row fetching 14.454 ms 14.312 ms -0.6% no signal
Batched row fetching 123.313 ms 120.468 ms -3.1% no signal
Transaction commit and rollback 118.991 ms 118.203 ms -0.4% no signal
Arrow row fetching 96.966 ms 98.366 ms +1.4% no signal
100,000-row insertion 487.818 ms 483.132 ms -0.7% no signal
Row fetching in batches of 100 121.985 ms 123.698 ms +0.1% no signal
Row fetching in batches of 10,000 143.481 ms 133.561 ms -8.3% no signal
Repeated positional queries 34.773 ms 34.648 ms -0.1% no signal
Repeated named-parameter queries 37.191 ms 36.843 ms -0.9% no signal
Legacy 100,000-row insertion 385.051 ms 378.886 ms -1.0% no signal
Insertion with explicit input sizes 519.320 ms 518.573 ms -1.6% no signal
Joined aggregation queries 182.658 ms 188.294 ms +4.0% no signal
Large joined-result fetching 200.835 ms 202.556 ms +0.2% no signal
1.2-million-row fetching 3588.678 ms 3602.000 ms +1.7% no signal
Common table expression queries 5.853 ms 5.691 ms -1.6% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.327 ms 1.287 ms -3.2% no signal
10,000 scalar values / fetchval() (debug disabled) 107.929 ms 109.777 ms +0.8% no signal

Unix / SQL Server 2025

Database task Before After Paired change Result
Connection opening 96.098 ms 96.318 ms +0.2% no signal
SELECT queries 1.104 ms 1.066 ms -1.1% no signal
Row insertion 34.942 ms 33.965 ms -3.0% no signal
Executemany inserts 150.418 ms 149.614 ms -0.5% no signal
Fetch-all queries 120.193 ms 121.330 ms +1.0% no signal
Row-by-row fetching 14.419 ms 14.274 ms -0.3% no signal
Batched row fetching 116.958 ms 128.407 ms +7.0% no signal
Transaction commit and rollback 115.038 ms 117.994 ms -0.4% no signal
Arrow row fetching 96.460 ms 94.624 ms -3.3% no signal
100,000-row insertion 433.366 ms 432.047 ms +0.1% no signal
Row fetching in batches of 100 122.290 ms 121.677 ms -0.5% no signal
Row fetching in batches of 10,000 126.081 ms 137.914 ms -1.3% no signal
Repeated positional queries 33.485 ms 33.678 ms -0.0% no signal
Repeated named-parameter queries 36.032 ms 35.899 ms -0.6% no signal
Legacy 100,000-row insertion 366.762 ms 344.969 ms -6.1% no signal
Insertion with explicit input sizes 504.281 ms 486.369 ms -4.6% no signal
Joined aggregation queries 161.344 ms 160.264 ms -1.6% no signal
Large joined-result fetching 177.542 ms 180.911 ms +0.1% no signal
1.2-million-row fetching 3541.616 ms 3543.349 ms +0.7% no signal
Common table expression queries 5.104 ms 5.069 ms -0.7% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.444 ms 1.455 ms +0.5% no signal
10,000 scalar values / fetchval() (debug disabled) 108.309 ms 107.236 ms -0.4% no signal
Build and measurement details

ADO build 180427

PR head: d80c829109bc4294d70317e3922c0531d7b83b63
Base: 436b6cbc94092eee59c803d2acf3bcaf8c21aa0f
Measured merge: db3d9483b3c68b60f0577f63820fa6802b29083c

  • Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4295.3; 5 paired comparisons and 1 warmup.
  • Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5005.3; 5 paired comparisons and 1 warmup.

A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation consistently preserves wide-character binding and provides comprehensive regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds explicit warnings when unsupported setencoding() requests cannot affect UTF-16LE wide-character binding.

Changes:

  • Warns while preserving requested settings and existing validation.
  • Aligns native execute() and executemany() encoding gates.
  • Documents and tests effective binding behavior.
File Description
connection.py Adds warnings and clarifies encoding semantics.
constants.py Documents the legacy C-type alias.
ddbc_bindings.cpp Aligns native encoding handling.
test_013_encoding_decoding.py Adds strict warning and round-trip coverage.
README.md Documents requested versus effective encoding.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@jahnvi480
Jahnvi Thakkar (jahnvi480) marked this pull request as ready for review September 30, 2026 12:25
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

📊 Code Coverage Report

🔥 Diff Coverage

81%


🎯 Overall Coverage

85%


📈 Total Lines Covered: 9445 out of 11094
📁 Project: mssql-python


Diff Coverage

Diff: main...HEAD, staged and unstaged changes

  • mssql_python/connection.py (100%)
  • mssql_python/pybind/ddbc_bindings.cpp (77.8%): Missing lines 2140-2141

Summary

  • Total: 11 lines
  • Missing: 2 lines
  • Coverage: 81%

mssql_python/pybind/ddbc_bindings.cpp

Lines 2136-2145

  2136                            (SQLPOINTER)SQL_CONCUR_READ_ONLY, 0);
  2137     }
  2138 
  2139     // This codec only applies to parameters already typed as real SQL_C_CHAR (1).
! 2140     // Public text parameter detection uses SQL_C_WCHAR (-8), including the
! 2141     // Python layer's legacy SQL_C_CHAR alias. setencoding() does not change
  2142     // paramCType and warns when the requested settings cannot be applied.
  2143     std::string charEncoding = "utf-8";
  2144     if (encoding_settings.contains("ctype") && encoding_settings.contains("encoding")) {
  2145         int ctype = encoding_settings["ctype"].cast<int>();


📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)
mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 62.6%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.connection.connection_pool.cpp: 82.3%
mssql_python.pybind.connection.connection.cpp: 83.1%
mssql_python.logging.py: 86.2%
mssql_python.pooling.py: 90.1%
mssql_python.pybind.fetch_temporal.hpp: 92.1%
mssql_python.cursor.py: 92.5%

🔗 Quick Links

⚙️ Build Summary 📋 Coverage Details

View Azure DevOps Build

Browse Full Coverage Report

Copilot AI balanced review requested due to automatic review settings October 1, 2026 10:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The executemany() streaming path still ignores the selected encoding and parameter C type.

Review effort: Balanced
Findings: 1 High severity · 2 Low severity

Open (3)

Comment thread mssql_python/pybind/ddbc_bindings.cpp
Comment thread README.md Outdated
Comment thread mssql_python/connection.py Outdated
Comment thread tests/test_013_encoding_decoding.py Outdated
Address review feedback by documenting validation errors before warnings and removing permissive ASCII DAE tests. Assert public executemany uses DDBCSQLExecute for streaming, with exact UTF-16LE data preservation and native bridge call counts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 1, 2026 11:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Valid UTF-16LE codec aliases can incorrectly select SQL_CHAR and emit an unsupported-setting warning.

Review effort: Balanced
Findings: None

Resolved since last review (3)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Canonicalize codec aliases before ctype validation

mssql_python/​connection.py:1293

A valid alias for the supported default can still trigger this warning. For example, Python accepts UTF-16-LE, but casefold() produces utf-16-le, which is not in UTF16_ENCODINGS; the preceding auto-selection therefore chooses SQL_CHAR, and this condition reports an unsupported request even though the codec is UTF-16LE. Canonicalize Python codec aliases before ctype selection/validation and add a warning-free alias case.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Native ODBC encoding behavior still requires the pending Linux and macOS CI validation.

Review effort: Balanced
Findings: None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this addresses the ignored-settings problem while preserving existing text handling. approving.

Copilot AI balanced review requested due to automatic review settings October 5, 2026 05:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation and coverage match the stated contract, with only a non-blocking warning-message clarification identified.

Review effort: Balanced
Findings: 1 Low severity

Open (1)

Comment thread mssql_python/connection.py
@jahnvi480
Jahnvi Thakkar (jahnvi480) merged commit 1cf04a1 into main Oct 5, 2026
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: data-types Type conversion and encoding: VARCHAR/NVARCHAR, UTF-8, decimal, datetime, UUID, binary, JSON. bug Something isn't working pr-size: medium Moderate update size

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants