FEAT: Support Emojis for data types

[gargsaumya]

Work Item / Issue Reference

AB#37664

GitHub Issue: #<ISSUE_NUMBER>

---

Summary

This pull request improves Unicode string handling in the SQL type mapping logic and adds comprehensive tests for round-tripping emoji and special characters in the database. The main focus is on ensuring that Unicode strings, including those with emojis and special characters, are correctly mapped and stored in SQL Server.

Unicode string mapping improvements:

• Updated the _map_sql_type method in cursor.py to use the UTF-16 code unit length for Unicode string parameters instead of the Python string length, ensuring correct sizing for both long and short Unicode strings. [1] [2]

Testing enhancements:

• Added a new test test_emoji_round_trip in test_004_cursor.py to verify that various emoji, accented, and non-Latin characters can be inserted and retrieved accurately from the database, improving coverage for Unicode edge cases.

[Copilot]

Pull Request Overview

This pull request enhances Unicode string handling in the SQL type mapping logic to properly support emojis and special characters. The key improvement addresses the incorrect sizing calculation for Unicode strings by using UTF-16 code unit length instead of Python string length.

• Updated _map_sql_type method to calculate UTF-16 code unit length for Unicode parameters
• Added comprehensive test coverage for emoji and special character round-trip operations

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
mssql_python/cursor.py	Fixed Unicode string length calculation by using UTF-16 encoding length instead of Python string length
tests/test_004_cursor.py	Added comprehensive test for emoji and special character round-trip database operations

[sumitmsft]

Can we wrap utf16_len = len(param.encode("utf-16-le")) // 2 in a function and then check on the following: Use try-catch for encoding failures (to handle Unicode Encode error)?

Something like:

def _calculate_utf16_length(self, param):
   ...
   ...
    try:
        return len(param.encode("utf-16-le")) // 2
    except UnicodeEncodeError as e:                    // or some other exception
        # Log error and fallback to Python length
        logger.warning(f"UTF-16 encoding failed: {e}")
        return len(param)

[sumitmsft]

Can we put some comments around Surrogate logic to better understand the code?

[sumitmsft]

Can we make 0xD800, 0xDFFF, 0x10FFFF named constants?

[sumitmsft]

Can you add a test case for "Hello 🚀 World" and "admin🔒user" string and see if there is any corruption happening? Just wanted to check if this goes through and that we do not drop drop surrogates.

If the surrogates are being dropped, then there could be a chance for SQL injection with these type of queries:

"SELECT * FROM users WHERE id = 1🚀' OR '1'='1"

[sumitmsft]

WStringToSQLWCHAR() can return vectors of unpredictable size because UTF-16 conversion may expand data significantly and surrogate pairs double the storage requirements for emoji (as emojis needs 2 code units in UTF-16).

But, vector::assign() doesn't validate capacity and will attempt to store all elements regardless of buffer size.

If utf16.size() > sqlwcharBuffer then we will hit buffer overflow.

There is no error handling or logging for size mismatches

[sumitmsft]

Currently, py::bytes py_bytes(raw_bytes, dataLen) does not validate that dataLen fits within the allocated buffer. If the dataLen is larger than the real buffer, this can leak memory beyond the intended area—exposing potentially sensitive data (like passwords, other users’ data, etc.) to Python.

Let's check if dataLen is non-negative and does not exceed the actual buffer size before creating the Python bytes object.

[sumitmsft]

The function assumes that each wchar_t wc from the input contains a valid Unicode Codepoint, but there's no validation that the wchar_t values are actually valid Unicode.

Since, wchar_t could be different on Win (UTF 16) and Linux (UTF32) it might cause issues.

wstring may contain invalid or malformed codepoint and may cause issues later on.

[sumitmsft]

Left a few comments, please see if we can fix them.