Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292

wants to merge 9 commits into from

Conversation

AlexRadch
Copy link
Contributor

Close #52947

This commit introduces the System.Text.Unicode.Utf8 type to the Microsoft.Bcl.Memory library.

It includes type forwarding for Utf8 in Microsoft.Bcl.Memory.Forwards.cs, updates the documentation in PACKAGE.md to include Utf8 functionality, and adds corresponding test cases in Microsoft.Bcl.Memory.Tests.csproj.

The documentation now emphasizes Utf8 alongside Index, Range, and Base64Url, highlighting its role in converting data between UTF-8 and UTF-16 encodings.

@AlexRadch
Add support for System.Text.Unicode.Utf8
c043cbd 
This commit introduces the `System.Text.Unicode.Utf8` type to the `Microsoft.Bcl.Memory` library.

It includes type forwarding for `Utf8` in `Microsoft.Bcl.Memory.Forwards.cs`, updates the documentation in `PACKAGE.md` to include `Utf8` functionality, and adds corresponding test cases in `Microsoft.Bcl.Memory.Tests.csproj`.

 The documentation now emphasizes `Utf8` alongside `Index`, `Range`, and `Base64Url`, highlighting its role in converting data between UTF-8 and UTF-16 encodings.
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jan 10, 2025
@dotnet-policy-service dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jan 10, 2025
@stephentoub
Copy link
Member

stephentoub commented Jan 11, 2025 •
edited
Loading

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

@stephentoub stephentoub added the NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) label Jan 11, 2025
@teo-tsirpanis teo-tsirpanis added area-System.Memory and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jan 11, 2025
@build-analysis build-analysis bot mentioned this pull request Jan 11, 2025
Open
3 tasks
@AlexRadch
Enhance UTF-8 and Unicode support in BCL Memory
f60df7c 
Updated `PackageDescription` to include "Utf8" support.

Added new `ItemGroup` for conditional compilation of UTF-8
and Unicode handling files for non-net8.0 frameworks.

Modified visibility and implementations in `Ascii.Utility.Helpers.cs`,
`Utf8.cs`, and `Utf8Utility` based on `MICROSOFT_BCL_MEMORY` define.
@AlexRadch
Copy link
Contributor Author

AlexRadch commented Jan 11, 2025 •
edited
Loading

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

I added Utf8 APIs to Microsoft.Bcl.Memory library. I haven't added tests yet. I'll add them too.

@AlexRadch
Enhance Unicode handling in tests and project structure
5b86cd6 
Updated `Microsoft.Bcl.Memory.Tests.csproj` to include `UnicodeUtility.cs` and removed .NET 8.0 targeting condition.

Modified `Utf8Tests.cs` by adjusting using directives and enhancing the `DecodeHex` method with conditional compilation for .NET 5.0+.
@AlexRadch
Add compilation constant for Microsoft BCL Memory
22be776 
Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.
@AlexRadch
Copy link
Contributor Author

AlexRadch commented Jan 11, 2025 •
edited
Loading

Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard.

@stephentoub I added Utf8 APIs to Microsoft.Bcl.Memory library for older .NET platforms. I've also added tests for older .NET platforms.

@build-analysis build-analysis bot mentioned this pull request Jan 11, 2025
Open
3 tasks
teo-tsirpanis
teo-tsirpanis reviewed Jan 11, 2025
View reviewed changes
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs Outdated Show resolved Hide resolved
src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs
@@ -9,7 +9,10 @@

namespace System.Text.Unicode
{
#if SYSTEM_PRIVATE_CORELIB
#if SYSTEM_PRIVATE_CORELIB || MICROSOFT_BCL_MEMORY
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is being referenced only by System.Private.CoreLib and Microsoft.BCL.Memory, so the class can always be public.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid to make such changes and let someone else make them as a separate PR.

src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs Outdated Show resolved Hide resolved
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.Helpers.cs Outdated Show resolved Hide resolved
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.Helpers.cs Outdated Show resolved Hide resolved
src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs Outdated
Comment on lines 99 to 104
#if !MICROSOFT_BCL_MEMORY
public
#else
private
#endif
static byte[] ToUtf8(Rune rune)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#if !MICROSOFT_BCL_MEMORY
public
#else
private
#endif
static byte[] ToUtf8(Rune rune)
public static byte[] ToUtf8(Rune rune)

Why can't this always be public?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In tests for older versions of .Net where there is no Rune class yet, I create it as an internal class, and the compiler complains that the public API cannot reference internal classes.

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs Outdated Show resolved Hide resolved
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs Outdated Show resolved Hide resolved
src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.Forwards.cs Outdated Show resolved Hide resolved
src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj Outdated Show resolved Hide resolved
@build-analysis build-analysis bot mentioned this pull request Jan 11, 2025
Open
3 tasks
AlexRadch and others added 5 commits January 12, 2025 08:29
@AlexRadch @teo-tsirpanis
Apply suggestions from code review
79214bd 
Co-authored-by: Theodore Tsirpanis <[email protected]>
@AlexRadch
Update Microsoft.Bcl.Memory for framework compatibility
def40c4 
- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.
@AlexRadch
Space
a1caf07
@AlexRadch
Remove MICROSOFT_BCL_MEMORY from project constants
695e362 
Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.
@AlexRadch
Refactor Utf8Tests for .NET compatibility improvements
063359a
@teo-tsirpanis teo-tsirpanis mentioned this pull request Jan 12, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@teo-tsirpanis teo-tsirpanis teo-tsirpanis left review comments

@jkotas jkotas jkotas left review comments

@akoeplinger akoeplinger akoeplinger left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
area-System.Memory community-contribution Indicates that the PR has been added by a community member NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Proposal: Create OOB package for Rune, System.Text.Unicode.Utf8, and related APIs
5 participants
@AlexRadch @stephentoub @akoeplinger @jkotas @teo-tsirpanis