Normalise charset in the ReadBufferAsString method in System.Net.Http fixes #133319

[kilasuit]

This PR Fixes

• Web Server returning utf8 (not utf-8) for ContentType header throws an error in ReadBufferAsString method #133319

I have built the System.Net.Http library, plonked the resulting dll into a local build of PowerShell to confirm it that this does indeed also fix PowerShell/PowerShell#27788

As can be seen in this screenshot

I felt this was the smallest code change needed and in the right place
Happy to take comment out if that's desired

[azure-pipelines]

Azure Pipelines:
Successfully started running 4 pipeline(s).
12 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[Copilot]

🟡 Changes recommended

It needs scope reduction (revert formatting churn) and test coverage for the new charset-acceptance behavior (including case/quoted variants).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates HttpContent.ReadBufferAsString in System.Net.Http to be more tolerant of non-standard charset tokens (e.g., utf8) when converting buffered HTTP content to a string.

Changes:

• Adds charset “normalization” before calling Encoding.GetEncoding(...).
• (Unintentionally) introduces large whitespace/indentation-only churn across HttpContent.cs, significantly increasing diff noise and review surface.

File summaries

File	Description
src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs	Adds charset normalization in ReadBufferAsString, but also includes extensive formatting-only changes that should be reverted to keep the PR focused.

Review details

Suppressed comments (1)

src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:152

• The new charset mapping is case-sensitive and only runs in the unquoted path, so inputs like charset=UTF8 or charset="utf8" will still throw. Consider unquoting first and then normalizing with OrdinalIgnoreCase before calling Encoding.GetEncoding (and avoid hard-linking to external repos in a code comment).

                        var normalised = charset switch
                        {
                            // discovered in https://github.com/PowerShell/PowerShell/issues/27788
                            // where a server may reply with utf8, added below additional options,
                            // with a fall back to current behaviour.

• Files reviewed: 1/1 changed files
• Comments generated: 2
• Review effort level: Lite

[kilasuit]

@dotnet-policy-service agree

[Copilot]

🟡 Changes recommended

The current normalization is case-sensitive and bypasses the quoted-charset path, so common variants like "UTF8" or "\"utf8\"" still throw, and the new behavior isn’t covered by tests.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/libraries/System.Net.Http/src/System/Net/Http/HttpContent.cs:157

• This introduces new behavior (treating several non-standard charset tokens as valid) but there are no new tests exercising it. There are already functional tests around ReadAsStringAsync charset handling (e.g., invalid/quoted charset cases), so adding coverage for values like "utf8", ""utf8"", "UTF8", and "utf_8" would help prevent regressions.

                            "utf8" or "utf-8" or "utf_8" => "utf-8",
                            "utf16" or "utf-16" or "utf_16" => "utf-16",
                            "unicode" => "utf-16",
                            "latin1" or "latin-1" => "iso-8859-1",
                            "us-ascii" or "ascii" => "us-ascii",

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite