Skip to content

Fix to fallback to utf-8 when provided an invalid charset in web responses - #27789

Draft
Ryan Yates (kilasuit) wants to merge 1 commit into
PowerShell:masterfrom
kilasuit:web-utf8-fix
Draft

Fix to fallback to utf-8 when provided an invalid charset in web responses#27789
Ryan Yates (kilasuit) wants to merge 1 commit into
PowerShell:masterfrom
kilasuit:web-utf8-fix

Conversation

@kilasuit

Copy link
Copy Markdown
Collaborator

PR Summary

This adds in the conversion when servers respond with charset=utf8 & other non recognised charsets.
By doing so the Debug message that contains the WebResponse Body is returned as expected by developers or end users of the WebCmdlets

PR Context

fixes #27788

PR Checklist

@kilasuit
Ryan Yates (kilasuit) requested a review from a team as a code owner August 9, 2026 17:41
Copilot AI lite review requested due to automatic review settings August 9, 2026 17:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@rhubarb-geek-nz

rhubarb-geek-nz commented Aug 11, 2026

Copy link
Copy Markdown

I don't think this is a good solution. It is not appropriate to assume anything you don't understand is UTF-8 without giving any indication of a problem. If somebody specifies a charset it is done for a reason. If you ignore it and carry on then you are risking corrupting the data.

PowerShell already has a mapping for 'utf8' and it is used by Get-Content and Set-Content. Rather than invent a new standard here, I suggest this component should be consistent with the rest of PowerShell. Instead of tying to get the system encoding, catching an exception and then pretending everything is UTF-8, it should be the same algorithm as Get-Content. There is a dictionary of string to encoding already maintained by PowerShell as System.Management.Automation.EncodingConversion.encodingMap.

            var typeClass = typeof(PSCmdlet).Assembly.GetType("System.Management.Automation.EncodingConversion");
            if (typeClass != null)
            {
                var fieldInfo = typeClass.GetField("encodingMap", BindingFlags.NonPublic | BindingFlags.Static);

                if (fieldInfo != null && typeof(IDictionary<string, Encoding>).IsAssignableFrom(fieldInfo.FieldType))
                {
                    encodingMap = (IDictionary<string, Encoding>)fieldInfo.GetValue(null);
                }
            }

This dictionary should be consulted first with a TryGetValue, if that fails it should then use the system provider with Encoding.GetEncoding(). Then if that throws an exception, then so be it.

PS> 'hello' | set-content -literalpath 'hello.txt' -encoding 'utf-8'
PS> 'hello' | set-content -literalpath 'hello.txt' -encoding 'utf8'
PS> 'hello' | set-content -literalpath 'hello.txt' -encoding 'utf-80'
Set-Content: Cannot process argument transformation on parameter 'Encoding'. 'utf-80' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method. (Parameter 'name')

That would address the 'utf8' issue in a PowerShell consistent manner and not break by misinterpreting unknown character set encodings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

When a Web Repsonse returns charset=utf8 this throws the exception into the verbose stream

3 participants