Skip to content

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vstinner merged 12 commits into from
Oct 14, 2020
+21 −2
Merged

bpo-39337: encodings.normalize_encoding() now ignores non-ASCII characters #22219

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
8ae6408
encodings.normalize_encoding() should ignore non-ASCII letters
shihai1991 Sep 12, 2020
0fcafb8
update test
shihai1991 Sep 12, 2020
dea24d8
Revert "update test"
shihai1991 Sep 12, 2020
dbb0062
apply victor's comment
shihai1991 Oct 3, 2020
3fa221f
apply victor's comment
shihai1991 Oct 9, 2020
a69eef8
Merge remote-tracking branch 'origin/master' into bpo_39337_new_2
shihai1991 Oct 9, 2020
2e73d13
apply victor's comments
shihai1991 Oct 10, 2020
95c1d98
apply victor's comment
shihai1991 Oct 12, 2020
03bfd9b
apply victor's comment
shihai1991 Oct 12, 2020
38f28bd
apply victor's comment
shihai1991 Oct 13, 2020
5982784
apply victor's comment
shihai1991 Oct 13, 2020
4ecb8a1
apply victor's comment
shihai1991 Oct 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'origin/master' into bpo_39337_new_2 
* origin/master: (147 commits)
  Fix the attribute names in the docstring of GenericAlias (GH-22594)
  bpo-39337: Add a test case for normalizing of codec names (GH-19069)
  bpo-41557: Update Windows installer to use SQLite 3.33.0 (GH-21960)
  bpo-41976: Fix the fallback to gcc of ctypes.util.find_library when using gcc>9 (GH-22598)
  bpo-41306: Allow scale value to not be rounded (GH-21715)
  bpo-41970: Avoid test failure in test_lib2to3 if the module is already imported (GH-22595)
  bpo-41376: Fix the documentation of `site.getusersitepackages()` (GH-21602)
  Revert "bpo-26680: Incorporate is_integer in all built-in and standard library numeric types (GH-6121)" (GH-22584)
  bpo-41923: PEP 613: Add TypeAlias to typing module (#22532)
  Fix comment about PyObject_IsTrue. (GH-22343)
  bpo-38605: Make 'from __future__ import annotations' the default (GH-20434)
  bpo-41905: Add abc.update_abstractmethods() (GH-22485)
  bpo-41944: No longer call eval() on content received via HTTP in the UnicodeNames tests (GH-22575)
  bpo-41944: No longer call eval() on content received via HTTP in the CJK codec tests (GH-22566)
  Post 3.10.0a1
  Python 3.10.0a1
  bpo-41584: clarify when the reflected method of a binary arithemtic operator is called (#22505)
  bpo-41939: Fix test_site.test_license_exists_at_url() (#22559)
  bpo-41774: Tweak new programming FAQ entry (GH-22562)
  bpo-41936. Remove macros Py_ALLOW_RECURSION/Py_END_ALLOW_RECURSION (GH-22552)
  ...
  • Loading branch information
@shihai1991
shihai1991 committed Oct 9, 2020
commit a69eef84c86b0b96724a58cdeac5d56a90cffef9
25 changes: 25 additions & 0 deletions Lib/test/test_codecs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3415,6 +3415,31 @@ def test_rot13_func(self):
'To be, or not to be, that is the question')


class CodecNameNormalizationTest(unittest.TestCase):
"""Test codec name normalization"""
def test_normalized_encoding(self):
FOUND = (1, 2, 3, 4)
NOT_FOUND = (None, None, None, None)
def search_function(encoding):
if encoding == "aaa_8":
return FOUND
else:
return NOT_FOUND

codecs.register(search_function)
self.addCleanup(codecs.unregister, search_function)
self.assertEqual(FOUND, codecs.lookup('aaa_8'))
self.assertEqual(FOUND, codecs.lookup('AAA-8'))
self.assertEqual(FOUND, codecs.lookup('AAA---8'))
self.assertEqual(FOUND, codecs.lookup('AAA 8'))
self.assertEqual(FOUND, codecs.lookup('aaa\xe9\u20ac-8'))
self.assertEqual(NOT_FOUND, codecs.lookup('AAA.8'))
self.assertEqual(NOT_FOUND, codecs.lookup('AAA...8'))
self.assertEqual(NOT_FOUND, codecs.lookup('BBB-8'))
self.assertEqual(NOT_FOUND, codecs.lookup('BBB.8'))
self.assertEqual(NOT_FOUND, codecs.lookup('a\xe9\u20ac-8'))


class EncodingNormalizationTest(unittest.TestCase):
Comment thread
shihai1991 marked this conversation as resolved.
Outdated

def test_bpo39337(self):
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.