Skip to content

bpo-40275: reduce importing module nums by lazy import in libregrtest #20207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
shihai1991 wants to merge 3 commits into from
Closed

bpo-40275: reduce importing module nums by lazy import in libregrtest #20207

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
27ab539
reduce importing module nums by lazy import
shihai1991 May 19, 2020
0bc2fbc
revert lazy import in basic test function
shihai1991 Jun 17, 2020
324ff77
cancel lazy import of faulthandler in runtest_mp
shihai1991 Jun 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Lib/test/libregrtest/refleak.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
import re
import sys
import warnings
from inspect import isabstract
Copy link
Copy Markdown
Member

@vstinner vstinner Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than modifying refleak imports, it would be more generic to only import the refleak module when it's used.

Currently, clear_caches() is always called. Maybe we can only clear caches when -R 3:3 option is used.

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, victor. I will try your suggestion.

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Jun 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than modifying refleak imports, it would be more generic to only import the refleak module when it's used.

Currently, clear_caches() is always called. Maybe we can only clear caches when -R 3:3 option is used.

about this idea, I found you and serihy discussed the clear_caches() in https://bugs.python.org/issue23839.
the point is that it's maybe raise some memoryError when calling clear_caches() after -R.
So I thought lazy import isabstract in dash_R() would be more safe(it would be called when we use -R 3:3 in tests.)

from test import support
try:
from _abc import _get_dump
Expand All @@ -26,6 +25,7 @@ def dash_R(ns, test_name, test_func):
# This code is hackish and inelegant, but it seems to do the job.
import copyreg
import collections.abc
from inspect import isabstract

if not hasattr(sys, 'gettotalrefcount'):
raise Exception("Tracking reference leaks requires a debug build "
Expand Down Expand Up @@ -148,6 +148,7 @@ def check_fd_deltas(deltas):
def dash_R_cleanup(fs, ps, pic, zdc, abcs):
import copyreg
import collections.abc
from inspect import isabstract

# Restore some original values.
warnings.filters[:] = fs
Expand Down
22 changes: 17 additions & 5 deletions Lib/test/libregrtest/save_env.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,14 @@
import asyncio
Copy link
Copy Markdown
Member

@vstinner vstinner Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO these changes are useless, since saved_test_environment() is used by runtest(). I mean that all these imports are imported anyway.

If someone wants to reduce the number of imported modules, saved_test_environment() must be redesigned. Currently, it has no knowledge of what test is run, and so all checks are always run.

Copy link
Copy Markdown
Member Author

@shihai1991 shihai1991 Jun 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, thanks, victor. I will check the detail of this function.

import builtins
import locale
import logging
import os
import shutil
import sys
import sysconfig
import threading
import urllib.request
import warnings
from test import support
from test.libregrtest.utils import print_warning
try:
import _multiprocessing, multiprocessing.process
import multiprocessing.process
except ImportError:
multiprocessing = None

Expand Down Expand Up @@ -73,19 +69,25 @@ def __init__(self, testname, verbose=0, quiet=False, *, pgo=False):
)

def get_urllib_requests__url_tempfiles(self):
import urllib.request
return list(urllib.request._url_tempfiles)

def restore_urllib_requests__url_tempfiles(self, tempfiles):
for filename in tempfiles:
support.unlink(filename)

def get_urllib_requests__opener(self):
import urllib.request
return urllib.request._opener

def restore_urllib_requests__opener(self, opener):
import urllib.request
urllib.request._opener = opener

def get_asyncio_events__event_loop_policy(self):
return support.maybe_get_event_loop_policy()
def restore_asyncio_events__event_loop_policy(self, policy):
import asyncio
asyncio.set_event_loop_policy(policy)

def get_sys_argv(self):
Expand Down Expand Up @@ -163,29 +165,39 @@ def get_shutil_archive_formats(self):
# we could call get_archives_formats() but that only returns the
# registry keys; we want to check the values too (the functions that
# are registered)
import shutil
return shutil._ARCHIVE_FORMATS, shutil._ARCHIVE_FORMATS.copy()

def restore_shutil_archive_formats(self, saved):
import shutil
shutil._ARCHIVE_FORMATS = saved[0]
shutil._ARCHIVE_FORMATS.clear()
shutil._ARCHIVE_FORMATS.update(saved[1])

def get_shutil_unpack_formats(self):
import shutil
return shutil._UNPACK_FORMATS, shutil._UNPACK_FORMATS.copy()

def restore_shutil_unpack_formats(self, saved):
import shutil
shutil._UNPACK_FORMATS = saved[0]
shutil._UNPACK_FORMATS.clear()
shutil._UNPACK_FORMATS.update(saved[1])

def get_logging__handlers(self):
# _handlers is a WeakValueDictionary
import logging
return id(logging._handlers), logging._handlers, logging._handlers.copy()

def restore_logging__handlers(self, saved_handlers):
# Can't easily revert the logging state
pass

def get_logging__handlerList(self):
# _handlerList is a list of weakrefs to handlers
import logging
return id(logging._handlerList), logging._handlerList, logging._handlerList[:]

def restore_logging__handlerList(self, saved_handlerList):
# Can't easily revert the logging state
pass
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/libregrtest/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import math
import os.path
import sys
import textwrap
from test import support


Expand Down Expand Up @@ -54,6 +53,7 @@ def printlist(x, width=70, indent=4, file=None):
begin each line.
"""

import textwrap
blanks = ' ' * indent
# Print the sorted list: 'x' may be a '--random' list or a set()
print(textwrap.fill(' '.join(str(elt) for elt in sorted(x)), width,
Expand Down