Skip to content

Commit df2d4a6

Browse files
zoobazooba
authored
bpo-37834: Normalise handling of reparse points on Windows (pythonGH-15231)
bpo-37834: Normalise handling of reparse points on Windows * ntpath.realpath() and nt.stat() will traverse all supported reparse points (previously was mixed) * nt.lstat() will let the OS traverse reparse points that are not name surrogates (previously would not traverse any reparse point) * nt.[l]stat() will only set S_IFLNK for symlinks (previous behaviour) * nt.readlink() will read destinations for symlinks and junction points only bpo-1311: os.path.exists('nul') now returns True on Windows * nt.stat('nul').st_mode is now S_IFCHR (previously was an error)
1 parent bcc446f commit df2d4a6

16 files changed

Lines changed: 470 additions & 233 deletions

File tree

‎Doc/library/os.rst‎

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1858,6 +1858,12 @@ features:
18581858
.. versionchanged:: 3.6
18591859
Accepts a :term:`path-like object` for *src* and *dst*.
18601860

1861+
.. versionchanged:: 3.8
1862+
On Windows, now opens reparse points that represent another path
1863+
(name surrogates), including symbolic links and directory junctions.
1864+
Other kinds of reparse points are resolved by the operating system as
1865+
for :func:`~os.stat`.
1866+
18611867

18621868
.. function:: mkdir(path, mode=0o777, *, dir_fd=None)
18631869

@@ -2039,6 +2045,10 @@ features:
20392045
This function can also support :ref:`paths relative to directory descriptors
20402046
<dir_fd>`.
20412047

2048+
When trying to resolve a path that may contain links, use
2049+
:func:`~os.path.realpath` to properly handle recursion and platform
2050+
differences.
2051+
20422052
.. availability:: Unix, Windows.
20432053

20442054
.. versionchanged:: 3.2
@@ -2053,6 +2063,11 @@ features:
20532063
.. versionchanged:: 3.8
20542064
Accepts a :term:`path-like object` and a bytes object on Windows.
20552065

2066+
.. versionchanged:: 3.8
2067+
Added support for directory junctions, and changed to return the
2068+
substitution path (which typically includes ``\\?\`` prefix) rather
2069+
than the optional "print name" field that was previously returned.
2070+
20562071
.. function:: remove(path, *, dir_fd=None)
20572072

20582073
Remove (delete) the file *path*. If *path* is a directory, an
@@ -2366,7 +2381,8 @@ features:
23662381

23672382
On Unix, this method always requires a system call. On Windows, it
23682383
only requires a system call if *follow_symlinks* is ``True`` and the
2369-
entry is a symbolic link.
2384+
entry is a reparse point (for example, a symbolic link or directory
2385+
junction).
23702386

23712387
On Windows, the ``st_ino``, ``st_dev`` and ``st_nlink`` attributes of the
23722388
:class:`stat_result` are always set to zero. Call :func:`os.stat` to
@@ -2403,6 +2419,17 @@ features:
24032419
This function can support :ref:`specifying a file descriptor <path_fd>` and
24042420
:ref:`not following symlinks <follow_symlinks>`.
24052421

2422+
On Windows, passing ``follow_symlinks=False`` will disable following all
2423+
name-surrogate reparse points, which includes symlinks and directory
2424+
junctions. Other types of reparse points that do not resemble links or that
2425+
the operating system is unable to follow will be opened directly. When
2426+
following a chain of multiple links, this may result in the original link
2427+
being returned instead of the non-link that prevented full traversal. To
2428+
obtain stat results for the final path in this case, use the
2429+
:func:`os.path.realpath` function to resolve the path name as far as
2430+
possible and call :func:`lstat` on the result. This does not apply to
2431+
dangling symlinks or junction points, which will raise the usual exceptions.
2432+
24062433
.. index:: module: stat
24072434

24082435
Example::
@@ -2427,6 +2454,14 @@ features:
24272454
.. versionchanged:: 3.6
24282455
Accepts a :term:`path-like object`.
24292456

2457+
.. versionchanged:: 3.8
2458+
On Windows, all reparse points that can be resolved by the operating
2459+
system are now followed, and passing ``follow_symlinks=False``
2460+
disables following all name surrogate reparse points. If the operating
2461+
system reaches a reparse point that it is not able to follow, *stat* now
2462+
returns the information for the original path as if
2463+
``follow_symlinks=False`` had been specified instead of raising an error.
2464+
24302465

24312466
.. class:: stat_result
24322467

@@ -2578,7 +2613,7 @@ features:
25782613

25792614
File type.
25802615

2581-
On Windows systems, the following attribute is also available:
2616+
On Windows systems, the following attributes are also available:
25822617

25832618
.. attribute:: st_file_attributes
25842619

@@ -2587,6 +2622,12 @@ features:
25872622
:c:func:`GetFileInformationByHandle`. See the ``FILE_ATTRIBUTE_*``
25882623
constants in the :mod:`stat` module.
25892624

2625+
.. attribute:: st_reparse_tag
2626+
2627+
When :attr:`st_file_attributes` has the ``FILE_ATTRIBUTE_REPARSE_POINT``
2628+
set, this field contains the tag identifying the type of reparse point.
2629+
See the ``IO_REPARSE_TAG_*`` constants in the :mod:`stat` module.
2630+
25902631
The standard module :mod:`stat` defines functions and constants that are
25912632
useful for extracting information from a :c:type:`stat` structure. (On
25922633
Windows, some items are filled with dummy values.)
@@ -2614,6 +2655,14 @@ features:
26142655
.. versionadded:: 3.7
26152656
Added the :attr:`st_fstype` member to Solaris/derivatives.
26162657

2658+
.. versionadded:: 3.8
2659+
Added the :attr:`st_reparse_tag` member on Windows.
2660+
2661+
.. versionchanged:: 3.8
2662+
On Windows, the :attr:`st_mode` member now identifies special
2663+
files as :const:`S_IFCHR`, :const:`S_IFIFO` or :const:`S_IFBLK`
2664+
as appropriate.
2665+
26172666
.. function:: statvfs(path)
26182667

26192668
Perform a :c:func:`statvfs` system call on the given path. The return value is

‎Doc/library/shutil.rst‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -304,6 +304,10 @@ Directory and files operations
304304
Added a symlink attack resistant version that is used automatically
305305
if platform supports fd-based functions.
306306

307+
.. versionchanged:: 3.8
308+
On Windows, will no longer delete the contents of a directory junction
309+
before removing the junction.
310+
307311
.. attribute:: rmtree.avoids_symlink_attacks
308312

309313
Indicates whether the current platform and implementation provides a

‎Doc/library/stat.rst‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -425,3 +425,13 @@ for more detail on the meaning of these constants.
425425
FILE_ATTRIBUTE_VIRTUAL
426426

427427
.. versionadded:: 3.5
428+
429+
On Windows, the following constants are available for comparing against the
430+
``st_reparse_tag`` member returned by :func:`os.lstat`. These are well-known
431+
constants, but are not an exhaustive list.
432+
433+
.. data:: IO_REPARSE_TAG_SYMLINK
434+
IO_REPARSE_TAG_MOUNT_POINT
435+
IO_REPARSE_TAG_APPEXECLINK
436+
437+
.. versionadded:: 3.8

‎Doc/whatsnew/3.8.rst‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -808,6 +808,21 @@ A new :func:`os.memfd_create` function was added to wrap the
808808
``memfd_create()`` syscall.
809809
(Contributed by Zackery Spytz and Christian Heimes in :issue:`26836`.)
810810

811+
On Windows, much of the manual logic for handling reparse points (including
812+
symlinks and directory junctions) has been delegated to the operating system.
813+
Specifically, :func:`os.stat` will now traverse anything supported by the
814+
operating system, while :func:`os.lstat` will only open reparse points that
815+
identify as "name surrogates" while others are opened as for :func:`os.stat`.
816+
In all cases, :attr:`stat_result.st_mode` will only have ``S_IFLNK`` set for
817+
symbolic links and not other kinds of reparse points. To identify other kinds
818+
of reparse point, check the new :attr:`stat_result.st_reparse_tag` attribute.
819+
820+
On Windows, :func:`os.readlink` is now able to read directory junctions. Note
821+
that :func:`~os.path.islink` will return ``False`` for directory junctions,
822+
and so code that checks ``islink`` first will continue to treat junctions as
823+
directories, while code that handles errors from :func:`os.readlink` may now
824+
treat junctions as links.
825+
811826

812827
os.path
813828
-------
@@ -824,6 +839,9 @@ characters or bytes unrepresentable at the OS level.
824839
environment variable and does not use :envvar:`HOME`, which is not normally set
825840
for regular user accounts.
826841

842+
:func:`~os.path.isdir` on Windows no longer returns true for a link to a
843+
non-existent directory.
844+
827845
:func:`~os.path.realpath` on Windows now resolves reparse points, including
828846
symlinks and directory junctions.
829847

@@ -912,6 +930,9 @@ format for new archives to improve portability and standards conformance,
912930
inherited from the corresponding change to the :mod:`tarfile` module.
913931
(Contributed by C.A.M. Gerlach in :issue:`30661`.)
914932

933+
:func:`shutil.rmtree` on Windows now removes directory junctions without
934+
recursively removing their contents first.
935+
915936

916937
ssl
917938
---

‎Include/fileutils.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ struct _Py_stat_struct {
8484
time_t st_ctime;
8585
int st_ctime_nsec;
8686
unsigned long st_file_attributes;
87+
unsigned long st_reparse_tag;
8788
};
8889
#else
8990
# define _Py_stat_struct stat

‎Lib/shutil.py‎

Lines changed: 41 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -452,7 +452,14 @@ def _copytree(entries, src, dst, symlinks, ignore, copy_function,
452452
dstname = os.path.join(dst, srcentry.name)
453453
srcobj = srcentry if use_srcentry else srcname
454454
try:
455-
if srcentry.is_symlink():
455+
is_symlink = srcentry.is_symlink()
456+
if is_symlink and os.name == 'nt':
457+
# Special check for directory junctions, which appear as
458+
# symlinks but we want to recurse.
459+
lstat = srcentry.stat(follow_symlinks=False)
460+
if lstat.st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT:
461+
is_symlink = False
462+
if is_symlink:
456463
linkto = os.readlink(srcname)
457464
if symlinks:
458465
# We can't just leave it to `copy_function` because legacy
@@ -537,6 +544,37 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2,
537544
ignore_dangling_symlinks=ignore_dangling_symlinks,
538545
dirs_exist_ok=dirs_exist_ok)
539546

547+
if hasattr(stat, 'FILE_ATTRIBUTE_REPARSE_POINT'):
548+
# Special handling for directory junctions to make them behave like
549+
# symlinks for shutil.rmtree, since in general they do not appear as
550+
# regular links.
551+
def _rmtree_isdir(entry):
552+
try:
553+
st = entry.stat(follow_symlinks=False)
554+
return (stat.S_ISDIR(st.st_mode) and not
555+
(st.st_file_attributes & stat.FILE_ATTRIBUTE_REPARSE_POINT
556+
and st.st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT))
557+
except OSError:
558+
return False
559+
560+
def _rmtree_islink(path):
561+
try:
562+
st = os.lstat(path)
563+
return (stat.S_ISLNK(st.st_mode) or
564+
(st.st_file_attributes & stat.FILE_ATTRIBUTE_REPARSE_POINT
565+
and st.st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT))
566+
except OSError:
567+
return False
568+
else:
569+
def _rmtree_isdir(entry):
570+
try:
571+
return entry.is_dir(follow_symlinks=False)
572+
except OSError:
573+
return False
574+
575+
def _rmtree_islink(path):
576+
return os.path.islink(path)
577+
540578
# version vulnerable to race conditions
541579
def _rmtree_unsafe(path, onerror):
542580
try:
@@ -547,11 +585,7 @@ def _rmtree_unsafe(path, onerror):
547585
entries = []
548586
for entry in entries:
549587
fullname = entry.path
550-
try:
551-
is_dir = entry.is_dir(follow_symlinks=False)
552-
except OSError:
553-
is_dir = False
554-
if is_dir:
588+
if _rmtree_isdir(entry):
555589
try:
556590
if entry.is_symlink():
557591
# This can only happen if someone replaces
@@ -681,7 +715,7 @@ def onerror(*args):
681715
os.close(fd)
682716
else:
683717
try:
684-
if os.path.islink(path):
718+
if _rmtree_islink(path):
685719
# symlinks to directories are forbidden, see bug #1669
686720
raise OSError("Cannot call rmtree on a symbolic link")
687721
except OSError:

0 commit comments

Comments
 (0)