Skip to content
This repository was archived by the owner on Jul 16, 2022. It is now read-only.

Passes kwargs through AtomicWriter and uses io.open as opener#38

Merged
untitaker merged 4 commits into
untitaker:masterfrom
lorengordon:issue-37
Aug 24, 2018
Merged

Passes kwargs through AtomicWriter and uses io.open as opener#38
untitaker merged 4 commits into
untitaker:masterfrom
lorengordon:issue-37

Conversation

@lorengordon

@lorengordon lorengordon commented Aug 23, 2018

Copy link
Copy Markdown
Contributor

This enables users to pass any option supported by tempfile.NamedTemporaryFile() to atomic_write(). However, mode, dir, and delete continue to be overwritten as they were previously, to enforce the atomic write behavior.

Let me know if this approach is alright. I'm happy to adjust as needed.

Fixes #37

@untitaker

Copy link
Copy Markdown
Owner

tbh i'd much rather pass through only the options that io.open supports, i.e. excluding prefix and suffix

@untitaker untitaker self-requested a review August 23, 2018 15:55
@untitaker untitaker self-assigned this Aug 23, 2018

@untitaker untitaker left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm except that we shouldn't expose prefix and suffix

@lorengordon

Copy link
Copy Markdown
Contributor Author

I can pop prefix and suffix, if that's what you mean. Seems a bit fragile to me, since it implies keeping up with other changes in the upstream libraries.

I don't think tempfile.NamedTemporaryFile is really using io.open under the covers... well, except on Python 3 since io.open is an alias for open. If you wanted this to work more similarly across Python 2 and Python 3, could maybe find a way to explicitly use io.open as the opener and pass the kwargs there instead...

@untitaker

untitaker commented Aug 23, 2018

Copy link
Copy Markdown
Owner

I was more thinking about a whitelist of options, or removing kwargs entirely in favor of explicit options (probably nicer for api docs)

@lorengordon

Copy link
Copy Markdown
Contributor Author

I guess I'm having trouble following what you're proposing. If you'd rather just have a newline parameter, I can try that. That's a little trickier than just leaving it up to the user, as I have it now, since the behavior on Python 2 is different (at least, without also changing to use the same opener).

Out of curiosity, how would prefix or suffix break the atomic write functionality? It seems like they would work fine, though I don't know why a user would specify them since the tempfile should just get removed anyway...

@untitaker

Copy link
Copy Markdown
Owner

@lorengordon I just want to keep the API surface minimal and I haven't seen a good usecase for changing the defaults. I do see the usecase for setting newline though.

I wonder if that's worth the effort though... I'll think about it

@lorengordon

Copy link
Copy Markdown
Contributor Author

I see. Ok. Well if using the same opener on python 2 and python 3 is desirable, here's an option... No more prefix/suffix issues, and only the io.open kwargs would work...

diff --git a/atomicwrites/__init__.py b/atomicwrites/__init__.py
index 108af8e..aa48a43 100644
--- a/atomicwrites/__init__.py
+++ b/atomicwrites/__init__.py
@@ -1,4 +1,5 @@
 import contextlib
+import io
 import os
 import sys
 import tempfile
@@ -163,10 +164,10 @@ class AtomicWriter(object):
         '''Return the temporary file to use.'''
         if dir is None:
             dir = os.path.normpath(os.path.dirname(self._path))
-        kwargs['dir'] = dir
-        kwargs['delete'] = False
+        descriptor, _ = tempfile.mkstemp(dir=dir)
         kwargs['mode'] = self._mode
-        return tempfile.NamedTemporaryFile(**kwargs)
+        kwargs['file'] = descriptor
+        return io.open(**kwargs)

     def sync(self, f):
         '''responsible for clearing as many file caches as possible before

@untitaker

Copy link
Copy Markdown
Owner

yuup that looks fine, thanks

@lorengordon

lorengordon commented Aug 24, 2018

Copy link
Copy Markdown
Contributor Author

Ok, updated the pr to use io.open as the opener.

Comment thread atomicwrites/__init__.py Outdated
try:
success = False
with get_fileobject() as f:
with get_fileobject(**self._tempfile_kwargs) as f:

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you rename this param to open_kwargs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite right. Meant to do that before. Fixed!

@untitaker

Copy link
Copy Markdown
Owner

I'll check later how we can fix the tests

@lorengordon

Copy link
Copy Markdown
Contributor Author

Yeah, looks like io.open makes that attribute read-only somehow or other. I can look into it late afternoon or this weekend if you don't get to it first.

ame is not writeable on _io.TextIOWrapper objects
@lorengordon

Copy link
Copy Markdown
Contributor Author

Just updated the test with one option. The second exception would be raised in a slightly different place, commit, instead of rollback. Not sure if that's close enough to what you needed from the test. However, I'm not sure that an exception in rollback would be raised anyway, since that's wrapped in a try/except block and the except is just pass. But I'm probably misunderstanding the setup.

@lorengordon

Copy link
Copy Markdown
Contributor Author

Hrm, the failure on py27 is more troubling. Introducing io.open here would likely break a lot of users on py27, since they'd need to ensure they were writing unicode instead of bytes. I missed the warning right at the top of the Python2 io.open docs...

Note

Since this module has been designed primarily for Python 3.x, you have to be aware that all uses of >“bytes” in this document refer to the str type (of which bytes is an alias), and all uses of “text” refer to >the unicode type. Furthermore, those two types are not interchangeable in the io APIs.

Enforcing unicode_literals does work, on the client side (in the tests)...

from __future__ import unicode_literals

Lemme know if you want to press forward or revert that.

@untitaker

Copy link
Copy Markdown
Owner

Please don't use that future import but prefer explicit u"" and b"", and only when it's needed. There are some APIs in Python 2 where passing in unicode strings makes for weird behavior. Probably not a problem here but I'd like to avoid this import in general.

@lorengordon

Copy link
Copy Markdown
Contributor Author

Sure thing. My preference has always been the other way around, and to just fix my python2 code accordingly. But, up to you of course.

@untitaker

Copy link
Copy Markdown
Owner

Cool, thx. Ping me if you need a release.

@untitaker untitaker merged commit 76818b3 into untitaker:master Aug 24, 2018
@lorengordon

Copy link
Copy Markdown
Contributor Author

I generally only install from pypi, and usually pin versions/ranges on dependencies, so a release would be great. Thanks! Always great working with a responsive maintainer! :D

@lorengordon lorengordon deleted the issue-37 branch August 24, 2018 18:00
@untitaker

untitaker commented Aug 24, 2018 via email

Copy link
Copy Markdown
Owner

@lorengordon

Copy link
Copy Markdown
Contributor Author

Yup, understood. Appreciate it!

@lorengordon lorengordon changed the title Passes kwargs through AtomicWriter to tempfile.NamedTemporaryFile Passes kwargs through AtomicWriter and uses io.open as opener Aug 24, 2018
GlenWalker added a commit to GlenWalker/python-atomicwrites that referenced this pull request Jan 30, 2019
The changes in untitaker#38 broke the ability to use prefix, suffix and bufsize arguments when creating the tempfile. This commit restores the ability.
GlenWalker added a commit to GlenWalker/python-atomicwrites that referenced this pull request Feb 1, 2019
The changes in untitaker#38 broke the ability to use prefix, suffix and bufsize arguments when creating the tempfile. This commit restores the ability.
untitaker pushed a commit that referenced this pull request Feb 1, 2019
The changes in #38 broke the ability to use prefix, suffix and bufsize arguments when creating the tempfile. This commit restores the ability.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants