Skip to content

bpo-2628: support BLOCK mode for retrbinary#29337

Closed
btjbell wants to merge 1 commit into
python:mainfrom
btjbell:feature-issue-2628
Closed

bpo-2628: support BLOCK mode for retrbinary#29337
btjbell wants to merge 1 commit into
python:mainfrom
btjbell:feature-issue-2628

Conversation

@btjbell

@btjbell btjbell commented Oct 30, 2021

Copy link
Copy Markdown

Comment thread Lib/ftplib.py
# shutdown ssl layer
if _SSLSocket is not None and isinstance(conn, _SSLSocket):
conn.unwrap()
self.voidcmd('MODE %s' % self.transmissionmode)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I expect few servers support BLOCK mode, I would explicitly send "MODE" command only if self.transmissionmode != S

Comment thread Lib/test/test_ftplib.py
return self.close_when_done()
super(DummyDTPHandler, self).push(what.encode(self.encoding))

if type(what) != bytes:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggested change: isinstance(what, bytes)

Comment thread Lib/ftplib.py

Currently, the mode set is applied only when a client calls retrbinary.
"""
self.transmissionmode = mode

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably makes sense to raise ValueError if specified mode is not in ("S", "B").

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, does the RFC say anything about what should happen if you switch back to STREAM mode from BLOCK mode (knowing RFC-959 myself, I imagine not)? Should the dataconn socket be closed?

Comment thread Lib/ftplib.py
"""Close the persistent data connection."""
if self.dataconn:
self.dataconn.close()
self.dataconn = None

@giampaolo giampaolo Oct 31, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should call this also from close() method (important).

Also, "data connection" is a very well established concept in FTP, but 99% of the times applies to STREAMing mode. I would be more explicit and call this method _close_block_dataconn or _close_persistent_dataconn.

Comment thread Lib/ftplib.py
elif self.transmissionmode == 'B':
with self.transfercmd(cmd, rest) as conn:
while 1:
# Receive one byte at a time, not all 3 at once -- seriously

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? Is it because returned data may be < 3 bytes?

Comment thread Lib/ftplib.py
print('*suspect data*')
else:
self.close_dataconn()
raise NotImplementedError

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest ValueError("Invalid transmission mode %r" % self.transmissionmode)

Comment thread Lib/ftplib.py
The response code.
"""
self.voidcmd('TYPE I')
self.voidcmd('MODE S')

@giampaolo giampaolo Oct 31, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I expect few servers to implement BLOCK mode, I would avoid sending MODE command unless it's really necessary. I would change this to:

if self.transmissionmode != 'S':
     # BLOCK mode only applies to RETR. STOR does not support  it.
     self.voidcmd('MODE S')

Comment thread Doc/library/ftplib.rst

Specifies the mode in which to transmit files retrieved in BINARY transfer mode.
Legal values per RFC 959 are 'S' (STREAM, the default), 'B' (BLOCK) and
'C' (COMPRESSED). This library supports STREAM and BLOCK modes only.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should probably mention that BLOCK is supported for RETR (donwload) only.

Comment thread Lib/ftplib.py
if self.debugging:
print("*got more data than desired!* %d vs %d" % (len(buff), blocklength-sread))
sread = sread + len(buff)
data = data + buff

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding (+) chunks of bytes on every iteration is not very efficient. You could use a bytearray instead. Not tested:

        buff = bytearray()
        curr_length = blocklength
        while len(buff) < blocklength:
            chunk = conn.recv(curr_length)
            buff.extend(chunk)
            curr_length -= len(chunk)
        data = bytes(buff)

In order to avoid making this method too complex, perhaps it makes sense to encapsulate this logic in a utility function (again, not tested):

def _read_until(sock, length):
    if length <= 0:
        return b""
    buff = bytearray()
    curr_length = length
    while len(buff) < length:
        chunk = sock.recv(curr_length)
        buff.extend(chunk)
        curr_length -= len(chunk)
    return bytes(buff)

@giampaolo giampaolo Oct 31, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if the server sends us a big number (say size = 1G), we will hog RAM. I wonder if it would make sense to implement an additional, internal buffer, that calls callback function after a certain threshold (e.g. 1M) and empties the internal memory buffer.

Comment thread Lib/ftplib.py
passiveserver = True
# Disables https://bugs.python.org/issue43285 security if set to True.
trust_server_pasv_ipv4_address = False
transmissionmode = 'S'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

transmission_mode?

@bedevere-bot

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@iritkatriel

iritkatriel commented Nov 5, 2021

Copy link
Copy Markdown
Member

bpo-2628 was closed, should this be as well?

@btjbell btjbell closed this Nov 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants