Skip to content

stream: limit unix read/write to 2GB at a time#1501

Closed
vtjnash wants to merge 3 commits intolibuv:v1.xfrom
vtjnash:jn/mach-read-write-limit
Closed

stream: limit unix read/write to 2GB at a time#1501
vtjnash wants to merge 3 commits intolibuv:v1.xfrom
vtjnash:jn/mach-read-write-limit

Conversation

@vtjnash
Copy link
Copy Markdown
Member

@vtjnash vtjnash commented Aug 21, 2017

On some platforms, notably OSX, read/readv/write/writev all return
an EINVAL if the requested number of bytes is greater than INT32_MAX.

We therefore limit each such call to 2GB on all platforms. On
platforms where this is not an issue, the overhead of an extra syscall
for every 2GB of read/write is negligible.

(For original PR and issue report, see JuliaLang#33)

Fixed link: JuliaAttic/libuv-archive#33

Copy link
Copy Markdown
Member

@bnoordhuis bnoordhuis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Learn something new every day. Thanks for the PR.

Since it's a MacOS-only issue, would it make sense to put the code in #ifdef guards?

Aside: a regression test would be nice!

Comment thread src/unix/stream.c Outdated
Comment thread src/unix/stream.c Outdated
Comment thread src/unix/stream.c Outdated
Comment thread src/unix/stream.c Outdated
@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Aug 22, 2017

Since it's a MacOS-only issue, would it make sense to put the code in #ifdef guards?

I'm not sure. On the original issue, it was noted that it it's largely inconsequential to do a syscall every 2GB. Also, on Linux, we can see from testing that it doesn't actually do larger writes, it's simply just better at reporting:

julia> ccall(:write, UInt, (UInt32, Ptr{UInt8}, UInt), 19, a, sizeof(a)) # a is 2^32 + 1 bytes
0x000000007ffff000 # 2^31 - 1 bytes, rounded down by a page

@vtjnash vtjnash force-pushed the jn/mach-read-write-limit branch 2 times, most recently from 15852a9 to 1e99118 Compare August 22, 2017 20:08
@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Aug 22, 2017

Please take another look, thanks!

(While writing tests for this, I noticed uv_fs_write seems to have a similar bug, in particular, the implementation appears to assume that write can't fail for any reason (even EINTER), and it just continues with processing later buffers. But that will have to be addressed in a separate issue.)

libuv/src/unix/fs.c

Lines 1010 to 1012 in 564677d

req->bufs += req->nbufs;
nbufs -= req->nbufs;
total += result;

@saghul
Copy link
Copy Markdown
Member

saghul commented Aug 25, 2017

Any reason not to target this at v1.x?

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Aug 25, 2017

Mostly just that I don't have any use case for v1.x, since all of my work requires LEP-0005 (#1166). The tests for this could also be made more exhaustive once #1498 is merged.

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Sep 8, 2017

Bump. Is there anything else to be done to get this merged?

@bwijen
Copy link
Copy Markdown
Contributor

bwijen commented Sep 14, 2017

vtjnash commented 22 days ago

Please take another look, thanks!

(While writing tests for this, I noticed uv_fs_write seems to have a similar bug, in particular, the implementation appears to assume that write can't fail for any reason (even EINTER), and it just continues with processing later buffers. But that will have to be addressed in a separate issue.)

libuv/src/unix/fs.c

Lines 1010 to 1012 in 564677d

req->bufs += req->nbufs;
nbufs -= req->nbufs;
total += result;

Hi @vtjnash this is already adressed in issue #640
(But for some reason, it doesn't seem to land.)

@trevnorris
Copy link
Copy Markdown
Member

@saghul @bnoordhuis The PR nodejs/node#11641 will need to be altered to work around this bug. Mind giving a status update as to whether this (or the other) fix will land?

@vtjnash vtjnash force-pushed the jn/mach-read-write-limit branch from 4c570e5 to 4b325df Compare November 27, 2017 21:09
@vtjnash vtjnash changed the base branch from master to v1.x November 27, 2017 21:09
@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Nov 27, 2017

This is updated now to merge into v1.x, in hopes that it'll get some review now. I've also discovered that libuv on NT can't handle writes this large (NT returns some sort of internal error code instead), so I've backed off on the max size on that platform (perhaps I should add a note to the docs that you shouldn't attempt to write more than about 2^30 bytes at a time?).

@bwijen
Copy link
Copy Markdown
Contributor

bwijen commented Nov 28, 2017

I've also discovered that libuv on NT can't handle writes this large (NT returns some sort of internal error code instead), so I've backed off on the max size on that platform (perhaps I should add a note to the docs that you shouldn't attempt to write more than about 2^30 bytes at a time?).

I once suggested a public implementation for uv_getiovmax in libuv #802.
I guess this is the answer to the windows implementation?

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Nov 28, 2017

Not really, since this isn't a limit on the number of iovcnt, and there is no maximum uv_iovcnt for linux (the limit is currently 1 for Windows). The situation with the code usage is also a bit different on linux vs windows, since the libuv API uses a long for nbytes (like the respective kernels), so you can do directly do 64-bit writes on Unix; but on Windows, the client already needs to do extra work to split them up manually into 2^30 byte chunks to fit into the libuv API.

@bwijen
Copy link
Copy Markdown
Contributor

bwijen commented Nov 28, 2017

Understood. (Thanks for the explanation!)

@bzoz
Copy link
Copy Markdown
Member

bzoz commented Nov 29, 2017

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Nov 29, 2017

What's the buildbot configurations? A lot of them are failing to calloc a measly 2 GB region, including the 64-bit hosts. A few are hitting a timeout (writing 2 GB of data is pretty hard on the kernel: it's fast on macOS and linux, but very slow on NT. I haven't timed FreeBSD to see where it falls, but it looks like it is hitting the timeout limit.). This is definitely a pretty hard bug to test, since it requires a very large malloc region on NT (on recent posix systems, I could alter the tests to use writev to amplify the data transfer size). The only notable build failure was on zos (INT32_MAX is C99).

also a few failures that seem unrelated:
https://ci.nodejs.org/job/libuv-test-commit-windows/698/nodes=win2012r2-vs2017/console
https://ci.nodejs.org/job/libuv-test-commit-linux/637/nodes=centos6-64/console
https://ci.nodejs.org/job/libuv-test-commit-linux/637/nodes=ppcle-ubuntu1404/console

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Dec 6, 2017

Alright, I've tried a trick to get 32-bit machines to give me a large allocation. This won't solve the issue with the x64 build bot configuration (what is their configuration?), but will hopefully get a few more passing.

Comment thread test/echo-server.c Outdated
@bzoz
Copy link
Copy Markdown
Member

bzoz commented Dec 6, 2017

@jBarz
Copy link
Copy Markdown
Contributor

jBarz commented Dec 6, 2017

I recommend the following change for z/OS.
Also, there is no anonymous map functionality on z/OS.

--- a/src/unix/internal.h
+++ b/src/unix/internal.h
@@ -30,10 +30,6 @@
 #include <fcntl.h>  /* O_CLOEXEC, may be */
 #include <stdio.h>

-#ifndef INT32_MAX
-#define INT32_MAX (2147483647)
-#endif
-
 #if defined(__STRICT_ANSI__)
 # define inline __inline
 #endif
diff --git a/uv.gyp b/uv.gyp
index 96fb801..72d18b4 100644
--- a/uv.gyp
+++ b/uv.gyp
@@ -28,7 +28,7 @@
               '_AE_BIMODAL',
               'PATH_MAX=255'
             ],
-            'cflags': [ '-qxplink' ],
+            'cflags': [ '-qxplink', '-qlanglvl=extc99' ],
             'ldflags': [ '-qxplink' ],
           }]
         ],

@vtjnash vtjnash force-pushed the jn/mach-read-write-limit branch from 074ed1e to 810d900 Compare December 13, 2017 05:52
@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Dec 13, 2017

Updated again to work even harder to try to make most of the CI configurations capable of running this test.

@santigimeno
Copy link
Copy Markdown
Member

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Dec 14, 2017

Hm, that's interesting, apparently many headers have #define SIZE_MAX LONG_MAX, but don't define LONG_MAX. It's surprisingly hard to compute SSIZE_MAX. I can just change this to limit it to INT32_MAX.

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Dec 14, 2017

I don't understand the smartos failure. Did they accidentally release a bad header for mmap (missing MAP_FILE) in v14, or is there something wrong with that CI container?

@cjihrig
Copy link
Copy Markdown
Contributor

cjihrig commented Dec 14, 2017

It looks like MAP_FILE does not exist on that platform. This is how fish-shell got around the problem: fish-shell/fish-shell#3340, fish-shell/fish-shell@35bee00

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Dec 15, 2017

Yeah, it's just weird that it's only missing on v15. The tests appear to have passed on smartos v14 and v16. Can I have CI run again, please?

@vtjnash vtjnash force-pushed the jn/mach-read-write-limit branch from 5d2dbed to cf0d1a2 Compare January 9, 2020 22:23
@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Jan 9, 2020

How would that error look like, so applications know what to do?

On Apple, you'd currently get EINVAL. On Linux, this already worked (it just truncates the indivisual writes like I do in this PR). On Windows, you'd currently get an undocumented internal NT error code 0x800705aa (I think it essentially maps to ENOMEM). So if we wanted to make this consistently an error, I'd probably go with making it EINVAL everywhere. In the PR, we've instead essentially gone with doing the Linux behavior (of splitting the writes).

partially working CI: https://ci.nodejs.org/job/libuv-test-commit/1697/

Copy link
Copy Markdown
Member

@bnoordhuis bnoordhuis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only did a partial review because the ABI breakage needs to be worked out first.

Comment thread docs/src/misc.rst
.. warning:: It is discouraged to set `len` to a large value as that may
result in spurious failures. Specifically, Windows may fail on
writes larger than about 511 MB, and various Unicies may fail
on IO larger than about 2 GB. Instead it is generally better to
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
on IO larger than about 2 GB. Instead it is generally better to
on I/O larger than about 2 GB. Instead it is generally better to

Comment thread include/uv.h
#define UV_STREAM_FIELDS \
/* number of bytes queued for writing */ \
size_t write_queue_size; \
uint64_t write_queue_size; \
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks ABI on 32-bits architectures, as does changing the return type of uv_stream_get_write_queue_size().

@vtjnash
Copy link
Copy Markdown
Member Author

vtjnash commented Jan 15, 2020

the ABI breakage needs to be worked out first

Yeah, as the bugfix keeps snowballing, I'm starting to feel like this can't entirely land in v1.x at all. Instead, I'm thinking we should just return an error code when you exceed the ability for libuv to represent the operation correctly or the OS to process it (not breaking because it's already broken, it'd just be more consistently handled). Then in v2 we can revisit this and consider whether more APIs should use 64-bit numbers.

@bnoordhuis
Copy link
Copy Markdown
Member

That sounds like an acceptable way forward to me. To recap: for v1.x that means returning UV_EINVAL when > 2 GB on Unices and > 511 MB (?) on Windows?

bnoordhuis added a commit to bnoordhuis/io.js that referenced this pull request Jan 18, 2020
Change the type of `Buffer::kMaxLength` to size_t because upcoming
changes in V8 will allow typed arrays > 2 GB on 64 bits platforms.

Not all platforms handle file reads and writes > 2 GB though so keep
enforcing the 2 GB typed array limit for I/O operations.

Fixes: nodejs#31399
Refs: libuv/libuv#1501
Trott pushed a commit to nodejs/node that referenced this pull request Jan 21, 2020
Change the type of `Buffer::kMaxLength` to size_t because upcoming
changes in V8 will allow typed arrays > 2 GB on 64 bits platforms.

Not all platforms handle file reads and writes > 2 GB though so keep
enforcing the 2 GB typed array limit for I/O operations.

Fixes: #31399
Refs: libuv/libuv#1501

PR-URL: #31406
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: David Carlier <devnexen@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Shelley Vohr <codebytere@gmail.com>
codebytere pushed a commit to nodejs/node that referenced this pull request Feb 17, 2020
Change the type of `Buffer::kMaxLength` to size_t because upcoming
changes in V8 will allow typed arrays > 2 GB on 64 bits platforms.

Not all platforms handle file reads and writes > 2 GB though so keep
enforcing the 2 GB typed array limit for I/O operations.

Fixes: #31399
Refs: libuv/libuv#1501

PR-URL: #31406
Reviewed-By: Richard Lau <riclau@uk.ibm.com>
Reviewed-By: David Carlier <devnexen@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Shelley Vohr <codebytere@gmail.com>
@anlexN
Copy link
Copy Markdown

anlexN commented Nov 16, 2020

ok, now we can support 64bit directly.

@vtjnash vtjnash mentioned this pull request May 25, 2021
vtjnash added a commit to vtjnash/libuv that referenced this pull request Mar 18, 2026
Because libuv truncates the result of every call to INT32_MAX, it needs
to internally limit operations to INT32_MAX to be safe to use libuv.
This isn't an API change, since these operations weren't guaranteed to
work, and in fact usually failed in bizare ways already. This is very
long in coming, since we've had a lot of compiler warnings about this
and several PRs to fix this open for a decade, but the main consumers
that usually fix things didn't care (nodejs is 32-bit and julia patched
this downstream more than a decade ago, though it did run into this
again recently by mistake with sendfile).

Replaces libuv#1501
vtjnash added a commit to vtjnash/libuv that referenced this pull request Mar 18, 2026
Because libuv truncates the result of every call to INT32_MAX, it needs
to internally limit operations to INT32_MAX to be safe to use libuv.
This isn't an API change, since these operations weren't guaranteed to
work, and in fact usually failed in bizare ways already. This is very
long in coming, since we've had a lot of compiler warnings about this
and several PRs to fix this open for a decade, but the main consumers
that usually fix things didn't care (nodejs is 32-bit and julia patched
this downstream more than a decade ago, though it did run into this
again recently by mistake with sendfile).

Replaces libuv#1501
Fixes libuv#3360
vtjnash added a commit to vtjnash/libuv that referenced this pull request Mar 18, 2026
Because libuv truncates the result of every call to INT32_MAX, it needs
to internally limit operations to INT32_MAX to be safe to use libuv.
This isn't an API change, since these operations weren't guaranteed to
work, and in fact usually failed in bizare ways already. This is very
long in coming, since we've had a lot of compiler warnings about this
and several PRs to fix this open for a decade, but the main consumers
that usually fix things didn't care (nodejs is 32-bit and julia patched
this downstream more than a decade ago, though it did run into this
again recently by mistake with sendfile).

Replaces libuv#1501
Fixes libuv#3360
vtjnash added a commit to vtjnash/libuv that referenced this pull request Mar 18, 2026
Because libuv truncates the result of every call to INT32_MAX, it needs
to internally limit operations to INT32_MAX to be safe to use libuv.
This isn't an API change, since these operations weren't guaranteed to
work, and in fact usually failed in bizare ways already. This is very
long in coming, since we've had a lot of compiler warnings about this
and several PRs to fix this open for a decade, but the main consumers
that usually fix things didn't care (nodejs is 32-bit and julia patched
this downstream more than a decade ago, though it did run into this
again recently by mistake with sendfile).

Replaces libuv#1501
Fixes libuv#3360
vtjnash added a commit to vtjnash/libuv that referenced this pull request Mar 18, 2026
Because libuv truncates the result of every call to INT32_MAX, it needs
to internally limit operations to INT32_MAX to be safe to use libuv.
This isn't an API change, since these operations weren't guaranteed to
work, and in fact usually failed in bizare ways already. This is very
long in coming, since we've had a lot of compiler warnings about this
and several PRs to fix this open for a decade, but the main consumers
that usually fix things didn't care (nodejs is 32-bit and julia patched
this downstream more than a decade ago, though it did run into this
again recently by mistake with sendfile).

Replaces libuv#1501
Fixes libuv#3360
@vtjnash vtjnash closed this Mar 19, 2026
vtjnash added a commit that referenced this pull request Mar 24, 2026
Because libuv truncates the result of every call to INT32_MAX, it needs
to internally limit operations to INT32_MAX to be safe to use libuv.
This isn't an API change, since these operations weren't guaranteed to
work, and in fact usually failed in bizare ways already. This is very
long in coming, since we've had a lot of compiler warnings about this
and several PRs to fix this open for a decade, but the main consumers
that usually fix things didn't care (nodejs is 32-bit and julia patched
this downstream more than a decade ago, though it did run into this
again recently by mistake with sendfile).

Replaces #1501
Fixes #3360
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

not-stale Issues that should never be marked stale

Projects

None yet

Development

Successfully merging this pull request may close these issues.