I use WebClient to upload binary files. When there are only a few of them everything works great, but when there are more (e.g. several hundred), WebClient sometimes corrupts binary data.
I described the problem in more detail in this question on StackOverflow. I also created a repository containing tests that illustrate this issue.
I don't know what is the reason for this behavior but it doesn't seem entirely deterministic.
My use case is that I am sending binary files from one service (let's call it A) to another (let's call it B).
This is how I read these binary files in the service B:
@RestController
class FilesController {
@PostMapping(value = "/files")
Mono<List<String>> uploadFiles(@RequestBody Flux<Part> parts) {
return parts
.filter(FilePart.class::isInstance)
.map(FilePart.class::cast)
.flatMap(part -> DataBufferUtils.join(part.content())
.map(buffer -> {
byte[] data = new byte[buffer.readableByteCount()];
buffer.read(data);
DataBufferUtils.release(buffer);
return Base64.getEncoder().encodeToString(data);
})
)
.collectList();
}
}And this is how I send these files in the service A (I wrote some tests to better illustrate the issue).
public class BinaryUploadTest {
private final List<String> sentBytes = new CopyOnWriteArrayList<>();
@BeforeEach
void before() {
sentBytes.clear();
}
/**
* this test passes all the time
*/
@Test
void shouldUpload5Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(5);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
/**
* this test fails most of the time
*/
@Test
void shouldUpload1000Files() {
// given
MultiValueMap<String, HttpEntity<?>> body = buildResources(1000);
// when
List<String> receivedBytes = sendPostRequest(body);
// then
assertEquals(sentBytes, receivedBytes);
}
private List<String> sendPostRequest(MultiValueMap<String, HttpEntity<?>> body) {
return WebClient.builder().build().post()
.uri("http://localhost:8080/files")
.contentType(MediaType.MULTIPART_FORM_DATA)
.body(BodyInserters.fromMultipartData(body))
.retrieve()
.bodyToMono(new ParameterizedTypeReference<List<String>>() {
})
.block();
}
private MultiValueMap<String, HttpEntity<?>> buildResources(int numberOfResources) {
MultipartBodyBuilder builder = new MultipartBodyBuilder();
for (int i = 0; i < numberOfResources; i++) {
builder.part("item-" + i, buildResource(i));
}
return builder.build();
}
private ByteArrayResource buildResource(int index) {
byte[] bytes = randomBytes();
sentBytes.add(Base64.getEncoder().encodeToString(bytes)); // keeps track of what has been sent
return new ByteArrayResource(bytes) {
@Override
public String getFilename() {
return "filename-" + index;
}
};
}
private byte[] randomBytes() {
byte[] bytes = new byte[ThreadLocalRandom.current().nextInt(16, 32)];
ThreadLocalRandom.current().nextBytes(bytes);
return bytes;
}
}As you can see, the only difference between these two tests is the number of files uploaded. The first test always works, and the second one doesn't work most of the time. When I was analyzing why these bytes do not match, I noticed that sometimes a few extra bytes appear at the end (something like padding).
Interestingly, when I use WebTestClient, the same problem does not occur, as shown in this test.
I use
WebClientto upload binary files. When there are only a few of them everything works great, but when there are more (e.g. several hundred),WebClientsometimes corrupts binary data.I described the problem in more detail in this question on StackOverflow. I also created a repository containing tests that illustrate this issue.
I don't know what is the reason for this behavior but it doesn't seem entirely deterministic.
My use case is that I am sending binary files from one service (let's call it
A) to another (let's call itB).This is how I read these binary files in the service
B:And this is how I send these files in the service
A(I wrote some tests to better illustrate the issue).As you can see, the only difference between these two tests is the number of files uploaded. The first test always works, and the second one doesn't work most of the time. When I was analyzing why these bytes do not match, I noticed that sometimes a few extra bytes appear at the end (something like padding).
Interestingly, when I use
WebTestClient, the same problem does not occur, as shown in this test.