Repository navigation
ByteBuf.nioBuffer() returns a buffer with all zeros #215
Description
Hi,
I'm running into a weird issue where calling nioBuffer on a ByteBuf received from lmdb-java results in a NIO ByteBuffer with the desired size but a value of all zeros.
I'm on Netty 4.1.90.Final and lmdbjava 0.8.3 and JDK 19 . The repro is as follows:
package foo;
import io.netty.buffer.ByteBufUtil;
import io.netty.buffer.PooledByteBufAllocator;
import io.netty.buffer.Unpooled;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.lmdbjava.ByteBufProxy;
import org.lmdbjava.DbiFlags;
import org.lmdbjava.Env;
import java.io.File;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.junit.jupiter.api.Assertions.assertNotNull;
public class ByteBufMinimalReproTest {
@TempDir
File tmp;
@Test
void minimalRepro() {
var env = Env.create(ByteBufProxy.PROXY_NETTY)
.setMapSize(10_485_760)
.setMaxDbs(1)
.open(tmp);
var db = env.openDbi("testdb", DbiFlags.MDB_CREATE);
var alloc = PooledByteBufAllocator.DEFAULT;
var key = alloc.directBuffer(env.getMaxKeySize());
var value = alloc.directBuffer(700);
key.writeCharSequence("greeting", UTF_8);
value.writeCharSequence("Hello World", UTF_8);
System.out.println("Sanity -- Key");
System.out.println(ByteBufUtil.prettyHexDump(key));
System.out.println("Sanity -- Key NIO");
System.out.println(ByteBufUtil.prettyHexDump(Unpooled.wrappedBuffer(key.nioBuffer())));
System.out.println("Sanity -- Value");
System.out.println(ByteBufUtil.prettyHexDump(value));
System.out.println("Sanity -- Value NIO");
System.out.println(ByteBufUtil.prettyHexDump(Unpooled.wrappedBuffer(value.nioBuffer())));
db.put(key, value);
// need a transaction to read values.
try (var txn = env.txnRead()) {
var found = db.get(txn, key);
assertNotNull(found);
System.out.println("Found Value");
System.out.println(ByteBufUtil.prettyHexDump(found));
System.out.println("Found Value - NIO (Bad: all zeros)");
System.out.println(ByteBufUtil.prettyHexDump(Unpooled.wrappedBuffer(
found.nioBuffer())));
var fetchedVal = txn.val();
System.out.println("val via txn");
System.out.println(ByteBufUtil.prettyHexDump(fetchedVal));
System.out.println("val via txn - NIO (Bad: all zeros)");
System.out.println(ByteBufUtil.prettyHexDump(Unpooled.wrappedBuffer(
txn.val().nioBuffer())));
}
env.close();
}
}
Which prints:
Sanity -- Key
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 67 72 65 65 74 69 6e 67 |greeting |
+--------+-------------------------------------------------+----------------+
Sanity -- Key NIO
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 67 72 65 65 74 69 6e 67 |greeting |
+--------+-------------------------------------------------+----------------+
Sanity -- Value
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 65 6c 6c 6f 20 57 6f 72 6c 64 |Hello World |
+--------+-------------------------------------------------+----------------+
Sanity -- Value NIO
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 65 6c 6c 6f 20 57 6f 72 6c 64 |Hello World |
+--------+-------------------------------------------------+----------------+
Found Value
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 65 6c 6c 6f 20 57 6f 72 6c 64 |Hello World |
+--------+-------------------------------------------------+----------------+
Found Value - NIO (Bad: all zeros)
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 00 00 00 00 00 00 00 00 00 00 00 |........... |
+--------+-------------------------------------------------+----------------+
val via txn
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 65 6c 6c 6f 20 57 6f 72 6c 64 |Hello World |
+--------+-------------------------------------------------+----------------+
val via txn - NIO (Bad: all zeros)
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 00 00 00 00 00 00 00 00 00 00 00 |........... |
+--------+-------------------------------------------------+----------------+
My repro is using the prettyHexDump from ByteBuf, which is why i need to re-wrap nioBuffer. I'm pretty sure this isn't causing any problems because when I write the bytebuf to file, I see all zeros.
The reason I'm doing nioBuffer on this netty buffer is because I need to pass it to Zstd java for decompression.
I'm new to both lmdb java and netty, and it's possible i'm doing something wrong, but I'd appreciate your help with this.
Activity
2018ik commentedon Apr 11, 2023
Hey, this is because on get, lmdbjava replaces the address field of ByteBuf with the lmdb pointer address directly (https://github.com/lmdbjava/lmdbjava/blob/master/src/main/java/org/lmdbjava/ByteBufProxy.java#L153-L154) to avoid a copy.
nioBuffer()returns a ByteBuffer created from thismemoryobject https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/PooledByteBuf.java#L192 which is actually a nio ByteBuffer that itself has an address field, located at a pre-defined offset behind the ByteBuf address, and overlaps with the ByteBuf memory.On a normal
writeBytes()call, data is copied to the ByteBuf address, so thememoryByteBuffer will contain this data as well. However, since lmdbjava never actually writes any data to the original ByteBuf address, and instead replaces it entirely, thememoryByteBuffer remains empty (which is why you see 0s).Looks like this can be fixed by also setting the
memoryByteBuffer address onByteBufProxy.out. In the meantime I don't know if there is an easy workaround without using unsafe/reflection besides copying the data and re-wrapping the buffer.bp-alex commentedon Apr 24, 2023
@2018ik thanks for the summary. I don't use Netty so I don't have any particular view on the correct way of going about this. Suggestions welcome.
bp-alex commentedon Feb 17, 2025
This ticket is now approaching two years old. It would be great if someone could submit a PR that addresses it.
feat(lmdbjava#215): address review — nioBufferCopy supplier overload,…