Skip to content

[Java] Apache Arrow API fails with exceeded limit on max bytes to buffer when fetching data from snowflake  #206

Closed as duplicate
Closed as duplicate
[Java] Apache Arrow API fails with exceeded limit on max bytes to buffer when fetching data from snowflake #206
Labels
Type: bugSomething isn't working

Description

@badhana
badhana
opened on Apr 11, 2023

Describe the bug, including details regarding any error messages, version, and platform.

snowflake-jdbc driver fails with openjdk 17 for memory allocation when using arrow Apache api. This API provide high performance but not compatibile with jdk 17.

References for this issue available here
--> snowflakedb/snowflake-jdbc#589
--> Apache Arrow Api library reference https://arrow.apache.org/docs/java/memory.html

2023-04-11 10:58:05.248 ERROR [,NONE,NONE] 1784 [ctor-http-nio-5] org.springframework.boot.autoconfigure.web.reactive.error.AbstractErrorWebExceptionHandler [CompositeLog.java:102]
: [28d1512f-2] 500 Server Error for HTTP GET "getRequest/from-snowflake"

Error description below:

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 262144
at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Original Stack Trace:
at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99)
at org.springframework.core.io.buffer.LimitedDataBufferList.updateCount(LimitedDataBufferList.java:92)
at org.springframework.core.io.buffer.LimitedDataBufferList.add(LimitedDataBufferList.java:58)
at reactor.core.publisher.MonoCollect$CollectSubscriber.onNext(MonoCollect.java:103)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:379)
at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:411)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:734)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)

2023-04-11 10:58:05.254 ERROR [,NONE,NONE] 1784 [ctor-http-nio-5] reactor.netty.channel.ChannelOperationsHandler [Loggers.java:324] : [3d4979e3-1, L:/127.0.0.1:64525 ! R:localhost/
127.0.0.1:8081] Error was received while reading the incoming data. The connection will be closed.

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:147)
at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
at io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)
at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:382)
at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:411)
at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:734)
at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)

2023-04-11 10:58:05.255 WARN [,NONE,NONE] 1784 [ctor-http-nio-5] reactor.netty.channel.FluxReceive [Loggers.java:299] : [3d4979e3-1, L:/127.0.0.1:64525 ! R:localhost/127.0.
0.1:8081] An exception has been observed post termination, use DEBUG level to see the full stack: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1

Component(s)

Java

Activity

  1. badhana
    added
    Type: bugSomething isn't working
    on Apr 11, 2023
  2. raulcd
    changed the title [-]Apache Arrow API fails with exceeded limit on max bytes to buffer when fetching data from snowflake [/-] [+][Java] Apache Arrow API fails with exceeded limit on max bytes to buffer when fetching data from snowflake [/+] on Apr 12, 2023
  3. assignUser
    transferred this issue fromapache/arrowon Nov 26, 2024

  4. lidavidm commented on Mar 13, 2025

    @lidavidm
    lidavidm
    on Mar 13, 2025
    Member

    It's not very clear what this stack trace has to do with Arrow. From reading the linked report, it seems the issue is that you need to add a JVM parameter (--add-opens)? Unfortunately, this is by design. Until/unless someone can contribute a rewrite of the memory system that uses the new FFM APIs and can articulate a migration path, this is currently something we can't change.

    (Note that even with FFM, you will likely still need a JVM argument, from what I understand; Java simply has decided to lock down all APIs that may access native memory directly, and unfortunately Arrow depends on this.)

    Closing as a duplicate of the FFM issue, please move discussion there (#163).

  5. lidavidm
    closed this as a duplicateof [Java] Implement arrow-memory-ffm #163 on Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions