Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/pom.xml b/pom.xml
index de24b12e73..9d89616356 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,7 +109,7 @@ under the License.
10.23.0true2.37.0
- 3.49.2
+ 3.49.31.5.18none-Xdoclint:none
From bf7cadad33921e6333f00302807b0879df128942 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 12 May 2025 10:44:31 +0200
Subject: [PATCH 002/147] MINOR: Bump
com.google.api.grpc:proto-google-common-protos from 2.54.1 to 2.56.0 (#750)
Bumps
[com.google.api.grpc:proto-google-common-protos](https://github.com/googleapis/sdk-platform-java)
from 2.54.1 to 2.56.0.
Release notes
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
flight/flight-core/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flight/flight-core/pom.xml b/flight/flight-core/pom.xml
index 757de85769..92f58b84de 100644
--- a/flight/flight-core/pom.xml
+++ b/flight/flight-core/pom.xml
@@ -134,7 +134,7 @@ under the License.
com.google.api.grpcproto-google-common-protos
- 2.54.1
+ 2.56.0test
From 93b7fdc2113c9c21e359edc910a6b44061b0ee50 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Mon, 12 May 2025 10:47:23 +0200
Subject: [PATCH 003/147] MINOR: Bump com.github.ben-manes.caffeine:caffeine
from 3.1.8 to 3.2.0 (#747)
Bumps
[com.github.ben-manes.caffeine:caffeine](https://github.com/ben-manes/caffeine)
from 3.1.8 to 3.2.0.
Release notes
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
---
flight/flight-sql-jdbc-core/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/flight/flight-sql-jdbc-core/pom.xml b/flight/flight-sql-jdbc-core/pom.xml
index a95fbaca30..04c4691faf 100644
--- a/flight/flight-sql-jdbc-core/pom.xml
+++ b/flight/flight-sql-jdbc-core/pom.xml
@@ -151,7 +151,7 @@ under the License.
com.github.ben-manes.caffeinecaffeine
- 3.1.8
+ 3.2.0
From 8c74ced9c03e4aba1d1c5fea5f223a42be234dcd Mon Sep 17 00:00:00 2001
From: Gang Wu
Date: Tue, 13 May 2025 23:43:14 +0800
Subject: [PATCH 004/147] MINOR: Bump version to 19.0.0-SNAPSHOT (#754)
---
adapter/avro/pom.xml | 2 +-
adapter/jdbc/pom.xml | 2 +-
adapter/orc/pom.xml | 2 +-
algorithm/pom.xml | 2 +-
bom/pom.xml | 4 ++--
c/pom.xml | 2 +-
compression/pom.xml | 2 +-
dataset/pom.xml | 2 +-
flight/flight-core/pom.xml | 2 +-
flight/flight-integration-tests/pom.xml | 2 +-
flight/flight-sql-jdbc-core/pom.xml | 2 +-
flight/flight-sql-jdbc-driver/pom.xml | 2 +-
flight/flight-sql/pom.xml | 2 +-
flight/pom.xml | 2 +-
format/pom.xml | 2 +-
gandiva/pom.xml | 2 +-
memory/memory-core/pom.xml | 2 +-
memory/memory-netty-buffer-patch/pom.xml | 2 +-
memory/memory-netty/pom.xml | 2 +-
memory/memory-unsafe/pom.xml | 2 +-
memory/pom.xml | 2 +-
performance/pom.xml | 2 +-
pom.xml | 4 ++--
tools/pom.xml | 2 +-
vector/pom.xml | 2 +-
25 files changed, 27 insertions(+), 27 deletions(-)
diff --git a/adapter/avro/pom.xml b/adapter/avro/pom.xml
index cf9d353330..827d19f2a2 100644
--- a/adapter/avro/pom.xml
+++ b/adapter/avro/pom.xml
@@ -23,7 +23,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOT../../pom.xml
diff --git a/adapter/jdbc/pom.xml b/adapter/jdbc/pom.xml
index f92863fbd6..2f621d7a05 100644
--- a/adapter/jdbc/pom.xml
+++ b/adapter/jdbc/pom.xml
@@ -23,7 +23,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOT../../pom.xml
diff --git a/adapter/orc/pom.xml b/adapter/orc/pom.xml
index e60a7ceb3c..e3ae7d5163 100644
--- a/adapter/orc/pom.xml
+++ b/adapter/orc/pom.xml
@@ -23,7 +23,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOT../../pom.xml
diff --git a/algorithm/pom.xml b/algorithm/pom.xml
index e934eb7b22..898c2605b6 100644
--- a/algorithm/pom.xml
+++ b/algorithm/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-algorithmArrow Algorithms
diff --git a/bom/pom.xml b/bom/pom.xml
index 80f03d1205..61b452b9c1 100644
--- a/bom/pom.xml
+++ b/bom/pom.xml
@@ -29,7 +29,7 @@ under the License.
org.apache.arrowarrow-bom
- 18.3.0
+ 19.0.0-SNAPSHOTpomArrow Bill of Materials
@@ -68,7 +68,7 @@ under the License.
scm:git:https://github.com/apache/arrow-java.gitscm:git:https://github.com/apache/arrow-java.git
- v18.3.0
+ mainhttps://github.com/apache/arrow-java/tree/${project.scm.tag}
diff --git a/c/pom.xml b/c/pom.xml
index 290cb561c1..c90b6dc0ef 100644
--- a/c/pom.xml
+++ b/c/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-c-data
diff --git a/compression/pom.xml b/compression/pom.xml
index 3443f11478..6f60eb7d0a 100644
--- a/compression/pom.xml
+++ b/compression/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-compressionArrow Compression
diff --git a/dataset/pom.xml b/dataset/pom.xml
index efbe310ea2..6e56d555b7 100644
--- a/dataset/pom.xml
+++ b/dataset/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-dataset
diff --git a/flight/flight-core/pom.xml b/flight/flight-core/pom.xml
index 92f58b84de..24beac391e 100644
--- a/flight/flight-core/pom.xml
+++ b/flight/flight-core/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-flight
- 18.3.0
+ 19.0.0-SNAPSHOTflight-core
diff --git a/flight/flight-integration-tests/pom.xml b/flight/flight-integration-tests/pom.xml
index e7fb999149..78a2d08ee1 100644
--- a/flight/flight-integration-tests/pom.xml
+++ b/flight/flight-integration-tests/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-flight
- 18.3.0
+ 19.0.0-SNAPSHOTflight-integration-tests
diff --git a/flight/flight-sql-jdbc-core/pom.xml b/flight/flight-sql-jdbc-core/pom.xml
index 04c4691faf..d8e012101c 100644
--- a/flight/flight-sql-jdbc-core/pom.xml
+++ b/flight/flight-sql-jdbc-core/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-flight
- 18.3.0
+ 19.0.0-SNAPSHOTflight-sql-jdbc-core
diff --git a/flight/flight-sql-jdbc-driver/pom.xml b/flight/flight-sql-jdbc-driver/pom.xml
index 3776e97f3f..559c42597d 100644
--- a/flight/flight-sql-jdbc-driver/pom.xml
+++ b/flight/flight-sql-jdbc-driver/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-flight
- 18.3.0
+ 19.0.0-SNAPSHOTflight-sql-jdbc-driver
diff --git a/flight/flight-sql/pom.xml b/flight/flight-sql/pom.xml
index 66ade30306..5f06a5e9eb 100644
--- a/flight/flight-sql/pom.xml
+++ b/flight/flight-sql/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-flight
- 18.3.0
+ 19.0.0-SNAPSHOTflight-sql
diff --git a/flight/pom.xml b/flight/pom.xml
index 7b31e8ce91..2fc3e89ef8 100644
--- a/flight/pom.xml
+++ b/flight/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-flight
diff --git a/format/pom.xml b/format/pom.xml
index 9b4eebfe3c..d3578b63d2 100644
--- a/format/pom.xml
+++ b/format/pom.xml
@@ -23,7 +23,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-format
diff --git a/gandiva/pom.xml b/gandiva/pom.xml
index 167bf39cb9..5367bfdedf 100644
--- a/gandiva/pom.xml
+++ b/gandiva/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTorg.apache.arrow.gandiva
diff --git a/memory/memory-core/pom.xml b/memory/memory-core/pom.xml
index 840d3464ba..72ee69d60a 100644
--- a/memory/memory-core/pom.xml
+++ b/memory/memory-core/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-memory
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-memory-core
diff --git a/memory/memory-netty-buffer-patch/pom.xml b/memory/memory-netty-buffer-patch/pom.xml
index e9a63b2122..07dc7d2403 100644
--- a/memory/memory-netty-buffer-patch/pom.xml
+++ b/memory/memory-netty-buffer-patch/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-memory
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-memory-netty-buffer-patch
diff --git a/memory/memory-netty/pom.xml b/memory/memory-netty/pom.xml
index 42f35efb33..6d660da117 100644
--- a/memory/memory-netty/pom.xml
+++ b/memory/memory-netty/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-memory
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-memory-netty
diff --git a/memory/memory-unsafe/pom.xml b/memory/memory-unsafe/pom.xml
index 0af306cbfc..92dc0c9fe5 100644
--- a/memory/memory-unsafe/pom.xml
+++ b/memory/memory-unsafe/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-memory
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-memory-unsafe
diff --git a/memory/pom.xml b/memory/pom.xml
index 09a5bc2924..bc34c26050 100644
--- a/memory/pom.xml
+++ b/memory/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-memorypom
diff --git a/performance/pom.xml b/performance/pom.xml
index 02bdf46a11..3f18188e3a 100644
--- a/performance/pom.xml
+++ b/performance/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-performancejar
diff --git a/pom.xml b/pom.xml
index 9d89616356..d40f428b22 100644
--- a/pom.xml
+++ b/pom.xml
@@ -28,7 +28,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTpomApache Arrow Java Root POM
@@ -81,7 +81,7 @@ under the License.
scm:git:https://github.com/apache/arrow-java.gitscm:git:https://github.com/apache/arrow-java.git
- v18.3.0
+ mainhttps://github.com/apache/arrow-java/tree/${project.scm.tag}
diff --git a/tools/pom.xml b/tools/pom.xml
index d60281d80c..cb9a161308 100644
--- a/tools/pom.xml
+++ b/tools/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-toolsArrow Tools
diff --git a/vector/pom.xml b/vector/pom.xml
index 450e4ff71d..52ad5105ea 100644
--- a/vector/pom.xml
+++ b/vector/pom.xml
@@ -22,7 +22,7 @@ under the License.
org.apache.arrowarrow-java-root
- 18.3.0
+ 19.0.0-SNAPSHOTarrow-vectorArrow Vectors
From b459647910a6a373fb9f3d9cd0eb8cc717932bed Mon Sep 17 00:00:00 2001
From: Gang Wu
Date: Wed, 14 May 2025 10:42:38 +0800
Subject: [PATCH 005/147] MINOR: add missing SOURCE_DIR in
dev/release/release.sh (#755)
## What's Changed
`dev/release/release.sh` requires `SOURCE_DIR` to locate `.env` but it
is missing.
---
dev/release/release.sh | 2 ++
1 file changed, 2 insertions(+)
diff --git a/dev/release/release.sh b/dev/release/release.sh
index 70e1f96454..f08a618c4f 100755
--- a/dev/release/release.sh
+++ b/dev/release/release.sh
@@ -19,6 +19,8 @@
set -eu
+SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
if [ "$#" -ne 2 ]; then
echo "Usage: $0 "
echo " e.g.: $0 19.0.1 1"
From 6be33bba2366f2335b962f25fe6ea0789665ae89 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Adam=20Po=C5=82omski?=
Date: Thu, 15 May 2025 04:57:23 +0200
Subject: [PATCH 006/147] MINOR: Empty stream double check (#742)
## What's Changed
In some cases InflaterInputStream
can return a non zero on `available()` method call, while it is actually
at EOS. A subsequent `read()` call would respond with -1 and eventually
cause:
Caused by: org.apache.arrow.flight.FlightRuntimeException: Failed to
read message.
at
org.apache.arrow.flight.CallStatus.toRuntimeException(CallStatus.java:121)
at
org.apache.arrow.flight.grpc.StatusUtils.fromGrpcRuntimeException(StatusUtils.java:161)
at
org.apache.arrow.flight.grpc.StatusUtils.fromThrowable(StatusUtils.java:182)
at
org.apache.arrow.flight.FlightStream$Observer.onError(FlightStream.java:489)
at org.apache.arrow.flight.FlightClient$1.onError(FlightClient.java:371)
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:564)
at
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at
org.apache.arrow.flight.grpc.ClientInterceptorAdapter$FlightClientCallListener.onClose(ClientInterceptorAdapter.java:118)
at
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
... 3 common frames omitted
Caused by: java.lang.RuntimeException:
com.google.protobuf.InvalidProtocolBufferException: While parsing a
protocol message, the input ended unexpectedly in the middle of a field.
This could mean either that the input has been truncated or that an
embedded message misreported its own length.
at org.apache.arrow.flight.ArrowMessage.frame(ArrowMessage.java:363)
at
org.apache.arrow.flight.ArrowMessage$ArrowMessageHolderMarshaller.parse(ArrowMessage.java:575)
at
org.apache.arrow.flight.ArrowMessage$ArrowMessageHolderMarshaller.parse(ArrowMessage.java:560)
at io.grpc.MethodDescriptor.parseResponse(MethodDescriptor.java:284)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:657)
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:644)
... 5 common frames omitted
Caused by: com.google.protobuf.InvalidProtocolBufferException: While
parsing a protocol message, the input ended unexpectedly in the middle
of a field. This could mean either that the input has been truncated or
that an embedded message misreported its own length.
at
com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:92)
at
com.google.protobuf.CodedInputStream.readRawVarint32(CodedInputStream.java:568)
at
org.apache.arrow.flight.ArrowMessage.readRawVarint32(ArrowMessage.java:369)
at org.apache.arrow.flight.ArrowMessage.frame(ArrowMessage.java:290)
... 10 common frames omitted
---
.../java/org/apache/arrow/flight/ArrowMessage.java | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/flight/flight-core/src/main/java/org/apache/arrow/flight/ArrowMessage.java b/flight/flight-core/src/main/java/org/apache/arrow/flight/ArrowMessage.java
index 9cefccb3fe..ab4eab3048 100644
--- a/flight/flight-core/src/main/java/org/apache/arrow/flight/ArrowMessage.java
+++ b/flight/flight-core/src/main/java/org/apache/arrow/flight/ArrowMessage.java
@@ -287,7 +287,11 @@ private static ArrowMessage frame(BufferAllocator allocator, final InputStream s
ArrowBuf body = null;
ArrowBuf appMetadata = null;
while (stream.available() > 0) {
- int tag = readRawVarint32(stream);
+ final int tagFirstByte = stream.read();
+ if (tagFirstByte == -1) {
+ break;
+ }
+ int tag = readRawVarint32(tagFirstByte, stream);
switch (tag) {
case DESCRIPTOR_TAG:
{
@@ -366,6 +370,10 @@ private static ArrowMessage frame(BufferAllocator allocator, final InputStream s
private static int readRawVarint32(InputStream is) throws IOException {
int firstByte = is.read();
+ return readRawVarint32(firstByte, is);
+ }
+
+ private static int readRawVarint32(int firstByte, InputStream is) throws IOException {
return CodedInputStream.readRawVarint32(firstByte, is);
}
From 7c25ce5d86490822600b49928d34a08b4dddad46 Mon Sep 17 00:00:00 2001
From: ViggoC
Date: Thu, 22 May 2025 21:40:44 +0800
Subject: [PATCH 007/147] GH-52: Make RangeEqualsVisitor of RunEndEncodedVector
more efficient (#761)
## What's Changed
Avoid doing a binary search on every step to make the RangeEqualsVisitor
of RunEndEncodedVector more efficient.
Closes #52 .
---
.../vector/compare/RangeEqualsVisitor.java | 44 ++++-----
.../vector/complex/RunEndEncodedVector.java | 98 +++++++++++++++++++
.../arrow/vector/TestRunEndEncodedVector.java | 18 ++--
.../compare/TestRangeEqualsVisitor.java | 52 ++++++++++
4 files changed, 181 insertions(+), 31 deletions(-)
diff --git a/vector/src/main/java/org/apache/arrow/vector/compare/RangeEqualsVisitor.java b/vector/src/main/java/org/apache/arrow/vector/compare/RangeEqualsVisitor.java
index abcf312c5e..bc2e3a6aab 100644
--- a/vector/src/main/java/org/apache/arrow/vector/compare/RangeEqualsVisitor.java
+++ b/vector/src/main/java/org/apache/arrow/vector/compare/RangeEqualsVisitor.java
@@ -43,6 +43,7 @@
import org.apache.arrow.vector.complex.ListViewVector;
import org.apache.arrow.vector.complex.NonNullableStructVector;
import org.apache.arrow.vector.complex.RunEndEncodedVector;
+import org.apache.arrow.vector.complex.RunEndEncodedVector.RangeIterator;
import org.apache.arrow.vector.complex.StructVector;
import org.apache.arrow.vector.complex.UnionVector;
@@ -270,42 +271,35 @@ protected boolean compareRunEndEncodedVectors(Range range) {
RunEndEncodedVector leftVector = (RunEndEncodedVector) left;
RunEndEncodedVector rightVector = (RunEndEncodedVector) right;
- final int leftRangeEnd = range.getLeftStart() + range.getLength();
- final int rightRangeEnd = range.getRightStart() + range.getLength();
+ final RunEndEncodedVector.RangeIterator leftIterator =
+ new RunEndEncodedVector.RangeIterator(leftVector, range.getLeftStart(), range.getLength());
+ final RunEndEncodedVector.RangeIterator rightIterator =
+ new RunEndEncodedVector.RangeIterator(
+ rightVector, range.getRightStart(), range.getLength());
FieldVector leftValuesVector = leftVector.getValuesVector();
FieldVector rightValuesVector = rightVector.getValuesVector();
RangeEqualsVisitor innerVisitor = createInnerVisitor(leftValuesVector, rightValuesVector, null);
- int leftLogicalIndex = range.getLeftStart();
- int rightLogicalIndex = range.getRightStart();
+ while (nextRun(leftIterator, rightIterator)) {
+ int leftPhysicalIndex = leftIterator.getRunIndex();
+ int rightPhysicalIndex = rightIterator.getRunIndex();
- while (leftLogicalIndex < leftRangeEnd) {
- // TODO: implement it more efficient
- // https://github.com/apache/arrow/issues/44157
- int leftPhysicalIndex = leftVector.getPhysicalIndex(leftLogicalIndex);
- int rightPhysicalIndex = rightVector.getPhysicalIndex(rightLogicalIndex);
- if (leftValuesVector.accept(
- innerVisitor, new Range(leftPhysicalIndex, rightPhysicalIndex, 1))) {
- int leftRunEnd = leftVector.getRunEnd(leftLogicalIndex);
- int rightRunEnd = rightVector.getRunEnd(rightLogicalIndex);
-
- int leftRunLength = Math.min(leftRunEnd, leftRangeEnd) - leftLogicalIndex;
- int rightRunLength = Math.min(rightRunEnd, rightRangeEnd) - rightLogicalIndex;
-
- if (leftRunLength != rightRunLength) {
- return false;
- } else {
- leftLogicalIndex = leftRunEnd;
- rightLogicalIndex = rightRunEnd;
- }
- } else {
+ if (leftIterator.getRunLength() != rightIterator.getRunLength()
+ || !leftValuesVector.accept(
+ innerVisitor, new Range(leftPhysicalIndex, rightPhysicalIndex, 1))) {
return false;
}
}
- return true;
+ return leftIterator.isEnd() && rightIterator.isEnd();
+ }
+
+ private static boolean nextRun(RangeIterator leftIterator, RangeIterator rightIterator) {
+ boolean left = leftIterator.nextRun();
+ boolean right = rightIterator.nextRun();
+ return left && right;
}
protected RangeEqualsVisitor createInnerVisitor(
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java
index 1bb9a3d6c0..b83e13449a 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/RunEndEncodedVector.java
@@ -28,6 +28,7 @@
import org.apache.arrow.memory.OutOfMemoryException;
import org.apache.arrow.memory.util.ByteFunctionHelpers;
import org.apache.arrow.memory.util.hash.ArrowBufHasher;
+import org.apache.arrow.util.Preconditions;
import org.apache.arrow.vector.BaseIntVector;
import org.apache.arrow.vector.BaseValueVector;
import org.apache.arrow.vector.BigIntVector;
@@ -820,4 +821,101 @@ static int getPhysicalIndex(FieldVector runEndVector, int logicalIndex) {
return result;
}
+
+ public static class RangeIterator {
+
+ private final RunEndEncodedVector runEndEncodedVector;
+ private final int rangeEnd;
+ private int runIndex;
+ private int runEnd;
+ private int logicalPos;
+
+ /**
+ * Constructs a new RangeIterator for iterating over a range of values in a RunEndEncodedVector.
+ *
+ * @param runEndEncodedVector The vector to iterate over
+ * @param startIndex The logical start index of the range (inclusive)
+ * @param length The number of values to include in the range
+ * @throws IllegalArgumentException if startIndex is negative or (startIndex + length) exceeds
+ * vector bounds
+ */
+ public RangeIterator(RunEndEncodedVector runEndEncodedVector, int startIndex, int length) {
+ int rangeEnd = startIndex + length;
+ Preconditions.checkArgument(
+ startIndex >= 0, "startIndex %s must be non negative.", startIndex);
+ Preconditions.checkArgument(
+ rangeEnd <= runEndEncodedVector.getValueCount(),
+ "(startIndex + length) %s out of range[0, %s].",
+ rangeEnd,
+ runEndEncodedVector.getValueCount());
+
+ this.rangeEnd = rangeEnd;
+ this.runEndEncodedVector = runEndEncodedVector;
+ this.runIndex = runEndEncodedVector.getPhysicalIndex(startIndex) - 1;
+ this.runEnd = startIndex;
+ this.logicalPos = -1;
+ }
+
+ /**
+ * Advances to the next run in the range.
+ *
+ * @return true if there is another run available, false if iteration has completed
+ */
+ public boolean nextRun() {
+ logicalPos = runEnd;
+ if (logicalPos >= rangeEnd) {
+ return false;
+ }
+ updateRun();
+ return true;
+ }
+
+ private void updateRun() {
+ runIndex++;
+ runEnd = (int) ((BaseIntVector) runEndEncodedVector.runEndsVector).getValueAsLong(runIndex);
+ }
+
+ /**
+ * Advances to the next value in the range.
+ *
+ * @return true if there is another value available, false if iteration has completed
+ */
+ public boolean nextValue() {
+ logicalPos++;
+ if (logicalPos >= rangeEnd) {
+ return false;
+ }
+ if (logicalPos == runEnd) {
+ updateRun();
+ }
+ return true;
+ }
+
+ /**
+ * Gets the current run index (physical position in the run-ends vector).
+ *
+ * @return the current run index
+ */
+ public int getRunIndex() {
+ return runIndex;
+ }
+
+ /**
+ * Gets the length of the current run within the iterator's range.
+ *
+ * @return the number of remaining values in current run within the iterator's range
+ */
+ public int getRunLength() {
+ return Math.min(runEnd, rangeEnd) - logicalPos;
+ }
+
+ /**
+ * Checks if iteration has completed.
+ *
+ * @return true if all values in the range have been processed, false otherwise
+ */
+ public boolean isEnd() {
+ return logicalPos >= rangeEnd;
+ }
+ }
}
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestRunEndEncodedVector.java b/vector/src/test/java/org/apache/arrow/vector/TestRunEndEncodedVector.java
index adf51c0730..9fa153e928 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestRunEndEncodedVector.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestRunEndEncodedVector.java
@@ -148,12 +148,18 @@ public void testRangeCompare() {
assertTrue(
constantVector.accept(
new RangeEqualsVisitor(constantVector, constantVector), new Range(1, 2, 13)));
- assertFalse(
- constantVector.accept(
- new RangeEqualsVisitor(constantVector, constantVector), new Range(1, 10, 10)));
- assertFalse(
- constantVector.accept(
- new RangeEqualsVisitor(constantVector, constantVector), new Range(10, 1, 10)));
+
+ // throws exception if the range end is out the bound of the vector
+ assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ constantVector.accept(
+ new RangeEqualsVisitor(constantVector, constantVector), new Range(1, 10, 10)));
+ assertThrows(
+ IllegalArgumentException.class,
+ () ->
+ constantVector.accept(
+ new RangeEqualsVisitor(constantVector, constantVector), new Range(10, 1, 10)));
// Create REE vector representing: [1, 2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 5].
RunEndEncodedVector reeVector =
diff --git a/vector/src/test/java/org/apache/arrow/vector/compare/TestRangeEqualsVisitor.java b/vector/src/test/java/org/apache/arrow/vector/compare/TestRangeEqualsVisitor.java
index 08da786eb2..9624734356 100644
--- a/vector/src/test/java/org/apache/arrow/vector/compare/TestRangeEqualsVisitor.java
+++ b/vector/src/test/java/org/apache/arrow/vector/compare/TestRangeEqualsVisitor.java
@@ -22,6 +22,7 @@
import java.nio.charset.Charset;
import java.util.Arrays;
+import java.util.List;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.BigIntVector;
@@ -39,6 +40,7 @@
import org.apache.arrow.vector.complex.LargeListViewVector;
import org.apache.arrow.vector.complex.ListVector;
import org.apache.arrow.vector.complex.ListViewVector;
+import org.apache.arrow.vector.complex.RunEndEncodedVector;
import org.apache.arrow.vector.complex.StructVector;
import org.apache.arrow.vector.complex.UnionVector;
import org.apache.arrow.vector.complex.impl.NullableStructWriter;
@@ -53,7 +55,9 @@
import org.apache.arrow.vector.holders.NullableUInt4Holder;
import org.apache.arrow.vector.types.FloatingPointPrecision;
import org.apache.arrow.vector.types.Types;
+import org.apache.arrow.vector.types.Types.MinorType;
import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.ArrowType.RunEndEncoded;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
import org.junit.jupiter.api.AfterEach;
@@ -1003,6 +1007,54 @@ public void testLargeListViewVectorApproxEquals() {
}
}
+ @Test
+ public void testRunEndEncodedFloat8ApproxEquals() {
+ try (final Float8Vector vector1 = new Float8Vector("float", allocator);
+ final Float8Vector vector2 = new Float8Vector("float", allocator);
+ final Float8Vector vector3 = new Float8Vector("float", allocator);
+ final IntVector reeVector = new IntVector("ree", allocator)) {
+
+ final float epsilon = 1.0E-6f;
+ setVector(vector1, 1.1, 2.2);
+ setVector(vector2, 1.1 + epsilon / 2, 2.2 + epsilon / 2);
+ setVector(vector3, 1.1 + epsilon * 2, 2.2 + epsilon * 2);
+ setVector(reeVector, 1, 3);
+
+ ArrowType type = MinorType.FLOAT8.getType();
+ final FieldType valueType = FieldType.notNullable(type);
+ final FieldType runEndType = FieldType.notNullable(MinorType.INT.getType());
+
+ final Field valueField = new Field("value", valueType, null);
+ final Field runEndField = new Field("ree", runEndType, null);
+
+ Field field =
+ new Field(
+ "ree_float",
+ FieldType.notNullable(RunEndEncoded.INSTANCE),
+ List.of(runEndField, valueField));
+
+ try (final RunEndEncodedVector encodedVector1 =
+ new RunEndEncodedVector(field, allocator, reeVector, vector1, null);
+ final RunEndEncodedVector encodedVector2 =
+ new RunEndEncodedVector(field, allocator, reeVector, vector2, null);
+ final RunEndEncodedVector encodedVector3 =
+ new RunEndEncodedVector(field, allocator, reeVector, vector3, null)) {
+
+ encodedVector1.setValueCount(3);
+ encodedVector2.setValueCount(3);
+ encodedVector3.setValueCount(3);
+
+ Range range = new Range(0, 0, encodedVector1.getValueCount());
+ assertTrue(
+ new ApproxEqualsVisitor(encodedVector1, encodedVector2, epsilon, epsilon)
+ .rangeEquals(range));
+ assertFalse(
+ new ApproxEqualsVisitor(encodedVector1, encodedVector3, epsilon, epsilon)
+ .rangeEquals(range));
+ }
+ }
+ }
+
private void writeStructVector(NullableStructWriter writer, int value1, long value2) {
writer.start();
writer.integer("f0").writeInt(value1);
From 2aef66df8ce771b3e634e28c98e4442b8cb39995 Mon Sep 17 00:00:00 2001
From: Sutou Kouhei
Date: Sat, 24 May 2025 14:22:40 +0900
Subject: [PATCH 008/147] GH-768: Use apache/arrow-js for JS in integration
test (#769)
## What's Changed
`js/` in apache/arrow moved to apache/arrow-js. So let's use
apache/arrow-js for JS.
Closes #768.
---
.github/workflows/test.yml | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 5db5c988eb..f1028ce029 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -176,6 +176,11 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
path: java
+ - name: Checkout Arrow JavaScript
+ uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+ with:
+ repository: apache/arrow-js
+ path: js
- name: Free up disk space
run: |
ci/scripts/util_free_space.sh
@@ -199,6 +204,7 @@ jobs:
-e ARCHERY_INTEGRATION_TARGET_IMPLEMENTATIONS=java \
-e ARCHERY_INTEGRATION_WITH_GO=1 \
-e ARCHERY_INTEGRATION_WITH_JAVA=1 \
+ -e ARCHERY_INTEGRATION_WITH_JS=1 \
-e ARCHERY_INTEGRATION_WITH_NANOARROW=1 \
-e ARCHERY_INTEGRATION_WITH_RUST=1 \
conda-integration
From abef7af8490d706d52821e56afc6d1eb21e24faf Mon Sep 17 00:00:00 2001
From: Sutou Kouhei
Date: Mon, 26 May 2025 08:55:13 +0900
Subject: [PATCH 009/147] GH-770: Ensure updating Homebrew Python on macos-13
(#771)
## What's Changed
If we update older Python (e.g. Python 3.12) to newer Python (e.g.
Python 3.13), depended packages may update newer Python when we update
old Python.
Let's use newer Python -> older Python order instead to avoid the
situation.
Closes #770.
---
.github/workflows/rc.yml | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/rc.yml b/.github/workflows/rc.yml
index 7e3cf5f6f2..72456fa556 100644
--- a/.github/workflows/rc.yml
+++ b/.github/workflows/rc.yml
@@ -221,6 +221,10 @@ jobs:
# llvm@14 because llvm is newer than llvm@14.
brew uninstall llvm || :
+ # We can remove this when we drop support for
+ # macos-13. because macos-14 or later uses /opt/homebrew/
+ # not /usr/local/.
+ #
# Ensure updating python@XXX with the "--overwrite" option.
# If python@XXX is updated without "--overwrite", it causes
# a conflict error. Because Python 3 installed not by
@@ -229,10 +233,10 @@ jobs:
# tries to replace /usr/local/bin/2to3 and so on and causes
# a conflict error.
brew update
- for python_package in $(brew list | grep python@); do
+ for python_package in $(brew list | grep python@ | sort -r); do
brew install --overwrite ${python_package}
done
- brew install --overwrite python
+ brew install --overwrite python3
if [ "$(uname -m)" = "arm64" ]; then
# pkg-config formula is deprecated but it's still installed
From c6f608e863d3cefdfc41a433ed854e43b20a0925 Mon Sep 17 00:00:00 2001
From: Pepijn Van Eeckhoudt
Date: Wed, 28 May 2025 03:06:02 +0200
Subject: [PATCH 010/147] GH-765: Do not close/free imported BaseStruct objects
(#766)
## What's Changed
This PR removes the direct and indirect calls to `BaseStruct#close` from
`org.apache.arrow.c.Data`. By not eagerly closing/freeing these objects
callers can reuse instances multiple times.
Closes #765.
---
.../org/apache/arrow/c/ArrayImporter.java | 1 -
.../arrow/c/ArrowArrayStreamReader.java | 1 -
c/src/main/java/org/apache/arrow/c/Data.java | 219 ++++++++++++++++--
.../org/apache/arrow/c/RoundtripTest.java | 106 +++++++--
4 files changed, 287 insertions(+), 40 deletions(-)
diff --git a/c/src/main/java/org/apache/arrow/c/ArrayImporter.java b/c/src/main/java/org/apache/arrow/c/ArrayImporter.java
index b74fb1b473..f31a8a1faa 100644
--- a/c/src/main/java/org/apache/arrow/c/ArrayImporter.java
+++ b/c/src/main/java/org/apache/arrow/c/ArrayImporter.java
@@ -58,7 +58,6 @@ void importArray(ArrowArray src) {
ArrowArray ownedArray = ArrowArray.allocateNew(allocator);
ownedArray.save(snapshot);
src.markReleased();
- src.close();
recursionLevel = 0;
diff --git a/c/src/main/java/org/apache/arrow/c/ArrowArrayStreamReader.java b/c/src/main/java/org/apache/arrow/c/ArrowArrayStreamReader.java
index 07a88cd8d7..34a9c4ec03 100644
--- a/c/src/main/java/org/apache/arrow/c/ArrowArrayStreamReader.java
+++ b/c/src/main/java/org/apache/arrow/c/ArrowArrayStreamReader.java
@@ -44,7 +44,6 @@ final class ArrowArrayStreamReader extends ArrowReader {
this.ownedStream = ArrowArrayStream.allocateNew(allocator);
this.ownedStream.save(snapshot);
stream.markReleased();
- stream.close();
}
@Override
diff --git a/c/src/main/java/org/apache/arrow/c/Data.java b/c/src/main/java/org/apache/arrow/c/Data.java
index 0b4da33b4e..f9d2ee4542 100644
--- a/c/src/main/java/org/apache/arrow/c/Data.java
+++ b/c/src/main/java/org/apache/arrow/c/Data.java
@@ -231,6 +231,22 @@ public static void exportArrayStream(
new ArrayStreamExporter(allocator).export(out, reader);
}
+ /**
+ * Equivalent to calling {@link #importField(BufferAllocator, ArrowSchema,
+ * CDataDictionaryProvider, boolean) importField(allocator, schema, provider, true)}.
+ *
+ * @param allocator Buffer allocator for allocating dictionary vectors
+ * @param schema C data interface struct representing the field [inout]
+ * @param provider A dictionary provider will be initialized with empty dictionary vectors
+ * (optional)
+ * @return Imported field object
+ * @see #importField(BufferAllocator, ArrowSchema, CDataDictionaryProvider, boolean)
+ */
+ public static Field importField(
+ BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider) {
+ return importField(allocator, schema, provider, true);
+ }
+
/**
* Import Java Field from the C data interface.
*
@@ -241,19 +257,42 @@ public static void exportArrayStream(
* @param schema C data interface struct representing the field [inout]
* @param provider A dictionary provider will be initialized with empty dictionary vectors
* (optional)
+ * @param closeImportedStructs if true, the ArrowSchema struct will be closed when this method
+ * completes.
* @return Imported field object
*/
public static Field importField(
- BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider) {
+ BufferAllocator allocator,
+ ArrowSchema schema,
+ CDataDictionaryProvider provider,
+ boolean closeImportedStructs) {
try {
SchemaImporter importer = new SchemaImporter(allocator);
return importer.importField(schema, provider);
} finally {
schema.release();
- schema.close();
+ if (closeImportedStructs) {
+ schema.close();
+ }
}
}
+ /**
+ * Equivalent to calling {@link #importSchema(BufferAllocator, ArrowSchema,
+ * CDataDictionaryProvider, boolean) importSchema(allocator, schema, provider, true)}.
+ *
+ * @param allocator Buffer allocator for allocating dictionary vectors
+ * @param schema C data interface struct representing the field
+ * @param provider A dictionary provider will be initialized with empty dictionary vectors
+ * (optional)
+ * @return Imported schema object
+ * @see #importSchema(BufferAllocator, ArrowSchema, CDataDictionaryProvider, boolean)
+ */
+ public static Schema importSchema(
+ BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider) {
+ return importSchema(allocator, schema, provider, true);
+ }
+
/**
* Import Java Schema from the C data interface.
*
@@ -264,11 +303,16 @@ public static Field importField(
* @param schema C data interface struct representing the field
* @param provider A dictionary provider will be initialized with empty dictionary vectors
* (optional)
+ * @param closeImportedStructs if true, the ArrowSchema struct will be closed when this method
+ * completes.
* @return Imported schema object
*/
public static Schema importSchema(
- BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider) {
- Field structField = importField(allocator, schema, provider);
+ BufferAllocator allocator,
+ ArrowSchema schema,
+ CDataDictionaryProvider provider,
+ boolean closeImportedStructs) {
+ Field structField = importField(allocator, schema, provider, closeImportedStructs);
if (structField.getType().getTypeID() != ArrowTypeID.Struct) {
throw new IllegalArgumentException(
"Cannot import schema: ArrowSchema describes non-struct type");
@@ -276,24 +320,67 @@ public static Schema importSchema(
return new Schema(structField.getChildren(), structField.getMetadata());
}
+ /**
+ * Equivalent to calling {@link #importIntoVector(BufferAllocator, ArrowArray, FieldVector,
+ * DictionaryProvider, boolean)} importIntoVector(allocator, array, vector, provider, true)}.
+ *
+ * @param allocator Buffer allocator
+ * @param array C data interface struct holding the array data
+ * @param vector Imported vector object [out]
+ * @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @see #importIntoVector(BufferAllocator, ArrowArray, FieldVector, DictionaryProvider, boolean)
+ */
+ public static void importIntoVector(
+ BufferAllocator allocator,
+ ArrowArray array,
+ FieldVector vector,
+ DictionaryProvider provider) {
+ importIntoVector(allocator, array, vector, provider, true);
+ }
+
/**
* Import Java vector from the C data interface.
*
- *
The ArrowArray struct has its contents moved (as per the C data interface specification) to
- * a private object held alive by the resulting array.
+ *
On successful completion, the ArrowArray struct will have been moved (as per the C data
+ * interface specification) to a private object held alive by the resulting array.
*
* @param allocator Buffer allocator
* @param array C data interface struct holding the array data
* @param vector Imported vector object [out]
* @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @param closeImportedStructs if true, the ArrowArray struct will be closed when this method
+ * completes successfully.
*/
public static void importIntoVector(
BufferAllocator allocator,
ArrowArray array,
FieldVector vector,
- DictionaryProvider provider) {
+ DictionaryProvider provider,
+ boolean closeImportedStructs) {
ArrayImporter importer = new ArrayImporter(allocator, vector, provider);
importer.importArray(array);
+ if (closeImportedStructs) {
+ array.close();
+ }
+ }
+
+ /**
+ * Equivalent to calling {@link #importVector(BufferAllocator, ArrowArray, ArrowSchema,
+ * CDataDictionaryProvider, boolean) importVector(allocator, array, schema, provider, true)}.
+ *
+ * @param allocator Buffer allocator for allocating the output FieldVector
+ * @param array C data interface struct holding the array data
+ * @param schema C data interface struct holding the array type
+ * @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @return Imported vector object
+ * @see #importVector(BufferAllocator, ArrowArray, ArrowSchema, CDataDictionaryProvider, boolean)
+ */
+ public static FieldVector importVector(
+ BufferAllocator allocator,
+ ArrowArray array,
+ ArrowSchema schema,
+ CDataDictionaryProvider provider) {
+ return importVector(allocator, array, schema, provider, true);
}
/**
@@ -307,19 +394,42 @@ public static void importIntoVector(
* @param array C data interface struct holding the array data
* @param schema C data interface struct holding the array type
* @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @param closeImportedStructs if true, the ArrowArray struct will be closed when this method
+ * completes successfully and the ArrowSchema struct will be always be closed.
* @return Imported vector object
*/
public static FieldVector importVector(
BufferAllocator allocator,
ArrowArray array,
ArrowSchema schema,
- CDataDictionaryProvider provider) {
- Field field = importField(allocator, schema, provider);
+ CDataDictionaryProvider provider,
+ boolean closeImportedStructs) {
+ Field field = importField(allocator, schema, provider, closeImportedStructs);
FieldVector vector = field.createVector(allocator);
- importIntoVector(allocator, array, vector, provider);
+ importIntoVector(allocator, array, vector, provider, closeImportedStructs);
return vector;
}
+ /**
+ * Equivalent to calling {@link #importIntoVectorSchemaRoot(BufferAllocator, ArrowArray,
+ * VectorSchemaRoot, DictionaryProvider, boolean) importIntoVectorSchemaRoot(allocator, array,
+ * root, provider, true)}.
+ *
+ * @param allocator Buffer allocator
+ * @param array C data interface struct holding the record batch data
+ * @param root vector schema root to load into
+ * @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @see #importIntoVectorSchemaRoot(BufferAllocator, ArrowArray, VectorSchemaRoot,
+ * DictionaryProvider, boolean)
+ */
+ public static void importIntoVectorSchemaRoot(
+ BufferAllocator allocator,
+ ArrowArray array,
+ VectorSchemaRoot root,
+ DictionaryProvider provider) {
+ importIntoVectorSchemaRoot(allocator, array, root, provider, true);
+ }
+
/**
* Import record batch from the C data interface into vector schema root.
*
@@ -333,15 +443,18 @@ public static FieldVector importVector(
* @param array C data interface struct holding the record batch data
* @param root vector schema root to load into
* @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @param closeImportedStructs if true, the ArrowArray struct will be closed when this method
+ * completes successfully
*/
public static void importIntoVectorSchemaRoot(
BufferAllocator allocator,
ArrowArray array,
VectorSchemaRoot root,
- DictionaryProvider provider) {
+ DictionaryProvider provider,
+ boolean closeImportedStructs) {
try (StructVector structVector = StructVector.emptyWithDuplicates("", allocator)) {
structVector.initializeChildrenFromFields(root.getSchema().getFields());
- importIntoVector(allocator, array, structVector, provider);
+ importIntoVector(allocator, array, structVector, provider, closeImportedStructs);
StructVectorUnloader unloader = new StructVectorUnloader(structVector);
VectorLoader loader = new VectorLoader(root);
try (ArrowRecordBatch recordBatch = unloader.getRecordBatch()) {
@@ -350,6 +463,21 @@ public static void importIntoVectorSchemaRoot(
}
}
+ /**
+ * Equivalent to calling {@link #importVectorSchemaRoot(BufferAllocator, ArrowSchema,
+ * CDataDictionaryProvider, boolean) importVectorSchemaRoot(allocator, schema, provider, true)}.
+ *
+ * @param allocator Buffer allocator for allocating the output VectorSchemaRoot
+ * @param schema C data interface struct holding the record batch schema
+ * @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @return Imported vector schema root
+ * @see #importVectorSchemaRoot(BufferAllocator, ArrowSchema, CDataDictionaryProvider, boolean)
+ */
+ public static VectorSchemaRoot importVectorSchemaRoot(
+ BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider) {
+ return importVectorSchemaRoot(allocator, schema, provider, true);
+ }
+
/**
* Import Java vector schema root from a C data interface Schema.
*
@@ -360,11 +488,37 @@ public static void importIntoVectorSchemaRoot(
* @param allocator Buffer allocator for allocating the output VectorSchemaRoot
* @param schema C data interface struct holding the record batch schema
* @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @param closeImportedStructs if true, the ArrowSchema struct will be closed when this method
+ * completes
* @return Imported vector schema root
*/
public static VectorSchemaRoot importVectorSchemaRoot(
- BufferAllocator allocator, ArrowSchema schema, CDataDictionaryProvider provider) {
- return importVectorSchemaRoot(allocator, null, schema, provider);
+ BufferAllocator allocator,
+ ArrowSchema schema,
+ CDataDictionaryProvider provider,
+ boolean closeImportedStructs) {
+ return importVectorSchemaRoot(allocator, null, schema, provider, closeImportedStructs);
+ }
+
+ /**
+ * Equivalent to calling {@link #importVectorSchemaRoot(BufferAllocator, ArrowArray, ArrowSchema,
+ * CDataDictionaryProvider, boolean) importVectorSchemaRoot(allocator, array, schema, provider,
+ * true)}.
+ *
+ * @param allocator Buffer allocator for allocating the output VectorSchemaRoot
+ * @param array C data interface struct holding the record batch data (optional)
+ * @param schema C data interface struct holding the record batch schema
+ * @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @return Imported vector schema root
+ * @see #importVectorSchemaRoot(BufferAllocator, ArrowArray, ArrowSchema, CDataDictionaryProvider,
+ * boolean)
+ */
+ public static VectorSchemaRoot importVectorSchemaRoot(
+ BufferAllocator allocator,
+ ArrowArray array,
+ ArrowSchema schema,
+ CDataDictionaryProvider provider) {
+ return importVectorSchemaRoot(allocator, array, schema, provider, true);
}
/**
@@ -383,29 +537,56 @@ public static VectorSchemaRoot importVectorSchemaRoot(
* @param array C data interface struct holding the record batch data (optional)
* @param schema C data interface struct holding the record batch schema
* @param provider Dictionary provider to load dictionary vectors to (optional)
+ * @param closeImportedStructs if true, the ArrowArray struct will be closed when this method
+ * completes successfully and the ArrowSchema struct will be always be closed.
* @return Imported vector schema root
*/
public static VectorSchemaRoot importVectorSchemaRoot(
BufferAllocator allocator,
ArrowArray array,
ArrowSchema schema,
- CDataDictionaryProvider provider) {
+ CDataDictionaryProvider provider,
+ boolean closeImportedStructs) {
VectorSchemaRoot vsr =
- VectorSchemaRoot.create(importSchema(allocator, schema, provider), allocator);
+ VectorSchemaRoot.create(
+ importSchema(allocator, schema, provider, closeImportedStructs), allocator);
if (array != null) {
- importIntoVectorSchemaRoot(allocator, array, vsr, provider);
+ importIntoVectorSchemaRoot(allocator, array, vsr, provider, closeImportedStructs);
}
return vsr;
}
/**
- * Import an ArrowArrayStream as an {@link ArrowReader}.
+ * Equivalent to calling {@link #importArrayStream(BufferAllocator, ArrowArrayStream, boolean)
+ * importArrayStream(allocator, stream, true)}.
*
* @param allocator Buffer allocator for allocating the output data.
* @param stream C stream interface struct to import.
* @return Imported reader
+ * @see #importArrayStream(BufferAllocator, ArrowArrayStream, boolean)
*/
public static ArrowReader importArrayStream(BufferAllocator allocator, ArrowArrayStream stream) {
- return new ArrowArrayStreamReader(allocator, stream);
+ return importArrayStream(allocator, stream, true);
+ }
+
+ /**
+ * Import an ArrowArrayStream as an {@link ArrowReader}.
+ *
+ *
On successful completion, the ArrowArrayStream struct will have been moved (as per the C
+ * data interface specification) to a private object held alive by the resulting ArrowReader.
+ *
+ * @param allocator Buffer allocator for allocating the output data.
+ * @param stream C stream interface struct to import.
+ * @param closeImportedStructs if true, the ArrowArrayStream struct will be closed when this
+ * method completes successfully
+ * @return Imported reader
+ */
+ public static ArrowReader importArrayStream(
+ BufferAllocator allocator, ArrowArrayStream stream, boolean closeImportedStructs) {
+ ArrowArrayStreamReader reader = new ArrowArrayStreamReader(allocator, stream);
+ if (closeImportedStructs) {
+ stream.close();
+ }
+ return reader;
}
}
diff --git a/c/src/test/java/org/apache/arrow/c/RoundtripTest.java b/c/src/test/java/org/apache/arrow/c/RoundtripTest.java
index 6d68449c0b..010a305495 100644
--- a/c/src/test/java/org/apache/arrow/c/RoundtripTest.java
+++ b/c/src/test/java/org/apache/arrow/c/RoundtripTest.java
@@ -17,9 +17,7 @@
package org.apache.arrow.c;
import static org.apache.arrow.vector.testing.ValueVectorDataPopulator.setVector;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertThrows;
-import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.*;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
@@ -958,6 +956,50 @@ public void testVectorSchemaRootWithDuplicatedFieldNames() {
@Test
public void testSchema() {
+ Schema schema = createSchema();
+ // Consumer allocates empty ArrowSchema
+ try (ArrowSchema consumerArrowSchema = ArrowSchema.allocateNew(allocator)) {
+ // Producer fills the schema with data
+ exportSchema(schema, consumerArrowSchema);
+
+ // Consumer imports schema
+ Schema importedSchema = Data.importSchema(allocator, consumerArrowSchema, null);
+ assertEquals(schema.toJson(), importedSchema.toJson());
+ }
+ }
+
+ @Test
+ public void testSchemaStructReuse() {
+ Schema schema = createSchema();
+ // Consumer allocates empty ArrowSchema
+ try (ArrowSchema consumerArrowSchema = ArrowSchema.allocateNew(allocator)) {
+ // Producer fills the schema with data
+ exportSchema(schema, consumerArrowSchema);
+
+ // Consumer imports schema
+ Schema importedSchema = Data.importSchema(allocator, consumerArrowSchema, null, false);
+ assertEquals(schema.toJson(), importedSchema.toJson());
+
+ // Imported struct should be released but not closed
+ assertEquals(0, consumerArrowSchema.snapshot().release);
+ assertNotEquals(0, consumerArrowSchema.memoryAddress());
+
+ // Export and import again
+ exportSchema(schema, consumerArrowSchema);
+ importedSchema = Data.importSchema(allocator, consumerArrowSchema, null, false);
+ assertEquals(schema.toJson(), importedSchema.toJson());
+ assertEquals(0, consumerArrowSchema.snapshot().release);
+ assertNotEquals(0, consumerArrowSchema.memoryAddress());
+ }
+ }
+
+ private void exportSchema(Schema schema, ArrowSchema targetArrowSchema) {
+ try (ArrowSchema arrowSchema = ArrowSchema.wrap(targetArrowSchema.memoryAddress())) {
+ Data.exportSchema(allocator, schema, null, arrowSchema);
+ }
+ }
+
+ private static Schema createSchema() {
Field decimalField =
new Field("inner1", FieldType.nullable(new ArrowType.Decimal(19, 4, 128)), null);
Field strField = new Field("inner2", FieldType.nullable(new ArrowType.Utf8()), null);
@@ -968,16 +1010,7 @@ public void testSchema() {
Arrays.asList(decimalField, strField));
Field intField = new Field("col2", FieldType.nullable(new ArrowType.Int(32, true)), null);
Schema schema = new Schema(Arrays.asList(itemField, intField));
- // Consumer allocates empty ArrowSchema
- try (ArrowSchema consumerArrowSchema = ArrowSchema.allocateNew(allocator)) {
- // Producer fills the schema with data
- try (ArrowSchema arrowSchema = ArrowSchema.wrap(consumerArrowSchema.memoryAddress())) {
- Data.exportSchema(allocator, schema, null, arrowSchema);
- }
- // Consumer imports schema
- Schema importedSchema = Data.importSchema(allocator, consumerArrowSchema, null);
- assertEquals(schema.toJson(), importedSchema.toJson());
- }
+ return schema;
}
@Test
@@ -1002,12 +1035,8 @@ public void testImportReleasedArray() {
try (ArrowSchema consumerArrowSchema = ArrowSchema.allocateNew(allocator);
ArrowArray consumerArrowArray = ArrowArray.allocateNew(allocator)) {
// Producer creates structures from existing memory pointers
- try (ArrowSchema arrowSchema = ArrowSchema.wrap(consumerArrowSchema.memoryAddress());
- ArrowArray arrowArray = ArrowArray.wrap(consumerArrowArray.memoryAddress())) {
- // Producer exports vector into the C Data Interface structures
- try (final NullVector vector = new NullVector()) {
- Data.exportVector(allocator, vector, null, arrowArray, arrowSchema);
- }
+ try (final NullVector vector = new NullVector()) {
+ exportFieldVector(vector, consumerArrowSchema, consumerArrowArray);
}
// Release array structure
@@ -1025,6 +1054,45 @@ public void testImportReleasedArray() {
}
}
+ @Test
+ public void testArrayStructReuse() {
+ // Consumer allocates empty structures
+ try (ArrowSchema consumerArrowSchema = ArrowSchema.allocateNew(allocator);
+ ArrowArray consumerArrowArray = ArrowArray.allocateNew(allocator)) {
+ // Producer creates structures from existing memory pointers
+ try (final NullVector vector = new NullVector()) {
+ exportFieldVector(vector, consumerArrowSchema, consumerArrowArray);
+ }
+ Data.importVector(allocator, consumerArrowArray, consumerArrowSchema, null, false);
+
+ // Imported structs should be released but not closed
+ assertEquals(0, consumerArrowSchema.snapshot().release);
+ assertNotEquals(0, consumerArrowSchema.memoryAddress());
+ assertEquals(0, consumerArrowArray.snapshot().release);
+ assertNotEquals(0, consumerArrowArray.memoryAddress());
+
+ try (final NullVector vector = new NullVector()) {
+ exportFieldVector(vector, consumerArrowSchema, consumerArrowArray);
+ }
+ Data.importVector(allocator, consumerArrowArray, consumerArrowSchema, null, false);
+
+ // Imported structs should be released but not closed
+ assertEquals(0, consumerArrowSchema.snapshot().release);
+ assertNotEquals(0, consumerArrowSchema.memoryAddress());
+ assertEquals(0, consumerArrowArray.snapshot().release);
+ assertNotEquals(0, consumerArrowArray.memoryAddress());
+ }
+ }
+
+ private void exportFieldVector(
+ FieldVector vector, ArrowSchema consumerArrowSchema, ArrowArray consumerArrowArray) {
+ try (ArrowSchema arrowSchema = ArrowSchema.wrap(consumerArrowSchema.memoryAddress());
+ ArrowArray arrowArray = ArrowArray.wrap(consumerArrowArray.memoryAddress())) {
+ // Producer exports vector into the C Data Interface structures
+ Data.exportVector(allocator, vector, null, arrowArray, arrowSchema);
+ }
+ }
+
private VectorSchemaRoot createTestVSR() {
BitVector bitVector = new BitVector("boolean", allocator);
From 17f85a1fba64aef4552ed2fabc5a76bc038a1ed6 Mon Sep 17 00:00:00 2001
From: rtadepalli <105760760+rtadepalli@users.noreply.github.com>
Date: Tue, 27 May 2025 21:06:55 -0400
Subject: [PATCH 011/147] GH-70: Move from `hamcrest` to `assertj` in
`flight-sql` (#772)
## What's Changed
Series of PRs to consolidate on using `assertj` in tests as part of
https://github.com/apache/arrow-java/issues/70.
---
flight/flight-sql/pom.xml | 10 +-
.../arrow/flight/sql/test/TestFlightSql.java | 264 +++++++++---------
.../sql/test/TestFlightSqlStateless.java | 9 +-
.../flight/sql/test/TestFlightSqlStreams.java | 29 +-
4 files changed, 150 insertions(+), 162 deletions(-)
diff --git a/flight/flight-sql/pom.xml b/flight/flight-sql/pom.xml
index 5f06a5e9eb..15d00e3e18 100644
--- a/flight/flight-sql/pom.xml
+++ b/flight/flight-sql/pom.xml
@@ -116,16 +116,16 @@ under the License.
1.13.1test
-
- org.hamcrest
- hamcrest
- test
- commons-clicommons-cli1.9.0true
+
+ org.assertj
+ assertj-core
+ test
+
diff --git a/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSql.java b/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSql.java
index 3f769363fb..e2934ab1e9 100644
--- a/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSql.java
+++ b/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSql.java
@@ -21,10 +21,7 @@
import static java.util.Collections.singletonList;
import static org.apache.arrow.flight.sql.util.FlightStreamUtils.getResults;
import static org.apache.arrow.util.AutoCloseables.close;
-import static org.hamcrest.CoreMatchers.containsString;
-import static org.hamcrest.CoreMatchers.is;
-import static org.hamcrest.CoreMatchers.notNullValue;
-import static org.hamcrest.CoreMatchers.nullValue;
+import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertAll;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -40,6 +37,7 @@
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.Optional;
import java.util.stream.IntStream;
import org.apache.arrow.flight.CancelFlightInfoRequest;
@@ -76,8 +74,7 @@
import org.apache.arrow.vector.types.pojo.Schema;
import org.apache.arrow.vector.util.Text;
import org.apache.arrow.vector.util.VectorBatchAppender;
-import org.hamcrest.Matcher;
-import org.hamcrest.MatcherAssert;
+import org.assertj.core.api.Condition;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
@@ -247,16 +244,15 @@ private static List> getNonConformingResultsForGetSqlInfo(
@Test
public void testGetTablesSchema() {
final FlightInfo info = sqlClient.getTables(null, null, null, null, true);
- MatcherAssert.assertThat(
- info.getSchemaOptional(), is(Optional.of(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA)));
+ assertThat(info.getSchemaOptional())
+ .isEqualTo(Optional.of(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA));
}
@Test
public void testGetTablesSchemaExcludeSchema() {
final FlightInfo info = sqlClient.getTables(null, null, null, null, false);
- MatcherAssert.assertThat(
- info.getSchemaOptional(),
- is(Optional.of(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA)));
+ assertThat(info.getSchemaOptional())
+ .isEqualTo(Optional.of(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA));
}
@Test
@@ -266,8 +262,8 @@ public void testGetTablesResultNoSchema() throws Exception {
sqlClient.getTables(null, null, null, null, false).getEndpoints().get(0).getTicket())) {
assertAll(
() -> {
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA));
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA);
},
() -> {
final List> results = getResults(stream);
@@ -301,7 +297,7 @@ public void testGetTablesResultNoSchema() throws Exception {
asList(null /* TODO No catalog yet */, "SYSIBM", "SYSDUMMY1", "SYSTEM TABLE"),
asList(null /* TODO No catalog yet */, "APP", "FOREIGNTABLE", "TABLE"),
asList(null /* TODO No catalog yet */, "APP", "INTTABLE", "TABLE"));
- MatcherAssert.assertThat(results, is(expectedResults));
+ assertThat(results).isEqualTo(expectedResults);
});
}
}
@@ -318,8 +314,8 @@ public void testGetTablesResultFilteredNoSchema() throws Exception {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA)),
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA),
() -> {
final List> results = getResults(stream);
final List> expectedResults =
@@ -327,7 +323,7 @@ public void testGetTablesResultFilteredNoSchema() throws Exception {
// catalog_name | schema_name | table_name | table_type | table_schema
asList(null /* TODO No catalog yet */, "APP", "FOREIGNTABLE", "TABLE"),
asList(null /* TODO No catalog yet */, "APP", "INTTABLE", "TABLE"));
- MatcherAssert.assertThat(results, is(expectedResults));
+ assertThat(results).isEqualTo(expectedResults);
});
}
}
@@ -343,11 +339,9 @@ public void testGetTablesResultFilteredWithSchema() throws Exception {
.getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA)),
+ assertThat(stream.getSchema()).isEqualTo(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA),
() -> {
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA));
+ assertThat(stream.getSchema()).isEqualTo(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA);
final List> results = getResults(stream);
final List> expectedResults =
ImmutableList.of(
@@ -487,7 +481,7 @@ public void testGetTablesResultFilteredWithSchema() throws Exception {
.getMetadataMap()),
null)))
.toJson()));
- MatcherAssert.assertThat(results, is(expectedResults));
+ assertThat(results).isEqualTo(expectedResults);
});
}
}
@@ -498,11 +492,11 @@ public void testSimplePreparedStatementSchema() throws Exception {
assertAll(
() -> {
final Schema actualSchema = preparedStatement.getResultSetSchema();
- MatcherAssert.assertThat(actualSchema, is(SCHEMA_INT_TABLE));
+ assertThat(actualSchema).isEqualTo(SCHEMA_INT_TABLE);
},
() -> {
final FlightInfo info = preparedStatement.execute();
- MatcherAssert.assertThat(info.getSchemaOptional(), is(Optional.of(SCHEMA_INT_TABLE)));
+ assertThat(info.getSchemaOptional()).isEqualTo(Optional.of(SCHEMA_INT_TABLE));
});
}
}
@@ -513,10 +507,8 @@ public void testSimplePreparedStatementResults() throws Exception {
final FlightStream stream =
sqlClient.getStream(preparedStatement.execute().getEndpoints().get(0).getTicket())) {
assertAll(
- () -> MatcherAssert.assertThat(stream.getSchema(), is(SCHEMA_INT_TABLE)),
- () ->
- MatcherAssert.assertThat(
- getResults(stream), is(EXPECTED_RESULTS_FOR_STAR_SELECT_QUERY)));
+ () -> assertThat(stream.getSchema()).isEqualTo(SCHEMA_INT_TABLE),
+ () -> assertThat(getResults(stream)).isEqualTo(EXPECTED_RESULTS_FOR_STAR_SELECT_QUERY));
}
}
@@ -538,10 +530,8 @@ public void testSimplePreparedStatementResultsWithParameterBinding() throws Exce
FlightStream stream = sqlClient.getStream(flightInfo.getEndpoints().get(0).getTicket());
assertAll(
- () -> MatcherAssert.assertThat(stream.getSchema(), is(SCHEMA_INT_TABLE)),
- () ->
- MatcherAssert.assertThat(
- getResults(stream), is(EXPECTED_RESULTS_FOR_PARAMETER_BINDING)));
+ () -> assertThat(stream.getSchema()).isEqualTo(SCHEMA_INT_TABLE),
+ () -> assertThat(getResults(stream)).isEqualTo(EXPECTED_RESULTS_FOR_PARAMETER_BINDING));
}
}
}
@@ -579,8 +569,8 @@ public void testSimplePreparedStatementUpdateResults() throws SQLException {
deletedRows = deletePrepare.executeUpdate();
}
assertAll(
- () -> MatcherAssert.assertThat(updatedRows, is(10L)),
- () -> MatcherAssert.assertThat(deletedRows, is(10L)));
+ () -> assertThat(updatedRows).isEqualTo(10L),
+ () -> assertThat(deletedRows).isEqualTo(10L));
}
}
}
@@ -647,7 +637,7 @@ public void testBulkIngest() throws IOException {
null,
null));
- MatcherAssert.assertThat(updatedRows, is(-1L));
+ assertThat(updatedRows).isEqualTo(-1L);
// Ingest directly using VectorSchemaRoot
populateNext10RowsInIngestRootBatch(
@@ -672,7 +662,7 @@ public void testBulkIngest() throws IOException {
deletedRows = deletePrepare.executeUpdate();
}
- MatcherAssert.assertThat(deletedRows, is(30L));
+ assertThat(deletedRows).isEqualTo(30L);
}
}
}
@@ -709,8 +699,7 @@ public void testSimplePreparedStatementUpdateResultsWithoutParameters() throws S
final long deletedRows = deletePrepare.executeUpdate();
assertAll(
- () -> MatcherAssert.assertThat(updatedRows, is(1L)),
- () -> MatcherAssert.assertThat(deletedRows, is(1L)));
+ () -> assertThat(updatedRows).isEqualTo(1L), () -> assertThat(deletedRows).isEqualTo(1L));
}
}
@@ -719,19 +708,19 @@ public void testSimplePreparedStatementClosesProperly() {
final PreparedStatement preparedStatement = sqlClient.prepare("SELECT * FROM intTable");
assertAll(
() -> {
- MatcherAssert.assertThat(preparedStatement.isClosed(), is(false));
+ assertThat(preparedStatement.isClosed()).isEqualTo(false);
},
() -> {
preparedStatement.close();
- MatcherAssert.assertThat(preparedStatement.isClosed(), is(true));
+ assertThat(preparedStatement.isClosed()).isEqualTo(true);
});
}
@Test
public void testGetCatalogsSchema() {
final FlightInfo info = sqlClient.getCatalogs();
- MatcherAssert.assertThat(
- info.getSchemaOptional(), is(Optional.of(FlightSqlProducer.Schemas.GET_CATALOGS_SCHEMA)));
+ assertThat(info.getSchemaOptional())
+ .isEqualTo(Optional.of(FlightSqlProducer.Schemas.GET_CATALOGS_SCHEMA));
}
@Test
@@ -740,11 +729,11 @@ public void testGetCatalogsResults() throws Exception {
sqlClient.getStream(sqlClient.getCatalogs().getEndpoints().get(0).getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_CATALOGS_SCHEMA)),
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_CATALOGS_SCHEMA),
() -> {
List> catalogs = getResults(stream);
- MatcherAssert.assertThat(catalogs, is(emptyList()));
+ assertThat(catalogs).isEqualTo(emptyList());
});
}
}
@@ -752,9 +741,8 @@ public void testGetCatalogsResults() throws Exception {
@Test
public void testGetTableTypesSchema() {
final FlightInfo info = sqlClient.getTableTypes();
- MatcherAssert.assertThat(
- info.getSchemaOptional(),
- is(Optional.of(FlightSqlProducer.Schemas.GET_TABLE_TYPES_SCHEMA)));
+ assertThat(info.getSchemaOptional())
+ .isEqualTo(Optional.of(FlightSqlProducer.Schemas.GET_TABLE_TYPES_SCHEMA));
}
@Test
@@ -763,8 +751,8 @@ public void testGetTableTypesResult() throws Exception {
sqlClient.getStream(sqlClient.getTableTypes().getEndpoints().get(0).getTicket())) {
assertAll(
() -> {
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_TABLE_TYPES_SCHEMA));
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_TABLE_TYPES_SCHEMA);
},
() -> {
final List> tableTypes = getResults(stream);
@@ -775,7 +763,7 @@ public void testGetTableTypesResult() throws Exception {
singletonList("SYSTEM TABLE"),
singletonList("TABLE"),
singletonList("VIEW"));
- MatcherAssert.assertThat(tableTypes, is(expectedTableTypes));
+ assertThat(tableTypes).isEqualTo(expectedTableTypes);
});
}
}
@@ -783,8 +771,8 @@ public void testGetTableTypesResult() throws Exception {
@Test
public void testGetSchemasSchema() {
final FlightInfo info = sqlClient.getSchemas(null, null);
- MatcherAssert.assertThat(
- info.getSchemaOptional(), is(Optional.of(FlightSqlProducer.Schemas.GET_SCHEMAS_SCHEMA)));
+ assertThat(info.getSchemaOptional())
+ .isEqualTo(Optional.of(FlightSqlProducer.Schemas.GET_SCHEMAS_SCHEMA));
}
@Test
@@ -793,8 +781,7 @@ public void testGetSchemasResult() throws Exception {
sqlClient.getStream(sqlClient.getSchemas(null, null).getEndpoints().get(0).getTicket())) {
assertAll(
() -> {
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_SCHEMAS_SCHEMA));
+ assertThat(stream.getSchema()).isEqualTo(FlightSqlProducer.Schemas.GET_SCHEMAS_SCHEMA);
},
() -> {
final List> schemas = getResults(stream);
@@ -812,7 +799,7 @@ public void testGetSchemasResult() throws Exception {
asList(null /* TODO Add catalog. */, "SYSIBM"),
asList(null /* TODO Add catalog. */, "SYSPROC"),
asList(null /* TODO Add catalog. */, "SYSSTAT"));
- MatcherAssert.assertThat(schemas, is(expectedSchemas));
+ assertThat(schemas).isEqualTo(expectedSchemas);
});
}
}
@@ -825,24 +812,24 @@ public void testGetPrimaryKey() {
final List> results = getResults(stream);
assertAll(
- () -> MatcherAssert.assertThat(results.size(), is(1)),
+ () -> assertThat(results.size()).isEqualTo(1),
() -> {
final List result = results.get(0);
assertAll(
- () -> MatcherAssert.assertThat(result.get(0), is("")),
- () -> MatcherAssert.assertThat(result.get(1), is("APP")),
- () -> MatcherAssert.assertThat(result.get(2), is("INTTABLE")),
- () -> MatcherAssert.assertThat(result.get(3), is("ID")),
- () -> MatcherAssert.assertThat(result.get(4), is("1")),
- () -> MatcherAssert.assertThat(result.get(5), notNullValue()));
+ () -> assertThat(result.get(0)).isEqualTo(""),
+ () -> assertThat(result.get(1)).isEqualTo("APP"),
+ () -> assertThat(result.get(2)).isEqualTo("INTTABLE"),
+ () -> assertThat(result.get(3)).isEqualTo("ID"),
+ () -> assertThat(result.get(4)).isEqualTo("1"),
+ () -> assertThat(result.get(5)).isNotNull());
});
}
@Test
public void testGetSqlInfoSchema() {
final FlightInfo info = sqlClient.getSqlInfo();
- MatcherAssert.assertThat(
- info.getSchemaOptional(), is(Optional.of(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA)));
+ assertThat(info.getSchemaOptional())
+ .isEqualTo(Optional.of(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA));
}
@Test
@@ -851,11 +838,11 @@ public void testGetSqlInfoResults() throws Exception {
try (final FlightStream stream = sqlClient.getStream(info.getEndpoints().get(0).getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA)),
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA),
() ->
- MatcherAssert.assertThat(
- getNonConformingResultsForGetSqlInfo(getResults(stream)), is(emptyList())));
+ assertThat(getNonConformingResultsForGetSqlInfo(getResults(stream)))
+ .isEqualTo(emptyList()));
}
}
@@ -866,11 +853,11 @@ public void testGetSqlInfoResultsWithSingleArg() throws Exception {
try (final FlightStream stream = sqlClient.getStream(info.getEndpoints().get(0).getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA)),
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA),
() ->
- MatcherAssert.assertThat(
- getNonConformingResultsForGetSqlInfo(getResults(stream), arg), is(emptyList())));
+ assertThat(getNonConformingResultsForGetSqlInfo(getResults(stream), arg))
+ .isEqualTo(emptyList()));
}
}
@@ -895,11 +882,11 @@ public void testGetSqlInfoResultsWithManyArgs() throws Exception {
try (final FlightStream stream = sqlClient.getStream(info.getEndpoints().get(0).getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA)),
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA),
() ->
- MatcherAssert.assertThat(
- getNonConformingResultsForGetSqlInfo(getResults(stream), args), is(emptyList())));
+ assertThat(getNonConformingResultsForGetSqlInfo(getResults(stream), args))
+ .isEqualTo(emptyList()));
}
}
@@ -915,28 +902,30 @@ public void testGetCommandExportedKeys() throws Exception {
final List> results = getResults(stream);
- final List> matchers =
+ final List> matchers =
asList(
- nullValue(String.class), // pk_catalog_name
- is("APP"), // pk_schema_name
- is("FOREIGNTABLE"), // pk_table_name
- is("ID"), // pk_column_name
- nullValue(String.class), // fk_catalog_name
- is("APP"), // fk_schema_name
- is("INTTABLE"), // fk_table_name
- is("FOREIGNID"), // fk_column_name
- is("1"), // key_sequence
- containsString("SQL"), // fk_key_name
- containsString("SQL"), // pk_key_name
- is("3"), // update_rule
- is("3")); // delete_rule
+ new Condition<>(Objects::isNull, "pk_catalog_name expected to be null"),
+ new Condition<>(c -> c.equals("APP"), "pk_schema_name expected to equal APP"),
+ new Condition<>(
+ c -> c.equals("FOREIGNTABLE"), "pk_table_name should equal FOREIGNTABLE"),
+ new Condition<>(c -> c.equals("ID"), "pk_column_name should equal ID"),
+ new Condition<>(Objects::isNull, "fk_catalog_name expected to be null"),
+ new Condition<>(c -> c.equals("APP"), "fk_schema_name expected to be APP"),
+ new Condition<>(c -> c.equals("INTTABLE"), "fk_table_name expeced to be INTTABLE"),
+ new Condition<>(
+ c -> c.equals("FOREIGNID"), "fk_column_name expected to equal FOREIGNID"),
+ new Condition<>(c -> c.equals("1"), "key_sequence expected to equal 1"),
+ new Condition<>(c -> c.contains("SQL"), "fk_key_name expected to contain SQL"),
+ new Condition<>(c -> c.contains("SQL"), "pk_key_name expected to contain SQL"),
+ new Condition<>(c -> c.equals("3"), "update_rule expected to equal 3"),
+ new Condition<>(c -> c.equals("3"), "delete_rule expected to equal 3"));
final List assertions = new ArrayList<>();
assertEquals(1, results.size());
for (int i = 0; i < matchers.size(); i++) {
final String actual = results.get(0).get(i);
- final Matcher expected = matchers.get(i);
- assertions.add(() -> MatcherAssert.assertThat(actual, expected));
+ final Condition expected = matchers.get(i);
+ assertions.add(() -> assertThat(actual).satisfies(expected));
}
assertAll(assertions);
}
@@ -954,28 +943,30 @@ public void testGetCommandImportedKeys() throws Exception {
final List> results = getResults(stream);
- final List> matchers =
+ final List> matchers =
asList(
- nullValue(String.class), // pk_catalog_name
- is("APP"), // pk_schema_name
- is("FOREIGNTABLE"), // pk_table_name
- is("ID"), // pk_column_name
- nullValue(String.class), // fk_catalog_name
- is("APP"), // fk_schema_name
- is("INTTABLE"), // fk_table_name
- is("FOREIGNID"), // fk_column_name
- is("1"), // key_sequence
- containsString("SQL"), // fk_key_name
- containsString("SQL"), // pk_key_name
- is("3"), // update_rule
- is("3")); // delete_rule
+ new Condition<>(Objects::isNull, "pk_catalog_name expected to be null"),
+ new Condition<>(c -> c.equals("APP"), "pk_schema_name expected to equal APP"),
+ new Condition<>(
+ c -> c.equals("FOREIGNTABLE"), "pk_table_name should equal FOREIGNTABLE"),
+ new Condition<>(c -> c.equals("ID"), "pk_column_name should equal ID"),
+ new Condition<>(Objects::isNull, "fk_catalog_name expected to be null"),
+ new Condition<>(c -> c.equals("APP"), "fk_schema_name expected to be APP"),
+ new Condition<>(c -> c.equals("INTTABLE"), "fk_table_name expeced to be INTTABLE"),
+ new Condition<>(
+ c -> c.equals("FOREIGNID"), "fk_column_name expected to equal FOREIGNID"),
+ new Condition<>(c -> c.equals("1"), "key_sequence expected to equal 1"),
+ new Condition<>(c -> c.contains("SQL"), "fk_key_name expected to contain SQL"),
+ new Condition<>(c -> c.contains("SQL"), "pk_key_name expected to contain SQL"),
+ new Condition<>(c -> c.equals("3"), "update_rule expected to equal 3"),
+ new Condition<>(c -> c.equals("3"), "delete_rule expected to equal 3"));
assertEquals(1, results.size());
final List assertions = new ArrayList<>();
for (int i = 0; i < matchers.size(); i++) {
final String actual = results.get(0).get(i);
- final Matcher expected = matchers.get(i);
- assertions.add(() -> MatcherAssert.assertThat(actual, expected));
+ final Condition expected = matchers.get(i);
+ assertions.add(() -> assertThat(actual).satisfies(expected));
}
assertAll(assertions);
}
@@ -1431,7 +1422,7 @@ public void testGetTypeInfo() throws Exception {
null,
null,
null));
- MatcherAssert.assertThat(results, is(matchers));
+ assertThat(results).isEqualTo(matchers);
}
}
@@ -1465,7 +1456,7 @@ public void testGetTypeInfoWithFiltering() throws Exception {
null,
"10",
null));
- MatcherAssert.assertThat(results, is(matchers));
+ assertThat(results).isEqualTo(matchers);
}
}
@@ -1479,28 +1470,30 @@ public void testGetCommandCrossReference() throws Exception {
final List> results = getResults(stream);
- final List> matchers =
+ final List> matchers =
asList(
- nullValue(String.class), // pk_catalog_name
- is("APP"), // pk_schema_name
- is("FOREIGNTABLE"), // pk_table_name
- is("ID"), // pk_column_name
- nullValue(String.class), // fk_catalog_name
- is("APP"), // fk_schema_name
- is("INTTABLE"), // fk_table_name
- is("FOREIGNID"), // fk_column_name
- is("1"), // key_sequence
- containsString("SQL"), // fk_key_name
- containsString("SQL"), // pk_key_name
- is("3"), // update_rule
- is("3")); // delete_rule
+ new Condition<>(Objects::isNull, "pk_catalog_name expected to be null"),
+ new Condition<>(c -> c.equals("APP"), "pk_schema_name expected to equal APP"),
+ new Condition<>(
+ c -> c.equals("FOREIGNTABLE"), "pk_table_name should equal FOREIGNTABLE"),
+ new Condition<>(c -> c.equals("ID"), "pk_column_name should equal ID"),
+ new Condition<>(Objects::isNull, "fk_catalog_name expected to be null"),
+ new Condition<>(c -> c.equals("APP"), "fk_schema_name expected to be APP"),
+ new Condition<>(c -> c.equals("INTTABLE"), "fk_table_name expeced to be INTTABLE"),
+ new Condition<>(
+ c -> c.equals("FOREIGNID"), "fk_column_name expected to equal FOREIGNID"),
+ new Condition<>(c -> c.equals("1"), "key_sequence expected to equal 1"),
+ new Condition<>(c -> c.contains("SQL"), "fk_key_name expected to contain SQL"),
+ new Condition<>(c -> c.contains("SQL"), "pk_key_name expected to contain SQL"),
+ new Condition<>(c -> c.equals("3"), "update_rule expected to equal 3"),
+ new Condition<>(c -> c.equals("3"), "delete_rule expected to equal 3"));
assertEquals(1, results.size());
final List assertions = new ArrayList<>();
for (int i = 0; i < matchers.size(); i++) {
final String actual = results.get(0).get(i);
- final Matcher expected = matchers.get(i);
- assertions.add(() -> MatcherAssert.assertThat(actual, expected));
+ final Condition expected = matchers.get(i);
+ assertions.add(() -> assertThat(actual).satisfies(expected));
}
assertAll(assertions);
}
@@ -1509,7 +1502,7 @@ public void testGetCommandCrossReference() throws Exception {
@Test
public void testCreateStatementSchema() throws Exception {
final FlightInfo info = sqlClient.execute("SELECT * FROM intTable");
- MatcherAssert.assertThat(info.getSchemaOptional(), is(Optional.of(SCHEMA_INT_TABLE)));
+ assertThat(info.getSchemaOptional()).isEqualTo(Optional.of(SCHEMA_INT_TABLE));
// Consume statement to close connection before cache eviction
try (FlightStream stream = sqlClient.getStream(info.getEndpoints().get(0).getTicket())) {
@@ -1526,11 +1519,10 @@ public void testCreateStatementResults() throws Exception {
sqlClient.execute("SELECT * FROM intTable").getEndpoints().get(0).getTicket())) {
assertAll(
() -> {
- MatcherAssert.assertThat(stream.getSchema(), is(SCHEMA_INT_TABLE));
+ assertThat(stream.getSchema()).isEqualTo(SCHEMA_INT_TABLE);
},
() -> {
- MatcherAssert.assertThat(
- getResults(stream), is(EXPECTED_RESULTS_FOR_STAR_SELECT_QUERY));
+ assertThat(getResults(stream)).isEqualTo(EXPECTED_RESULTS_FOR_STAR_SELECT_QUERY);
});
}
}
@@ -1543,19 +1535,19 @@ public void testExecuteUpdate() {
sqlClient.executeUpdate(
"INSERT INTO INTTABLE (keyName, value) VALUES "
+ "('KEYNAME1', 1001), ('KEYNAME2', 1002), ('KEYNAME3', 1003)");
- MatcherAssert.assertThat(insertedCount, is(3L));
+ assertThat(insertedCount).isEqualTo(3L);
},
() -> {
long updatedCount =
sqlClient.executeUpdate(
"UPDATE INTTABLE SET keyName = 'KEYNAME1' "
+ "WHERE keyName = 'KEYNAME2' OR keyName = 'KEYNAME3'");
- MatcherAssert.assertThat(updatedCount, is(2L));
+ assertThat(updatedCount).isEqualTo(2L);
},
() -> {
long deletedCount =
sqlClient.executeUpdate("DELETE FROM INTTABLE WHERE keyName = 'KEYNAME1'");
- MatcherAssert.assertThat(deletedCount, is(3L));
+ assertThat(deletedCount).isEqualTo(3L);
});
}
@@ -1566,10 +1558,10 @@ public void testQueryWithNoResultsShouldNotHang() throws Exception {
final FlightStream stream =
sqlClient.getStream(preparedStatement.execute().getEndpoints().get(0).getTicket())) {
assertAll(
- () -> MatcherAssert.assertThat(stream.getSchema(), is(SCHEMA_INT_TABLE)),
+ () -> assertThat(stream.getSchema()).isEqualTo(SCHEMA_INT_TABLE),
() -> {
final List> result = getResults(stream);
- MatcherAssert.assertThat(result, is(emptyList()));
+ assertThat(result).isEqualTo(emptyList());
});
}
}
diff --git a/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStateless.java b/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStateless.java
index 36d621ad64..ee1507b6af 100644
--- a/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStateless.java
+++ b/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStateless.java
@@ -18,7 +18,7 @@
import static org.apache.arrow.flight.sql.util.FlightStreamUtils.getResults;
import static org.apache.arrow.util.AutoCloseables.close;
-import static org.hamcrest.CoreMatchers.is;
+import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertAll;
import org.apache.arrow.flight.FlightClient;
@@ -34,7 +34,6 @@
import org.apache.arrow.vector.IntVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.types.pojo.Schema;
-import org.hamcrest.MatcherAssert;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
@@ -89,10 +88,10 @@ public void testSimplePreparedStatementResultsWithParameterBinding() throws Exce
for (FlightEndpoint endpoint : flightInfo.getEndpoints()) {
try (FlightStream stream = sqlClient.getStream(endpoint.getTicket())) {
assertAll(
- () -> MatcherAssert.assertThat(stream.getSchema(), is(SCHEMA_INT_TABLE)),
+ () -> assertThat(stream.getSchema()).isEqualTo(SCHEMA_INT_TABLE),
() ->
- MatcherAssert.assertThat(
- getResults(stream), is(EXPECTED_RESULTS_FOR_PARAMETER_BINDING)));
+ assertThat(getResults(stream))
+ .isEqualTo(EXPECTED_RESULTS_FOR_PARAMETER_BINDING));
}
}
}
diff --git a/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStreams.java b/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStreams.java
index 71c0dc88e4..3f527f961e 100644
--- a/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStreams.java
+++ b/flight/flight-sql/src/test/java/org/apache/arrow/flight/sql/test/TestFlightSqlStreams.java
@@ -22,7 +22,7 @@
import static org.apache.arrow.flight.sql.util.FlightStreamUtils.getResults;
import static org.apache.arrow.util.AutoCloseables.close;
import static org.apache.arrow.vector.types.Types.MinorType.INT;
-import static org.hamcrest.CoreMatchers.is;
+import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertAll;
import com.google.common.collect.ImmutableList;
@@ -53,7 +53,6 @@
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.apache.arrow.vector.util.Text;
-import org.hamcrest.MatcherAssert;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
@@ -245,15 +244,15 @@ public void testGetTablesResultNoSchema() throws Exception {
sqlClient.getTables(null, null, null, null, false).getEndpoints().get(0).getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA)),
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_TABLES_SCHEMA_NO_SCHEMA),
() -> {
final List> results = getResults(stream);
final List> expectedResults =
ImmutableList.of(
// catalog_name | schema_name | table_name | table_type | table_schema
asList(null, null, "test_table", "TABLE"));
- MatcherAssert.assertThat(results, is(expectedResults));
+ assertThat(results).isEqualTo(expectedResults);
});
}
}
@@ -264,15 +263,15 @@ public void testGetTableTypesResult() throws Exception {
sqlClient.getStream(sqlClient.getTableTypes().getEndpoints().get(0).getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_TABLE_TYPES_SCHEMA)),
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_TABLE_TYPES_SCHEMA),
() -> {
final List> tableTypes = getResults(stream);
final List> expectedTableTypes =
ImmutableList.of(
// table_type
singletonList("TABLE"));
- MatcherAssert.assertThat(tableTypes, is(expectedTableTypes));
+ assertThat(tableTypes).isEqualTo(expectedTableTypes);
});
}
}
@@ -283,9 +282,9 @@ public void testGetSqlInfoResults() throws Exception {
try (final FlightStream stream = sqlClient.getStream(info.getEndpoints().get(0).getTicket())) {
assertAll(
() ->
- MatcherAssert.assertThat(
- stream.getSchema(), is(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA)),
- () -> MatcherAssert.assertThat(getResults(stream), is(emptyList())));
+ assertThat(stream.getSchema())
+ .isEqualTo(FlightSqlProducer.Schemas.GET_SQL_INFO_SCHEMA),
+ () -> assertThat(getResults(stream)).isEqualTo(emptyList()));
}
}
@@ -303,7 +302,7 @@ public void testGetTypeInfo() throws Exception {
"Integer", "4", "400", null, null, "3", "true", null, "true", null, "true",
"Integer", null, null, "4", null, "10", null));
- MatcherAssert.assertThat(results, is(matchers));
+ assertThat(results).isEqualTo(matchers);
}
}
@@ -317,10 +316,8 @@ public void testExecuteQuery() throws Exception {
.get(0)
.getTicket())) {
assertAll(
- () ->
- MatcherAssert.assertThat(stream.getSchema(), is(FlightSqlTestProducer.FIXED_SCHEMA)),
- () ->
- MatcherAssert.assertThat(getResults(stream), is(singletonList(singletonList("1")))));
+ () -> assertThat(stream.getSchema()).isEqualTo(FlightSqlTestProducer.FIXED_SCHEMA),
+ () -> assertThat(getResults(stream)).isEqualTo(singletonList(singletonList("1"))));
}
}
}
From fad4e142f398492533bb1dea32accfcad0a33be8 Mon Sep 17 00:00:00 2001
From: David Li
Date: Mon, 2 Jun 2025 11:42:45 +0900
Subject: [PATCH 012/147] MINOR: Add missing permission to milestone assignment
bot (#673)
## What's Changed
This step needs permissions to write to issues so we can set the
milestone.
---
.github/workflows/dev_pr.yml | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/.github/workflows/dev_pr.yml b/.github/workflows/dev_pr.yml
index 34b3363c50..7352137b09 100644
--- a/.github/workflows/dev_pr.yml
+++ b/.github/workflows/dev_pr.yml
@@ -80,5 +80,9 @@ jobs:
if: '! github.event.pull_request.draft'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ permissions:
+ contents: read
+ issues: write
+ pull-requests: write
run: |
./.github/workflows/dev_pr_milestone.sh "${GITHUB_REPOSITORY}" ${{ github.event.number }}
From 17958324c65e0c2109733ed4022ccd25b23e1ff2 Mon Sep 17 00:00:00 2001
From: rtadepalli <105760760+rtadepalli@users.noreply.github.com>
Date: Wed, 4 Jun 2025 08:15:34 -0400
Subject: [PATCH 013/147] GH-774: Consoliate
`BitVectorHelper.getValidityBufferSize` and
`BaseValueVector.getValidityBufferSizeFromCount` (#775)
## What's Changed
Just removing some duplicate functions in anticipation of cleaning out
some transferPair duplication across complex vectors.
Closes #774
---------
Co-authored-by: David Li
---
.../arrow/vector/BaseFixedWidthVector.java | 10 ++++----
.../vector/BaseLargeVariableWidthVector.java | 6 ++---
.../apache/arrow/vector/BaseValueVector.java | 9 +++++++-
.../arrow/vector/BaseVariableWidthVector.java | 6 ++---
.../vector/BaseVariableWidthViewVector.java | 6 ++---
.../org/apache/arrow/vector/BitVector.java | 6 ++---
.../apache/arrow/vector/BitVectorHelper.java | 12 +++++-----
.../vector/complex/FixedSizeListVector.java | 16 +++++++------
.../arrow/vector/complex/LargeListVector.java | 18 ++++++++-------
.../vector/complex/LargeListViewVector.java | 20 ++++++++--------
.../arrow/vector/complex/ListVector.java | 20 ++++++++--------
.../arrow/vector/complex/ListViewVector.java | 20 ++++++++--------
.../arrow/vector/complex/MapVector.java | 2 +-
.../arrow/vector/complex/StructVector.java | 23 ++++++++++---------
.../arrow/vector/ipc/JsonFileReader.java | 3 ++-
.../arrow/vector/TestLargeListVector.java | 3 ++-
.../arrow/vector/TestLargeListViewVector.java | 3 ++-
.../apache/arrow/vector/TestListVector.java | 3 ++-
.../arrow/vector/TestListViewVector.java | 3 ++-
.../apache/arrow/vector/TestValueVector.java | 13 ++++++-----
.../vector/TestVariableWidthViewVector.java | 3 ++-
.../arrow/vector/TestVectorUnloadLoad.java | 3 ++-
22 files changed, 117 insertions(+), 91 deletions(-)
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java
index 4be55396b7..d126266cf5 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java
@@ -359,7 +359,7 @@ public int getBufferSizeFor(final int count) {
if (count == 0) {
return 0;
}
- return (count * typeWidth) + getValidityBufferSizeFromCount(count);
+ return (count * typeWidth) + BitVectorHelper.getValidityBufferSizeFromCount(count);
}
/**
@@ -372,7 +372,7 @@ public int getBufferSize() {
if (valueCount == 0) {
return 0;
}
- return (valueCount * typeWidth) + getValidityBufferSizeFromCount(valueCount);
+ return (valueCount * typeWidth) + BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
}
/**
@@ -536,10 +536,10 @@ private void setReaderAndWriterIndex() {
validityBuffer.writerIndex(0);
valueBuffer.writerIndex(0);
} else {
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
if (typeWidth == 0) {
/* specialized handling for BitVector */
- valueBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ valueBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
} else {
valueBuffer.writerIndex((long) valueCount * typeWidth);
}
@@ -664,7 +664,7 @@ private void splitAndTransferValidityBuffer(
int startIndex, int length, BaseFixedWidthVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java
index 7e0d0affc6..4245e0053b 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java
@@ -379,7 +379,7 @@ private void setReaderAndWriterIndex() {
valueBuffer.writerIndex(0);
} else {
final long lastDataOffset = getStartOffset(valueCount);
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
offsetBuffer.writerIndex((long) (valueCount + 1) * OFFSET_WIDTH);
valueBuffer.writerIndex(lastDataOffset);
}
@@ -633,7 +633,7 @@ public int getBufferSizeFor(final int valueCount) {
return 0;
}
- final long validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final long validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
final long offsetBufferSize = (long) (valueCount + 1) * OFFSET_WIDTH;
/* get the end offset for this valueCount */
final long dataBufferSize = getStartOffset(valueCount);
@@ -816,7 +816,7 @@ private void splitAndTransferValidityBuffer(
int startIndex, int length, BaseLargeVariableWidthVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java
index 9befcb890f..7bff431e40 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java
@@ -110,7 +110,14 @@ protected ArrowBuf releaseBuffer(ArrowBuf buffer) {
return buffer;
}
- /* number of bytes for the validity buffer for the given valueCount */
+ /**
+ * Compute the size of validity buffer required to manage a given number of elements in a vector.
+ *
+ * @param valueCount number of elements in the vector
+ * @return buffer size
+ * @deprecated -- use {@link BitVectorHelper#getValidityBufferSizeFromCount} instead.
+ */
+ @Deprecated(forRemoval = true, since = "18.4.0")
protected static int getValidityBufferSizeFromCount(final int valueCount) {
return DataSizeRoundingUtil.divideBy8Ceil(valueCount);
}
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java
index 1609e64ca5..4f681311ed 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java
@@ -395,7 +395,7 @@ private void setReaderAndWriterIndex() {
valueBuffer.writerIndex(0);
} else {
final int lastDataOffset = getStartOffset(valueCount);
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
offsetBuffer.writerIndex((long) (valueCount + 1) * OFFSET_WIDTH);
valueBuffer.writerIndex(lastDataOffset);
}
@@ -673,7 +673,7 @@ public int getBufferSizeFor(final int valueCount) {
return 0;
}
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
final int offsetBufferSize = (valueCount + 1) * OFFSET_WIDTH;
/* get the end offset for this valueCount */
final int dataBufferSize = offsetBuffer.getInt((long) valueCount * OFFSET_WIDTH);
@@ -867,7 +867,7 @@ private void splitAndTransferValidityBuffer(
final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
final int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- final int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ final int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
final int offset = startIndex % 8;
if (offset == 0) {
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java
index beda91dc3f..5e25ffa568 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java
@@ -400,7 +400,7 @@ private void setReaderAndWriterIndex() {
validityBuffer.writerIndex(0);
viewBuffer.writerIndex(0);
} else {
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
viewBuffer.writerIndex(valueCount * ELEMENT_SIZE);
}
}
@@ -683,7 +683,7 @@ public int getBufferSizeFor(final int valueCount) {
return 0;
}
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
final int viewBufferSize = valueCount * ELEMENT_SIZE;
final int dataBufferSize = getDataBufferSize();
return validityBufferSize + viewBufferSize + dataBufferSize;
@@ -872,7 +872,7 @@ private void splitAndTransferValidityBuffer(
final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
final int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- final int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ final int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
final int offset = startIndex % 8;
if (offset == 0) {
diff --git a/vector/src/main/java/org/apache/arrow/vector/BitVector.java b/vector/src/main/java/org/apache/arrow/vector/BitVector.java
index f8e3342625..ecee02f665 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BitVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BitVector.java
@@ -98,7 +98,7 @@ public MinorType getMinorType() {
*/
@Override
public void setInitialCapacity(int valueCount) {
- final int size = getValidityBufferSizeFromCount(valueCount);
+ final int size = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
if (size * 2L > MAX_ALLOCATION_SIZE) {
throw new OversizedAllocationException("Requested amount of memory is more than max allowed");
}
@@ -121,7 +121,7 @@ public int getBufferSizeFor(final int count) {
if (count == 0) {
return 0;
}
- return 2 * getValidityBufferSizeFromCount(count);
+ return 2 * BitVectorHelper.getValidityBufferSizeFromCount(count);
}
/**
@@ -165,7 +165,7 @@ private ArrowBuf splitAndTransferBuffer(
int startIndex, int length, ArrowBuf sourceBuffer, ArrowBuf destBuffer) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
diff --git a/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java b/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java
index 0ac56691a6..bc2c3da98f 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BitVectorHelper.java
@@ -135,11 +135,11 @@ public static void setValidityBit(ArrowBuf validityBuffer, int index, int value)
public static ArrowBuf setValidityBit(
ArrowBuf validityBuffer, BufferAllocator allocator, int valueCount, int index, int value) {
if (validityBuffer == null) {
- validityBuffer = allocator.buffer(getValidityBufferSize(valueCount));
+ validityBuffer = allocator.buffer(getValidityBufferSizeFromCount(valueCount));
}
setValidityBit(validityBuffer, index, value);
if (index == (valueCount - 1)) {
- validityBuffer.writerIndex(getValidityBufferSize(valueCount));
+ validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
}
return validityBuffer;
@@ -165,7 +165,7 @@ public static int get(final ArrowBuf buffer, int index) {
* @param valueCount number of elements in the vector
* @return buffer size
*/
- public static int getValidityBufferSize(int valueCount) {
+ public static int getValidityBufferSizeFromCount(int valueCount) {
return DataSizeRoundingUtil.divideBy8Ceil(valueCount);
}
@@ -182,7 +182,7 @@ public static int getNullCount(final ArrowBuf validityBuffer, final int valueCou
return 0;
}
int count = 0;
- final int sizeInBytes = getValidityBufferSize(valueCount);
+ final int sizeInBytes = getValidityBufferSizeFromCount(valueCount);
// If value count is not a multiple of 8, then calculate number of used bits in the last byte
final int remainder = valueCount % 8;
final int fullBytesCount = remainder == 0 ? sizeInBytes : sizeInBytes - 1;
@@ -233,7 +233,7 @@ public static boolean checkAllBitsEqualTo(
if (valueCount == 0) {
return true;
}
- final int sizeInBytes = getValidityBufferSize(valueCount);
+ final int sizeInBytes = getValidityBufferSizeFromCount(valueCount);
// boundary check
validityBuffer.checkBytes(0, sizeInBytes);
@@ -325,7 +325,7 @@ public static ArrowBuf loadValidityBuffer(
sourceValidityBuffer == null || sourceValidityBuffer.capacity() == 0;
if (isValidityBufferNull
&& (fieldNode.getNullCount() == 0 || fieldNode.getNullCount() == valueCount)) {
- newBuffer = allocator.buffer(getValidityBufferSize(valueCount));
+ newBuffer = allocator.buffer(getValidityBufferSizeFromCount(valueCount));
newBuffer.setZero(0, newBuffer.capacity());
if (fieldNode.getNullCount() != 0) {
/* all NULLs */
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
index c762eb5172..36d9ff40ed 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
@@ -110,7 +110,8 @@ public FixedSizeListVector(
this.listSize = ((ArrowType.FixedSizeList) field.getFieldType().getType()).getListSize();
Preconditions.checkArgument(listSize >= 0, "list size must be non-negative");
this.valueCount = 0;
- this.validityAllocationSizeInBytes = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
+ this.validityAllocationSizeInBytes =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
}
@Override
@@ -189,7 +190,7 @@ public List getFieldBuffers() {
private void setReaderAndWriterIndex() {
validityBuffer.readerIndex(0);
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
}
/**
@@ -268,7 +269,8 @@ private void reallocValidityBuffer() {
if (validityAllocationSizeInBytes > 0) {
newAllocationSize = validityAllocationSizeInBytes;
} else {
- newAllocationSize = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
+ newAllocationSize =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
}
}
@@ -311,7 +313,7 @@ public UnionFixedSizeListWriter getWriter() {
@Override
public void setInitialCapacity(int numRecords) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
vector.setInitialCapacity(numRecords * listSize);
}
@@ -328,7 +330,7 @@ public int getBufferSize() {
if (getValueCount() == 0) {
return 0;
}
- return getValidityBufferSizeFromCount(valueCount) + vector.getBufferSize();
+ return BitVectorHelper.getValidityBufferSizeFromCount(valueCount) + vector.getBufferSize();
}
@Override
@@ -336,7 +338,7 @@ public int getBufferSizeFor(int valueCount) {
if (valueCount == 0) {
return 0;
}
- return getValidityBufferSizeFromCount(valueCount)
+ return BitVectorHelper.getValidityBufferSizeFromCount(valueCount)
+ vector.getBufferSizeFor(valueCount * listSize);
}
@@ -654,7 +656,7 @@ private void splitAndTransferValidityBuffer(
int startIndex, int length, FixedSizeListVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java
index ed075352c9..71633441cb 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java
@@ -131,7 +131,8 @@ public LargeListVector(Field field, BufferAllocator allocator, CallBack callBack
this.field = field;
this.validityBuffer = allocator.getEmpty();
this.callBack = callBack;
- this.validityAllocationSizeInBytes = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
+ this.validityAllocationSizeInBytes =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
this.lastSet = -1;
this.offsetBuffer = allocator.getEmpty();
this.vector = vector == null ? DEFAULT_DATA_VECTOR : vector;
@@ -156,7 +157,7 @@ public void initializeChildrenFromFields(List children) {
@Override
public void setInitialCapacity(int numRecords) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
offsetAllocationSizeInBytes = (long) (numRecords + 1) * OFFSET_WIDTH;
if (vector instanceof BaseFixedWidthVector || vector instanceof BaseVariableWidthVector) {
vector.setInitialCapacity(numRecords * RepeatedValueVector.DEFAULT_REPEAT_PER_RECORD);
@@ -184,7 +185,7 @@ public void setInitialCapacity(int numRecords) {
*/
@Override
public void setInitialCapacity(int numRecords, double density) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
if ((numRecords * density) >= Integer.MAX_VALUE) {
throw new OversizedAllocationException("Requested amount of memory is more than max allowed");
}
@@ -311,7 +312,7 @@ private void setReaderAndWriterIndex() {
validityBuffer.writerIndex(0);
offsetBuffer.writerIndex(0);
} else {
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
offsetBuffer.writerIndex((valueCount + 1) * OFFSET_WIDTH);
}
}
@@ -442,7 +443,8 @@ private void reallocValidityBuffer() {
if (validityAllocationSizeInBytes > 0) {
newAllocationSize = validityAllocationSizeInBytes;
} else {
- newAllocationSize = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
+ newAllocationSize =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
}
}
newAllocationSize = CommonUtil.nextPowerOfTwo(newAllocationSize);
@@ -699,7 +701,7 @@ private void splitAndTransferValidityBuffer(
int startIndex, int length, LargeListVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
@@ -821,7 +823,7 @@ public int getBufferSize() {
return 0;
}
final int offsetBufferSize = (valueCount + 1) * OFFSET_WIDTH;
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
return offsetBufferSize + validityBufferSize + vector.getBufferSize();
}
@@ -830,7 +832,7 @@ public int getBufferSizeFor(int valueCount) {
if (valueCount == 0) {
return 0;
}
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
long innerVectorValueCount = offsetBuffer.getLong((long) valueCount * OFFSET_WIDTH);
return ((valueCount + 1) * OFFSET_WIDTH)
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java
index 84c6f03edb..1b7e6b2280 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java
@@ -113,7 +113,8 @@ public LargeListViewVector(Field field, BufferAllocator allocator, CallBack call
this.validityBuffer = allocator.getEmpty();
this.field = field;
this.callBack = callBack;
- this.validityAllocationSizeInBytes = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
+ this.validityAllocationSizeInBytes =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
}
@Override
@@ -134,7 +135,7 @@ public void initializeChildrenFromFields(List children) {
@Override
public void setInitialCapacity(int numRecords) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords);
}
@@ -157,7 +158,7 @@ public void setInitialCapacity(int numRecords) {
*/
@Override
public void setInitialCapacity(int numRecords, double density) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords, density);
}
@@ -176,7 +177,7 @@ public void setInitialCapacity(int numRecords, double density) {
*/
@Override
public void setInitialTotalCapacity(int numRecords, int totalNumberOfElements) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialTotalCapacity(numRecords, totalNumberOfElements);
}
@@ -226,7 +227,7 @@ private void setReaderAndWriterIndex() {
offsetBuffer.writerIndex(0);
sizeBuffer.writerIndex(0);
} else {
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
offsetBuffer.writerIndex((long) valueCount * OFFSET_WIDTH);
sizeBuffer.writerIndex((long) valueCount * SIZE_WIDTH);
}
@@ -323,7 +324,8 @@ private long getNewAllocationSize(int currentBufferCapacity) {
if (validityAllocationSizeInBytes > 0) {
newAllocationSize = validityAllocationSizeInBytes;
} else {
- newAllocationSize = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
+ newAllocationSize =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
}
}
newAllocationSize = CommonUtil.nextPowerOfTwo(newAllocationSize);
@@ -536,7 +538,7 @@ private void splitAndTransferValidityBuffer(
int startIndex, int length, LargeListViewVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
@@ -629,7 +631,7 @@ public int getBufferSize() {
}
final int offsetBufferSize = valueCount * OFFSET_WIDTH;
final int sizeBufferSize = valueCount * SIZE_WIDTH;
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
return offsetBufferSize + sizeBufferSize + validityBufferSize + vector.getBufferSize();
}
@@ -644,7 +646,7 @@ public int getBufferSizeFor(int valueCount) {
if (valueCount == 0) {
return 0;
}
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
return super.getBufferSizeFor(valueCount) + validityBufferSize;
}
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
index 3daeb6d77b..a8e8dcc436 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
@@ -108,7 +108,8 @@ public ListVector(Field field, BufferAllocator allocator, CallBack callBack) {
this.validityBuffer = allocator.getEmpty();
this.field = field;
this.callBack = callBack;
- this.validityAllocationSizeInBytes = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
+ this.validityAllocationSizeInBytes =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
this.lastSet = -1;
}
@@ -130,7 +131,7 @@ public void initializeChildrenFromFields(List children) {
@Override
public void setInitialCapacity(int numRecords) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords);
}
@@ -153,7 +154,7 @@ public void setInitialCapacity(int numRecords) {
*/
@Override
public void setInitialCapacity(int numRecords, double density) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords, density);
}
@@ -172,7 +173,7 @@ public void setInitialCapacity(int numRecords, double density) {
*/
@Override
public void setInitialTotalCapacity(int numRecords, int totalNumberOfElements) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialTotalCapacity(numRecords, totalNumberOfElements);
}
@@ -269,7 +270,7 @@ private void setReaderAndWriterIndex() {
validityBuffer.writerIndex(0);
offsetBuffer.writerIndex(0);
} else {
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
offsetBuffer.writerIndex((valueCount + 1) * OFFSET_WIDTH);
}
}
@@ -366,7 +367,8 @@ private long getNewAllocationSize(int currentBufferCapacity) {
if (validityAllocationSizeInBytes > 0) {
newAllocationSize = validityAllocationSizeInBytes;
} else {
- newAllocationSize = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
+ newAllocationSize =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
}
}
newAllocationSize = CommonUtil.nextPowerOfTwo(newAllocationSize);
@@ -579,7 +581,7 @@ public void splitAndTransfer(int startIndex, int length) {
private void splitAndTransferValidityBuffer(int startIndex, int length, ListVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
@@ -678,7 +680,7 @@ public int getBufferSize() {
return 0;
}
final int offsetBufferSize = (valueCount + 1) * OFFSET_WIDTH;
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
return offsetBufferSize + validityBufferSize + vector.getBufferSize();
}
@@ -687,7 +689,7 @@ public int getBufferSizeFor(int valueCount) {
if (valueCount == 0) {
return 0;
}
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
return super.getBufferSizeFor(valueCount) + validityBufferSize;
}
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java
index 9b4e6b4c0c..ada25bbaf5 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java
@@ -112,7 +112,8 @@ public ListViewVector(Field field, BufferAllocator allocator, CallBack callBack)
this.validityBuffer = allocator.getEmpty();
this.field = field;
this.callBack = callBack;
- this.validityAllocationSizeInBytes = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
+ this.validityAllocationSizeInBytes =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION);
}
@Override
@@ -133,7 +134,7 @@ public void initializeChildrenFromFields(List children) {
@Override
public void setInitialCapacity(int numRecords) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords);
}
@@ -156,7 +157,7 @@ public void setInitialCapacity(int numRecords) {
*/
@Override
public void setInitialCapacity(int numRecords, double density) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords, density);
}
@@ -175,7 +176,7 @@ public void setInitialCapacity(int numRecords, double density) {
*/
@Override
public void setInitialTotalCapacity(int numRecords, int totalNumberOfElements) {
- validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
+ validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSizeFromCount(numRecords);
super.setInitialTotalCapacity(numRecords, totalNumberOfElements);
}
@@ -225,7 +226,7 @@ private void setReaderAndWriterIndex() {
offsetBuffer.writerIndex(0);
sizeBuffer.writerIndex(0);
} else {
- validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
+ validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSizeFromCount(valueCount));
offsetBuffer.writerIndex(valueCount * OFFSET_WIDTH);
sizeBuffer.writerIndex(valueCount * SIZE_WIDTH);
}
@@ -322,7 +323,8 @@ private long getNewAllocationSize(int currentBufferCapacity) {
if (validityAllocationSizeInBytes > 0) {
newAllocationSize = validityAllocationSizeInBytes;
} else {
- newAllocationSize = getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
+ newAllocationSize =
+ BitVectorHelper.getValidityBufferSizeFromCount(INITIAL_VALUE_ALLOCATION) * 2L;
}
}
newAllocationSize = CommonUtil.nextPowerOfTwo(newAllocationSize);
@@ -542,7 +544,7 @@ public void splitAndTransfer(int startIndex, int length) {
private void splitAndTransferValidityBuffer(int startIndex, int length, ListViewVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
@@ -634,7 +636,7 @@ public int getBufferSize() {
}
final int offsetBufferSize = valueCount * OFFSET_WIDTH;
final int sizeBufferSize = valueCount * SIZE_WIDTH;
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
return offsetBufferSize + sizeBufferSize + validityBufferSize + vector.getBufferSize();
}
@@ -649,7 +651,7 @@ public int getBufferSizeFor(int valueCount) {
if (valueCount == 0) {
return 0;
}
- final int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
+ final int validityBufferSize = BitVectorHelper.getValidityBufferSizeFromCount(valueCount);
return super.getBufferSizeFor(valueCount) + validityBufferSize;
}
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
index 23cda8401b..5eb857ab94 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
@@ -238,7 +238,7 @@ public void splitAndTransfer(int startIndex, int length) {
private void splitAndTransferValidityBuffer(int startIndex, int length, MapVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/StructVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/StructVector.java
index ca5f572034..5e5bb7fc21 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/StructVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/StructVector.java
@@ -18,6 +18,7 @@
import static org.apache.arrow.memory.util.LargeMemoryUtil.checkedCastToInt;
import static org.apache.arrow.util.Preconditions.checkNotNull;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import java.util.ArrayList;
import java.util.Arrays;
@@ -89,7 +90,7 @@ public StructVector(
super(name, checkNotNull(allocator), fieldType, callBack);
this.validityBuffer = allocator.getEmpty();
this.validityAllocationSizeInBytes =
- BitVectorHelper.getValidityBufferSize(BaseValueVector.INITIAL_VALUE_ALLOCATION);
+ getValidityBufferSizeFromCount(BaseValueVector.INITIAL_VALUE_ALLOCATION);
}
/**
@@ -118,7 +119,7 @@ public StructVector(
allowConflictPolicyChanges);
this.validityBuffer = allocator.getEmpty();
this.validityAllocationSizeInBytes =
- BitVectorHelper.getValidityBufferSize(BaseValueVector.INITIAL_VALUE_ALLOCATION);
+ getValidityBufferSizeFromCount(BaseValueVector.INITIAL_VALUE_ALLOCATION);
}
/**
@@ -132,7 +133,7 @@ public StructVector(Field field, BufferAllocator allocator, CallBack callBack) {
super(field, checkNotNull(allocator), callBack);
this.validityBuffer = allocator.getEmpty();
this.validityAllocationSizeInBytes =
- BitVectorHelper.getValidityBufferSize(BaseValueVector.INITIAL_VALUE_ALLOCATION);
+ getValidityBufferSizeFromCount(BaseValueVector.INITIAL_VALUE_ALLOCATION);
}
/**
@@ -153,7 +154,7 @@ public StructVector(
super(field, checkNotNull(allocator), callBack, conflictPolicy, allowConflictPolicyChanges);
this.validityBuffer = allocator.getEmpty();
this.validityAllocationSizeInBytes =
- BitVectorHelper.getValidityBufferSize(BaseValueVector.INITIAL_VALUE_ALLOCATION);
+ getValidityBufferSizeFromCount(BaseValueVector.INITIAL_VALUE_ALLOCATION);
}
@Override
@@ -182,7 +183,7 @@ public List getFieldBuffers() {
private void setReaderAndWriterIndex() {
validityBuffer.readerIndex(0);
- validityBuffer.writerIndex(BitVectorHelper.getValidityBufferSize(valueCount));
+ validityBuffer.writerIndex(getValidityBufferSizeFromCount(valueCount));
}
/**
@@ -318,7 +319,7 @@ public void splitAndTransfer(int startIndex, int length) {
private void splitAndTransferValidityBuffer(int startIndex, int length, StructVector target) {
int firstByteSource = BitVectorHelper.byteIndex(startIndex);
int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSize(length);
+ int byteSizeTarget = getValidityBufferSizeFromCount(length);
int offset = startIndex % 8;
if (length > 0) {
@@ -464,7 +465,7 @@ public int getBufferSize() {
if (valueCount == 0) {
return 0;
}
- return super.getBufferSize() + BitVectorHelper.getValidityBufferSize(valueCount);
+ return super.getBufferSize() + getValidityBufferSizeFromCount(valueCount);
}
/**
@@ -478,18 +479,18 @@ public int getBufferSizeFor(final int valueCount) {
if (valueCount == 0) {
return 0;
}
- return super.getBufferSizeFor(valueCount) + BitVectorHelper.getValidityBufferSize(valueCount);
+ return super.getBufferSizeFor(valueCount) + getValidityBufferSizeFromCount(valueCount);
}
@Override
public void setInitialCapacity(int numRecords) {
- validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSize(numRecords);
+ validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords);
}
@Override
public void setInitialCapacity(int numRecords, double density) {
- validityAllocationSizeInBytes = BitVectorHelper.getValidityBufferSize(numRecords);
+ validityAllocationSizeInBytes = getValidityBufferSizeFromCount(numRecords);
super.setInitialCapacity(numRecords, density);
}
@@ -547,7 +548,7 @@ private long getNewAllocationSize(int currentBufferCapacity) {
newAllocationSize = validityAllocationSizeInBytes;
} else {
newAllocationSize =
- BitVectorHelper.getValidityBufferSize(BaseValueVector.INITIAL_VALUE_ALLOCATION) * 2L;
+ getValidityBufferSizeFromCount(BaseValueVector.INITIAL_VALUE_ALLOCATION) * 2L;
}
}
newAllocationSize = CommonUtil.nextPowerOfTwo(newAllocationSize);
diff --git a/vector/src/main/java/org/apache/arrow/vector/ipc/JsonFileReader.java b/vector/src/main/java/org/apache/arrow/vector/ipc/JsonFileReader.java
index fe0803d298..e4bab7eb80 100644
--- a/vector/src/main/java/org/apache/arrow/vector/ipc/JsonFileReader.java
+++ b/vector/src/main/java/org/apache/arrow/vector/ipc/JsonFileReader.java
@@ -20,6 +20,7 @@
import static com.fasterxml.jackson.core.JsonToken.END_OBJECT;
import static com.fasterxml.jackson.core.JsonToken.START_ARRAY;
import static com.fasterxml.jackson.core.JsonToken.START_OBJECT;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.apache.arrow.vector.BufferLayout.BufferType.DATA;
import static org.apache.arrow.vector.BufferLayout.BufferType.OFFSET;
import static org.apache.arrow.vector.BufferLayout.BufferType.SIZE;
@@ -381,7 +382,7 @@ private class BufferHelper {
new BufferReader() {
@Override
protected ArrowBuf read(BufferAllocator allocator, int count) throws IOException {
- final int bufferSize = BitVectorHelper.getValidityBufferSize(count);
+ final int bufferSize = getValidityBufferSizeFromCount(count);
ArrowBuf buf = allocator.buffer(bufferSize);
// C++ integration test fails without this.
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestLargeListVector.java b/vector/src/test/java/org/apache/arrow/vector/TestLargeListVector.java
index 101d942d2a..d5cbf925b2 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestLargeListVector.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestLargeListVector.java
@@ -16,6 +16,7 @@
*/
package org.apache.arrow.vector;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
@@ -943,7 +944,7 @@ public void testGetBufferSizeFor() {
int[] indices = new int[] {0, 2, 4, 6, 10, 14};
for (int valueCount = 1; valueCount <= 5; valueCount++) {
- int validityBufferSize = BitVectorHelper.getValidityBufferSize(valueCount);
+ int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
int offsetBufferSize = (valueCount + 1) * LargeListVector.OFFSET_WIDTH;
int expectedSize =
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestLargeListViewVector.java b/vector/src/test/java/org/apache/arrow/vector/TestLargeListViewVector.java
index 26e7bb4a0d..256aa99687 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestLargeListViewVector.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestLargeListViewVector.java
@@ -16,6 +16,7 @@
*/
package org.apache.arrow.vector;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertSame;
@@ -1062,7 +1063,7 @@ public void testGetBufferSizeFor() {
int[] indices = new int[] {0, 2, 4, 6, 10, 14};
for (int valueCount = 1; valueCount <= 5; valueCount++) {
- int validityBufferSize = BitVectorHelper.getValidityBufferSize(valueCount);
+ int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
int offsetBufferSize = valueCount * BaseLargeRepeatedValueViewVector.OFFSET_WIDTH;
int sizeBufferSize = valueCount * BaseLargeRepeatedValueViewVector.SIZE_WIDTH;
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestListVector.java b/vector/src/test/java/org/apache/arrow/vector/TestListVector.java
index 1d6fa39f9e..5b2043a014 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestListVector.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestListVector.java
@@ -16,6 +16,7 @@
*/
package org.apache.arrow.vector;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
@@ -1123,7 +1124,7 @@ public void testGetBufferSizeFor() {
int[] indices = new int[] {0, 2, 4, 6, 10, 14};
for (int valueCount = 1; valueCount <= 5; valueCount++) {
- int validityBufferSize = BitVectorHelper.getValidityBufferSize(valueCount);
+ int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
int offsetBufferSize = (valueCount + 1) * BaseRepeatedValueVector.OFFSET_WIDTH;
int expectedSize =
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestListViewVector.java b/vector/src/test/java/org/apache/arrow/vector/TestListViewVector.java
index 639585fc48..2f282e1988 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestListViewVector.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestListViewVector.java
@@ -16,6 +16,7 @@
*/
package org.apache.arrow.vector;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
@@ -1075,7 +1076,7 @@ public void testGetBufferSizeFor() {
int[] indices = new int[] {0, 2, 4, 6, 10, 14};
for (int valueCount = 1; valueCount <= 5; valueCount++) {
- int validityBufferSize = BitVectorHelper.getValidityBufferSize(valueCount);
+ int validityBufferSize = getValidityBufferSizeFromCount(valueCount);
int offsetBufferSize = valueCount * BaseRepeatedValueViewVector.OFFSET_WIDTH;
int sizeBufferSize = valueCount * BaseRepeatedValueViewVector.SIZE_WIDTH;
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java b/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java
index daec331831..ac82246671 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestValueVector.java
@@ -16,6 +16,7 @@
*/
package org.apache.arrow.vector;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.apache.arrow.vector.TestUtils.newVarBinaryVector;
import static org.apache.arrow.vector.TestUtils.newVarCharVector;
import static org.apache.arrow.vector.TestUtils.newVector;
@@ -1233,7 +1234,7 @@ public void testSplitAndTransfer3() {
// the size needed for the validity buffer
final long validitySize =
DefaultRoundingPolicy.DEFAULT_ROUNDING_POLICY.getRoundedSize(
- BaseValueVector.getValidityBufferSizeFromCount(2));
+ getValidityBufferSizeFromCount(2));
assertEquals(allocatedMem + validitySize, allocator.getAllocatedMemory());
// The validity and offset buffers are sliced from a same buffer.See
// BaseFixedWidthVector#allocateBytes.
@@ -2464,7 +2465,7 @@ public void testDefaultAllocNewAll() {
assertTrue(intVector.getValueCapacity() >= defaultCapacity);
expectedSize =
(defaultCapacity * IntVector.TYPE_WIDTH)
- + BaseFixedWidthVector.getValidityBufferSizeFromCount(defaultCapacity);
+ + getValidityBufferSizeFromCount(defaultCapacity);
assertTrue(childAllocator.getAllocatedMemory() - beforeSize <= expectedSize * 1.05);
// verify that the wastage is within bounds for BigIntVector.
@@ -2473,7 +2474,7 @@ public void testDefaultAllocNewAll() {
assertTrue(bigIntVector.getValueCapacity() >= defaultCapacity);
expectedSize =
(defaultCapacity * bigIntVector.TYPE_WIDTH)
- + BaseFixedWidthVector.getValidityBufferSizeFromCount(defaultCapacity);
+ + getValidityBufferSizeFromCount(defaultCapacity);
assertTrue(childAllocator.getAllocatedMemory() - beforeSize <= expectedSize * 1.05);
// verify that the wastage is within bounds for DecimalVector.
@@ -2482,7 +2483,7 @@ public void testDefaultAllocNewAll() {
assertTrue(decimalVector.getValueCapacity() >= defaultCapacity);
expectedSize =
(defaultCapacity * decimalVector.TYPE_WIDTH)
- + BaseFixedWidthVector.getValidityBufferSizeFromCount(defaultCapacity);
+ + getValidityBufferSizeFromCount(defaultCapacity);
assertTrue(childAllocator.getAllocatedMemory() - beforeSize <= expectedSize * 1.05);
// verify that the wastage is within bounds for VarCharVector.
@@ -2492,7 +2493,7 @@ public void testDefaultAllocNewAll() {
assertTrue(varCharVector.getValueCapacity() >= defaultCapacity - 1);
expectedSize =
(defaultCapacity * VarCharVector.OFFSET_WIDTH)
- + BaseFixedWidthVector.getValidityBufferSizeFromCount(defaultCapacity)
+ + getValidityBufferSizeFromCount(defaultCapacity)
+ defaultCapacity * 8;
// wastage should be less than 5%.
assertTrue(childAllocator.getAllocatedMemory() - beforeSize <= expectedSize * 1.05);
@@ -2501,7 +2502,7 @@ public void testDefaultAllocNewAll() {
beforeSize = childAllocator.getAllocatedMemory();
bitVector.allocateNew();
assertTrue(bitVector.getValueCapacity() >= defaultCapacity);
- expectedSize = BaseFixedWidthVector.getValidityBufferSizeFromCount(defaultCapacity) * 2;
+ expectedSize = getValidityBufferSizeFromCount(defaultCapacity) * 2;
assertTrue(childAllocator.getAllocatedMemory() - beforeSize <= expectedSize * 1.05);
}
}
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestVariableWidthViewVector.java b/vector/src/test/java/org/apache/arrow/vector/TestVariableWidthViewVector.java
index 7a3a1bae63..f7c66a00be 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestVariableWidthViewVector.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestVariableWidthViewVector.java
@@ -16,6 +16,7 @@
*/
package org.apache.arrow.vector;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.apache.arrow.vector.TestUtils.newVector;
import static org.apache.arrow.vector.TestUtils.newViewVarBinaryVector;
import static org.apache.arrow.vector.TestUtils.newViewVarCharVector;
@@ -2367,7 +2368,7 @@ private void testSplitAndTransferOnValiditySplitHelper(
// the allocation only consists in the size needed for the validity buffer
final long validitySize =
DefaultRoundingPolicy.DEFAULT_ROUNDING_POLICY.getRoundedSize(
- BaseValueVector.getValidityBufferSizeFromCount(2));
+ getValidityBufferSizeFromCount(2));
// we allocate view and data buffers for the target vector
assertTrue(allocatedMem + validitySize < allocator.getAllocatedMemory());
// The validity is sliced from the same buffer.See BaseFixedWidthViewVector#allocateBytes.
diff --git a/vector/src/test/java/org/apache/arrow/vector/TestVectorUnloadLoad.java b/vector/src/test/java/org/apache/arrow/vector/TestVectorUnloadLoad.java
index 6121fb67fe..782535fccc 100644
--- a/vector/src/test/java/org/apache/arrow/vector/TestVectorUnloadLoad.java
+++ b/vector/src/test/java/org/apache/arrow/vector/TestVectorUnloadLoad.java
@@ -17,6 +17,7 @@
package org.apache.arrow.vector;
import static java.util.Arrays.asList;
+import static org.apache.arrow.vector.BitVectorHelper.getValidityBufferSizeFromCount;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -215,7 +216,7 @@ public void testLoadValidityBuffer() throws IOException {
int count = 10;
ArrowBuf[] values = new ArrowBuf[4];
for (int i = 0; i < 4; i += 2) {
- ArrowBuf buf1 = allocator.buffer(BitVectorHelper.getValidityBufferSize(count));
+ ArrowBuf buf1 = allocator.buffer(getValidityBufferSizeFromCount(count));
ArrowBuf buf2 = allocator.buffer(count * 4); // integers
buf1.setZero(0, buf1.capacity());
buf2.setZero(0, buf2.capacity());
From acbc138f75711fc94e842f9b306a48a5a233bca8 Mon Sep 17 00:00:00 2001
From: rtadepalli <105760760+rtadepalli@users.noreply.github.com>
Date: Wed, 4 Jun 2025 22:57:25 -0400
Subject: [PATCH 014/147] GH-79: Move `splitAndTransferValidityBuffer` to
`BaseValueVector` (#777)
## What's Changed
Move `splitAndTransferValidityBuffer` up to `BaseValueVector`.
This PR is not touching the implementation of this function in
`StructVector` -- that is not being derived from `BaseValueVector` so
some amount of duplication is probably fine.
Closes #79
---
.../arrow/vector/BaseFixedWidthVector.java | 86 +++----------
.../vector/BaseLargeVariableWidthVector.java | 79 ++----------
.../apache/arrow/vector/BaseValueVector.java | 116 ++++++++++++++++++
.../arrow/vector/BaseVariableWidthVector.java | 79 ++----------
.../vector/BaseVariableWidthViewVector.java | 85 ++-----------
.../BaseLargeRepeatedValueViewVector.java | 1 -
.../complex/BaseRepeatedValueVector.java | 1 -
.../complex/BaseRepeatedValueViewVector.java | 1 -
.../vector/complex/FixedSizeListVector.java | 77 +-----------
.../arrow/vector/complex/LargeListVector.java | 77 +-----------
.../vector/complex/LargeListViewVector.java | 74 +----------
.../arrow/vector/complex/ListVector.java | 73 +----------
.../arrow/vector/complex/ListViewVector.java | 73 +----------
.../arrow/vector/complex/MapVector.java | 65 ----------
14 files changed, 182 insertions(+), 705 deletions(-)
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java
index d126266cf5..f6e2a3b225 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseFixedWidthVector.java
@@ -49,9 +49,7 @@ public abstract class BaseFixedWidthVector extends BaseValueVector
protected final Field field;
private int allocationMonitor;
- protected ArrowBuf validityBuffer;
protected ArrowBuf valueBuffer;
- protected int valueCount;
/**
* Constructs a new instance.
@@ -87,7 +85,7 @@ public String getName() {
/* TODO:
* Once the entire hierarchy has been refactored, move common functions
- * like getNullCount(), splitAndTransferValidityBuffer to top level
+ * like getNullCount() to top level
* base class BaseValueVector.
*
* Along with this, some class members (validityBuffer) can also be
@@ -342,9 +340,9 @@ private void allocateBytes(int valueCount) {
* slice the source buffer so we have to explicitly allocate the validityBuffer of the target
* vector. This is unlike the databuffer which we can always slice for the target vector.
*/
- private void allocateValidityBuffer(final int validityBufferSize) {
- validityBuffer = allocator.buffer(validityBufferSize);
- validityBuffer.readerIndex(0);
+ @Override
+ protected void allocateValidityBuffer(final long validityBufferSize) {
+ super.allocateValidityBuffer(validityBufferSize);
refreshValueCapacity();
}
@@ -656,72 +654,18 @@ private void splitAndTransferValueBuffer(
target.refreshValueCapacity();
}
- /**
- * Validity buffer has multiple cases of split and transfer depending on the starting position of
- * the source index.
- */
- private void splitAndTransferValidityBuffer(
- int startIndex, int length, BaseFixedWidthVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
-
- if (length > 0) {
- if (offset == 0) {
- /* slice */
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- ArrowBuf slicedValidityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer = transferBuffer(slicedValidityBuffer, target.allocator);
- target.refreshValueCapacity();
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- this.validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
-
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- this.validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
+ @Override
+ protected void sliceAndTransferValidityBuffer(
+ int startIndex, int length, BaseValueVector target) {
+ final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
+ final int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
+
+ if (target.validityBuffer != null) {
+ target.validityBuffer.getReferenceManager().release();
}
+ ArrowBuf slicedValidityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
+ target.validityBuffer = transferBuffer(slicedValidityBuffer, target.allocator);
+ ((BaseFixedWidthVector) target).refreshValueCapacity();
}
/*----------------------------------------------------------------*
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java
index 4245e0053b..6c451f10a7 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseLargeVariableWidthVector.java
@@ -52,10 +52,8 @@ public abstract class BaseLargeVariableWidthVector extends BaseValueVector
/* protected members */
public static final int OFFSET_WIDTH = 8; /* 8 byte unsigned int to track offsets */
protected static final byte[] emptyByteArray = new byte[] {};
- protected ArrowBuf validityBuffer;
protected ArrowBuf valueBuffer;
protected ArrowBuf offsetBuffer;
- protected int valueCount;
protected int lastSet;
protected final Field field;
@@ -501,10 +499,9 @@ private ArrowBuf allocateOffsetBuffer(final long size) {
}
/* allocate validity buffer */
- private void allocateValidityBuffer(final long size) {
- validityBuffer = allocator.buffer(size);
- validityBuffer.readerIndex(0);
- initValidityBuffer();
+ @Override
+ protected void allocateValidityBuffer(final long size) {
+ super.allocateValidityBuffer(size);
}
/**
@@ -809,69 +806,17 @@ private void splitAndTransferOffsetBuffer(
target.valueBuffer = transferBuffer(slicedBuffer, target.allocator);
}
- /*
- * Transfer the validity.
- */
- private void splitAndTransferValidityBuffer(
- int startIndex, int length, BaseLargeVariableWidthVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
+ @Override
+ protected void sliceAndTransferValidityBuffer(
+ int startIndex, int length, BaseValueVector target) {
+ final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
+ final int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- if (length > 0) {
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer.getReferenceManager().retain();
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- this.validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- this.validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
+ if (target.validityBuffer != null) {
+ target.validityBuffer.getReferenceManager().release();
}
+ target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
+ target.validityBuffer.getReferenceManager().retain();
}
/*----------------------------------------------------------------*
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java
index 7bff431e40..37dfa20616 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseValueVector.java
@@ -48,6 +48,10 @@ public abstract class BaseValueVector implements ValueVector {
protected volatile FieldReader fieldReader;
+ protected ArrowBuf validityBuffer;
+
+ protected int valueCount;
+
protected BaseValueVector(BufferAllocator allocator) {
this.allocator = Preconditions.checkNotNull(allocator, "allocator cannot be null");
}
@@ -255,4 +259,116 @@ public void copyFrom(int fromIndex, int thisIndex, ValueVector from) {
public void copyFromSafe(int fromIndex, int thisIndex, ValueVector from) {
throw new UnsupportedOperationException();
}
+
+ /**
+ * Transfer the validity buffer from `validityBuffer` to the target vector's `validityBuffer`.
+ * Start at `startIndex` and copy `length` number of elements. If the starting index is 8 byte
+ * aligned, then the buffer is sliced from that index and ownership is transferred. If not,
+ * individual bytes are copied.
+ *
+ * @param startIndex starting index
+ * @param length number of elements to be copied
+ * @param target target vector
+ */
+ protected void splitAndTransferValidityBuffer(
+ int startIndex, int length, BaseValueVector target) {
+ int offset = startIndex % 8;
+
+ if (length <= 0) {
+ return;
+ }
+ if (offset == 0) {
+ sliceAndTransferValidityBuffer(startIndex, length, target);
+ } else {
+ copyValidityBuffer(startIndex, length, target);
+ }
+ }
+
+ /**
+ * If the start index is 8 byte aligned, slice `validityBuffer` and transfer ownership to
+ * `target`'s `validityBuffer`.
+ *
+ * @param startIndex starting index
+ * @param length number of elements to be copied
+ * @param target target vector
+ */
+ protected void sliceAndTransferValidityBuffer(
+ int startIndex, int length, BaseValueVector target) {
+ final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
+ final int byteSizeTarget = getValidityBufferSizeFromCount(length);
+
+ if (target.validityBuffer != null) {
+ target.validityBuffer.getReferenceManager().release();
+ }
+ target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
+ target.validityBuffer.getReferenceManager().retain(1);
+ }
+
+ /**
+ * Allocate new validity buffer for `target` and copy bytes from `validityBuffer`. Precise details
+ * in the comments below.
+ *
+ * @param startIndex starting index
+ * @param length number of elements to be copied
+ * @param target target vector
+ */
+ protected void copyValidityBuffer(int startIndex, int length, BaseValueVector target) {
+ final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
+ final int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
+ final int byteSizeTarget = getValidityBufferSizeFromCount(length);
+ final int offset = startIndex % 8;
+
+ /* Copy data
+ * When the first bit starts from the middle of a byte (offset != 0),
+ * copy data from src BitVector.
+ * Each byte in the target is composed by a part in i-th byte,
+ * another part in (i+1)-th byte.
+ */
+ target.allocateValidityBuffer(byteSizeTarget);
+
+ for (int i = 0; i < byteSizeTarget - 1; i++) {
+ byte b1 =
+ BitVectorHelper.getBitsFromCurrentByte(this.validityBuffer, firstByteSource + i, offset);
+ byte b2 =
+ BitVectorHelper.getBitsFromNextByte(this.validityBuffer, firstByteSource + i + 1, offset);
+
+ target.validityBuffer.setByte(i, (b1 + b2));
+ }
+
+ /* Copying the last piece is done in the following manner:
+ * if the source vector has 1 or more bytes remaining, we copy
+ * the last piece as a byte formed by shifting data
+ * from the current byte and the next byte.
+ *
+ * if the source vector has no more bytes remaining
+ * (we are at the last byte), we copy the last piece as a byte
+ * by shifting data from the current byte.
+ */
+ if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
+ byte b1 =
+ BitVectorHelper.getBitsFromCurrentByte(
+ this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
+ byte b2 =
+ BitVectorHelper.getBitsFromNextByte(
+ this.validityBuffer, firstByteSource + byteSizeTarget, offset);
+
+ target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
+ } else {
+ byte b1 =
+ BitVectorHelper.getBitsFromCurrentByte(
+ this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
+ target.validityBuffer.setByte(byteSizeTarget - 1, b1);
+ }
+ }
+
+ /**
+ * Allocate new validity buffer for when the bytes need to be copied over.
+ *
+ * @param byteSizeTarget desired size of the buffer
+ */
+ protected void allocateValidityBuffer(long byteSizeTarget) {
+ validityBuffer = allocator.buffer(byteSizeTarget);
+ validityBuffer.readerIndex(0);
+ validityBuffer.setZero(0, validityBuffer.capacity());
+ }
}
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java
index 4f681311ed..96e2afbd29 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthVector.java
@@ -50,10 +50,8 @@ public abstract class BaseVariableWidthVector extends BaseValueVector
/* protected members */
public static final int OFFSET_WIDTH = 4; /* 4 byte unsigned int to track offsets */
protected static final byte[] emptyByteArray = new byte[] {};
- protected ArrowBuf validityBuffer;
protected ArrowBuf valueBuffer;
protected ArrowBuf offsetBuffer;
- protected int valueCount;
protected int lastSet;
protected final Field field;
@@ -87,7 +85,7 @@ public String getName() {
/* TODO:
* Once the entire hierarchy has been refactored, move common functions
- * like getNullCount(), splitAndTransferValidityBuffer to top level
+ * like getNullCount() to top level
* base class BaseValueVector.
*
* Along with this, some class members (validityBuffer) can also be
@@ -519,11 +517,9 @@ private ArrowBuf allocateOffsetBuffer(final long size) {
}
/* allocate validity buffer */
- private void allocateValidityBuffer(final long size) {
- final int curSize = (int) size;
- validityBuffer = allocator.buffer(curSize);
- validityBuffer.readerIndex(0);
- initValidityBuffer();
+ @Override
+ protected void allocateValidityBuffer(final long size) {
+ super.allocateValidityBuffer(size);
}
/**
@@ -856,70 +852,17 @@ private void splitAndTransferOffsetBuffer(
target.valueBuffer = transferBuffer(slicedBuffer, target.allocator);
}
- /*
- * Transfer the validity.
- */
- private void splitAndTransferValidityBuffer(
- int startIndex, int length, BaseVariableWidthVector target) {
- if (length <= 0) {
- return;
- }
-
+ @Override
+ protected void sliceAndTransferValidityBuffer(
+ int startIndex, int length, BaseValueVector target) {
final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- final int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
final int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- final int offset = startIndex % 8;
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- final ArrowBuf slicedValidityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer = transferBuffer(slicedValidityBuffer, target.allocator);
- return;
- }
-
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(this.validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(this.validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- this.validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
+ if (target.validityBuffer != null) {
+ target.validityBuffer.getReferenceManager().release();
}
+ final ArrowBuf slicedValidityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
+ target.validityBuffer = transferBuffer(slicedValidityBuffer, target.allocator);
}
/*----------------------------------------------------------------*
diff --git a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java
index 5e25ffa568..ea9de8320e 100644
--- a/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/BaseVariableWidthViewVector.java
@@ -78,13 +78,11 @@ public abstract class BaseVariableWidthViewVector extends BaseValueVector
// The third 4 bytes of view are allocated for buffer index
public static final int BUF_INDEX_WIDTH = 4;
public static final byte[] EMPTY_BYTE_ARRAY = new byte[] {};
- protected ArrowBuf validityBuffer;
// The view buffer is used to store the variable width view elements
protected ArrowBuf viewBuffer;
// The external buffer which stores the long strings
protected List dataBuffers;
protected int initialDataBufferSize;
- protected int valueCount;
protected int lastSet;
protected final Field field;
@@ -117,7 +115,7 @@ public String getName() {
/* TODO:
* Once the entire hierarchy has been refactored, move common functions
- * like getNullCount(), splitAndTransferValidityBuffer to top level
+ * like getNullCount() to top level
* base class BaseValueVector.
*
* Along with this, some class members (validityBuffer) can also be
@@ -129,12 +127,6 @@ public String getName() {
* the top class as of now is not a good idea.
*/
- /* TODO:
- * Implement TransferPair functionality
- * https://github.com/apache/arrow/issues/40932
- *
- */
-
/**
* Get buffer that manages the validity (NULL or NON-NULL nature) of elements in the vector.
* Consider it as a buffer for internal bit vector data structure.
@@ -854,77 +846,22 @@ public void splitAndTransferTo(int startIndex, int length, BaseVariableWidthView
}
/* allocate validity buffer */
- private void allocateValidityBuffer(final long size) {
- final int curSize = (int) size;
- validityBuffer = allocator.buffer(curSize);
- validityBuffer.readerIndex(0);
- initValidityBuffer();
+ @Override
+ protected void allocateValidityBuffer(final long size) {
+ super.allocateValidityBuffer(size);
}
- /*
- * Transfer the validity.
- */
- private void splitAndTransferValidityBuffer(
- int startIndex, int length, BaseVariableWidthViewVector target) {
- if (length <= 0) {
- return;
- }
-
+ @Override
+ protected void sliceAndTransferValidityBuffer(
+ int startIndex, int length, BaseValueVector target) {
final int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- final int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
final int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- final int offset = startIndex % 8;
-
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- final ArrowBuf slicedValidityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer = transferBuffer(slicedValidityBuffer, target.allocator);
- return;
- }
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(this.validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(this.validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- this.validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- this.validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
+ if (target.validityBuffer != null) {
+ target.validityBuffer.getReferenceManager().release();
}
+ final ArrowBuf slicedValidityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
+ target.validityBuffer = transferBuffer(slicedValidityBuffer, target.allocator);
}
/**
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/BaseLargeRepeatedValueViewVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/BaseLargeRepeatedValueViewVector.java
index 12edd6557b..fac3f86bba 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/BaseLargeRepeatedValueViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/BaseLargeRepeatedValueViewVector.java
@@ -52,7 +52,6 @@ public abstract class BaseLargeRepeatedValueViewVector extends BaseValueVector
protected ArrowBuf sizeBuffer;
protected FieldVector vector;
protected final CallBack repeatedCallBack;
- protected int valueCount;
protected long offsetAllocationSizeInBytes = INITIAL_VALUE_ALLOCATION * OFFSET_WIDTH;
protected long sizeAllocationSizeInBytes = INITIAL_VALUE_ALLOCATION * SIZE_WIDTH;
private final String name;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueVector.java
index fbe83bad52..ee1d65d3e3 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueVector.java
@@ -54,7 +54,6 @@ public abstract class BaseRepeatedValueVector extends BaseValueVector
protected ArrowBuf offsetBuffer;
protected FieldVector vector;
protected final CallBack repeatedCallBack;
- protected int valueCount;
protected long offsetAllocationSizeInBytes = INITIAL_VALUE_ALLOCATION * OFFSET_WIDTH;
private final String name;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueViewVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueViewVector.java
index e6213316b5..fd7a4ff2c6 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/BaseRepeatedValueViewVector.java
@@ -52,7 +52,6 @@ public abstract class BaseRepeatedValueViewVector extends BaseValueVector
protected ArrowBuf sizeBuffer;
protected FieldVector vector;
protected final CallBack repeatedCallBack;
- protected int valueCount;
protected long offsetAllocationSizeInBytes = INITIAL_VALUE_ALLOCATION * OFFSET_WIDTH;
protected long sizeAllocationSizeInBytes = INITIAL_VALUE_ALLOCATION * SIZE_WIDTH;
private final String name;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
index 36d9ff40ed..e3b4ab477f 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
@@ -69,12 +69,10 @@ public static FixedSizeListVector empty(String name, int size, BufferAllocator a
}
private FieldVector vector;
- private ArrowBuf validityBuffer;
private final int listSize;
private Field field;
private UnionFixedSizeListReader reader;
- private int valueCount;
private int validityAllocationSizeInBytes;
/**
@@ -248,12 +246,10 @@ public boolean allocateNewSafe() {
return success;
}
- private void allocateValidityBuffer(final long size) {
- final int curSize = (int) size;
- validityBuffer = allocator.buffer(curSize);
- validityBuffer.readerIndex(0);
- validityAllocationSizeInBytes = curSize;
- validityBuffer.setZero(0, validityBuffer.capacity());
+ @Override
+ protected void allocateValidityBuffer(final long size) {
+ super.allocateValidityBuffer(size);
+ validityAllocationSizeInBytes = (int) size;
}
@Override
@@ -649,71 +645,6 @@ public void splitAndTransfer(int startIndex, int length) {
to.setValueCount(length);
}
- /*
- * transfer the validity.
- */
- private void splitAndTransferValidityBuffer(
- int startIndex, int length, FixedSizeListVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
-
- if (length > 0) {
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer.getReferenceManager().retain(1);
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
-
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
- }
- }
-
@Override
public ValueVector getTo() {
return to;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java
index 71633441cb..835d3468f3 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListVector.java
@@ -94,11 +94,9 @@ public static LargeListVector empty(String name, BufferAllocator allocator) {
protected ArrowBuf offsetBuffer;
protected FieldVector vector;
protected final CallBack callBack;
- protected int valueCount;
protected long offsetAllocationSizeInBytes = INITIAL_VALUE_ALLOCATION * OFFSET_WIDTH;
protected String defaultDataVectorName = DATA_VECTOR_NAME;
- protected ArrowBuf validityBuffer;
protected UnionLargeListReader reader;
private Field field;
private int validityAllocationSizeInBytes;
@@ -375,12 +373,10 @@ public boolean allocateNewSafe() {
return success;
}
- private void allocateValidityBuffer(final long size) {
- final int curSize = (int) size;
- validityBuffer = allocator.buffer(curSize);
- validityBuffer.readerIndex(0);
- validityAllocationSizeInBytes = curSize;
- validityBuffer.setZero(0, validityBuffer.capacity());
+ @Override
+ protected void allocateValidityBuffer(final long size) {
+ super.allocateValidityBuffer(size);
+ validityAllocationSizeInBytes = (int) size;
}
protected ArrowBuf allocateOffsetBuffer(final long size) {
@@ -694,71 +690,6 @@ public void splitAndTransfer(int startIndex, int length) {
to.setValueCount(length);
}
- /*
- * transfer the validity.
- */
- private void splitAndTransferValidityBuffer(
- int startIndex, int length, LargeListVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
-
- if (length > 0) {
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer.getReferenceManager().retain(1);
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
-
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
- }
- }
-
@Override
public ValueVector getTo() {
return to;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java
index 1b7e6b2280..394c3c67bb 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/LargeListViewVector.java
@@ -77,7 +77,6 @@
public class LargeListViewVector extends BaseLargeRepeatedValueViewVector
implements PromotableVector, ValueIterableVector> {
- protected ArrowBuf validityBuffer;
protected UnionLargeListViewReader reader;
private CallBack callBack;
protected Field field;
@@ -285,12 +284,10 @@ public boolean allocateNewSafe() {
return success;
}
+ @Override
protected void allocateValidityBuffer(final long size) {
- final int curSize = (int) size;
- validityBuffer = allocator.buffer(curSize);
- validityBuffer.readerIndex(0);
- validityAllocationSizeInBytes = curSize;
- validityBuffer.setZero(0, validityBuffer.capacity());
+ super.allocateValidityBuffer(size);
+ validityAllocationSizeInBytes = (int) size;
}
@Override
@@ -531,71 +528,6 @@ public void splitAndTransfer(int startIndex, int length) {
}
}
- /*
- * transfer the validity.
- */
- private void splitAndTransferValidityBuffer(
- int startIndex, int length, LargeListViewVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
-
- if (length > 0) {
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer.getReferenceManager().retain(1);
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
-
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
- }
- }
-
@Override
public ValueVector getTo() {
return to;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
index a8e8dcc436..2b2817515f 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
@@ -74,7 +74,6 @@ public static ListVector empty(String name, BufferAllocator allocator) {
return new ListVector(name, allocator, FieldType.nullable(ArrowType.List.INSTANCE), null);
}
- protected ArrowBuf validityBuffer;
protected UnionListReader reader;
private CallBack callBack;
protected Field field;
@@ -324,12 +323,10 @@ public boolean allocateNewSafe() {
return success;
}
+ @Override
protected void allocateValidityBuffer(final long size) {
- final int curSize = (int) size;
- validityBuffer = allocator.buffer(curSize);
- validityBuffer.readerIndex(0);
- validityAllocationSizeInBytes = curSize;
- validityBuffer.setZero(0, validityBuffer.capacity());
+ super.allocateValidityBuffer(size);
+ validityAllocationSizeInBytes = (int) size;
}
/**
@@ -575,70 +572,6 @@ public void splitAndTransfer(int startIndex, int length) {
}
}
- /*
- * transfer the validity.
- */
- private void splitAndTransferValidityBuffer(int startIndex, int length, ListVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
-
- if (length > 0) {
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer.getReferenceManager().retain(1);
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
-
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
- }
- }
-
@Override
public ValueVector getTo() {
return to;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java
index ada25bbaf5..2b80101926 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/ListViewVector.java
@@ -76,7 +76,6 @@
public class ListViewVector extends BaseRepeatedValueViewVector
implements PromotableVector, ValueIterableVector> {
- protected ArrowBuf validityBuffer;
protected UnionListViewReader reader;
private CallBack callBack;
protected Field field;
@@ -284,12 +283,10 @@ public boolean allocateNewSafe() {
return success;
}
+ @Override
protected void allocateValidityBuffer(final long size) {
- final int curSize = (int) size;
- validityBuffer = allocator.buffer(curSize);
- validityBuffer.readerIndex(0);
- validityAllocationSizeInBytes = curSize;
- validityBuffer.setZero(0, validityBuffer.capacity());
+ super.allocateValidityBuffer(size);
+ validityAllocationSizeInBytes = (int) size;
}
@Override
@@ -538,70 +535,6 @@ public void splitAndTransfer(int startIndex, int length) {
}
}
- /*
- * transfer the validity.
- */
- private void splitAndTransferValidityBuffer(int startIndex, int length, ListViewVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
-
- if (length > 0) {
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer.getReferenceManager().retain(1);
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
-
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
- }
- }
-
@Override
public ValueVector getTo() {
return to;
diff --git a/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java b/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
index 5eb857ab94..3f98322ba9 100644
--- a/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
+++ b/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
@@ -22,7 +22,6 @@
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.util.Preconditions;
import org.apache.arrow.vector.AddOrGetResult;
-import org.apache.arrow.vector.BitVectorHelper;
import org.apache.arrow.vector.FieldVector;
import org.apache.arrow.vector.ValueVector;
import org.apache.arrow.vector.ZeroVector;
@@ -232,70 +231,6 @@ public void splitAndTransfer(int startIndex, int length) {
}
}
- /*
- * transfer the validity.
- */
- private void splitAndTransferValidityBuffer(int startIndex, int length, MapVector target) {
- int firstByteSource = BitVectorHelper.byteIndex(startIndex);
- int lastByteSource = BitVectorHelper.byteIndex(valueCount - 1);
- int byteSizeTarget = BitVectorHelper.getValidityBufferSizeFromCount(length);
- int offset = startIndex % 8;
-
- if (length > 0) {
- if (offset == 0) {
- // slice
- if (target.validityBuffer != null) {
- target.validityBuffer.getReferenceManager().release();
- }
- target.validityBuffer = validityBuffer.slice(firstByteSource, byteSizeTarget);
- target.validityBuffer.getReferenceManager().retain(1);
- } else {
- /* Copy data
- * When the first bit starts from the middle of a byte (offset != 0),
- * copy data from src BitVector.
- * Each byte in the target is composed by a part in i-th byte,
- * another part in (i+1)-th byte.
- */
- target.allocateValidityBuffer(byteSizeTarget);
-
- for (int i = 0; i < byteSizeTarget - 1; i++) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(validityBuffer, firstByteSource + i, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + i + 1, offset);
-
- target.validityBuffer.setByte(i, (b1 + b2));
- }
-
- /* Copying the last piece is done in the following manner:
- * if the source vector has 1 or more bytes remaining, we copy
- * the last piece as a byte formed by shifting data
- * from the current byte and the next byte.
- *
- * if the source vector has no more bytes remaining
- * (we are at the last byte), we copy the last piece as a byte
- * by shifting data from the current byte.
- */
- if ((firstByteSource + byteSizeTarget - 1) < lastByteSource) {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- byte b2 =
- BitVectorHelper.getBitsFromNextByte(
- validityBuffer, firstByteSource + byteSizeTarget, offset);
-
- target.validityBuffer.setByte(byteSizeTarget - 1, b1 + b2);
- } else {
- byte b1 =
- BitVectorHelper.getBitsFromCurrentByte(
- validityBuffer, firstByteSource + byteSizeTarget - 1, offset);
- target.validityBuffer.setByte(byteSizeTarget - 1, b1);
- }
- }
- }
- }
-
@Override
public ValueVector getTo() {
return to;
From 43b6b6ccf9107353d22cd2a90632ef2608d872c5 Mon Sep 17 00:00:00 2001
From: wangyunlai
Date: Tue, 1 Jul 2025 09:16:52 +0800
Subject: [PATCH 015/147] GH-759: Get length of byte[] in TryCopyLastError
(#760)
## What's Changed
We should get the length of byte[] by `GetArrayLength`, not `strlen`
which may cause invalid memory access.
Closes #759.
---
c/src/main/cpp/jni_wrapper.cc | 3 +-
.../org/apache/arrow/c/ExceptionTest.java | 150 ++++++++++++++++++
2 files changed, 152 insertions(+), 1 deletion(-)
create mode 100644 c/src/test/java/org/apache/arrow/c/ExceptionTest.java
diff --git a/c/src/main/cpp/jni_wrapper.cc b/c/src/main/cpp/jni_wrapper.cc
index 35c2b7787e..436cbdc806 100644
--- a/c/src/main/cpp/jni_wrapper.cc
+++ b/c/src/main/cpp/jni_wrapper.cc
@@ -205,8 +205,9 @@ void TryCopyLastError(JNIEnv* env, InnerPrivateData* private_data) {
return;
}
+ jsize error_bytes_len = env->GetArrayLength(arr);
char* error_str = reinterpret_cast(error_bytes);
- private_data->last_error_ = std::string(error_str, std::strlen(error_str));
+ private_data->last_error_ = std::string(error_str, error_bytes_len);
env->ReleaseByteArrayElements(arr, error_bytes, JNI_ABORT);
}
diff --git a/c/src/test/java/org/apache/arrow/c/ExceptionTest.java b/c/src/test/java/org/apache/arrow/c/ExceptionTest.java
new file mode 100644
index 0000000000..5bc96a8f99
--- /dev/null
+++ b/c/src/test/java/org/apache/arrow/c/ExceptionTest.java
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.c;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.catchThrowableOfType;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.apache.arrow.memory.BufferAllocator;
+import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.VectorLoader;
+import org.apache.arrow.vector.VectorSchemaRoot;
+import org.apache.arrow.vector.dictionary.Dictionary;
+import org.apache.arrow.vector.dictionary.DictionaryProvider;
+import org.apache.arrow.vector.ipc.ArrowReader;
+import org.apache.arrow.vector.ipc.message.ArrowRecordBatch;
+import org.apache.arrow.vector.types.pojo.ArrowType;
+import org.apache.arrow.vector.types.pojo.Field;
+import org.apache.arrow.vector.types.pojo.Schema;
+import org.junit.jupiter.api.Test;
+
+// Regression test for https://github.com/apache/arrow-java/issues/759
+final class ExceptionTest {
+ @Test
+ public void testException() throws IOException {
+ final Schema schema =
+ new Schema(Collections.singletonList(Field.nullable("ints", new ArrowType.Int(32, true))));
+ final List