This class serializes numbers with a leading byte (ASCII character) indicating the type. - * The class keeps the values byte aligned, even though only 3 bits are strictly necessary to - * encode one of the 6 different primitives with object types that extend Number.
- * - *Classes handled are: Long, Integer, Short,
- * Byte, Double, and Float.
The serialization
- *
- * @author Alexander Saydakov
- */
-public class ArrayOfStringsSerDe extends ArrayOfItemsSerDe This class computes an approximation to the Clopper-Pearson confidence interval
- * for a binomial proportion. Exact Clopper-Pearson intervals are strictly
- * conservative, but these approximations are not. The main inputs are numbers n and k, which are not the same as other things
- * that are called n and k in our sketching library. There is also a third
- * parameter, numStdDev, that specifies the desired confidence level. Alternatively, consider a coin with unknown heads probability p. Where
- * n is the number of independent flips of that coin, and k is the number
- * of times that the coin comes up heads during a given batch of n flips.
- * This class computes a frequentist confidence interval [lowerBoundOnP, upperBoundOnP] for the
- * unknown p. Conceptually, the desired confidence level is specified by a tail probability delta. Ideally, over a large ensemble of independent batches of trials,
- * the fraction of batches in which the true p lies below lowerBoundOnP would be at most
- * delta, and the fraction of batches in which the true p lies above upperBoundOnP
- * would also be at most delta.
- *
- * Setting aside the philosophical difficulties attaching to that statement, it isn't quite
- * true because we are approximating the Clopper-Pearson interval. Finally, we point out that in this class's interface, the confidence parameter delta is
- * not specified directly, but rather through a "number of standard deviations" numStdDev.
- * The library effectively converts that to a delta via delta = normalCDF (-1.0 * numStdDev). It is perhaps worth emphasizing that the library is NOT merely adding and subtracting
- * numStdDev standard deviations to the estimate. It is doing something better, that to some
- * extent accounts for the fact that the binomial distribution has a non-gaussian shape. In particular, it is using an approximation to the inverse of the incomplete beta function
- * that appears as formula 26.5.22 on page 945 of the "Handbook of Mathematical Functions"
- * by Abramowitz and Stegun. Implementation Notes: Implementation Notes: If the AlphaSketch is fed into any ThetaSetOperation, the error distribution reverts back to the
- * normal QuickSelect/KMV error distribution (~1/sqrt(k)). For this reason, the AlphaSketch
- * does not have a sister class for off-heap operation. The AlphaSketch has a roughly 30% faster
- * overall update time as compared to the QuickSelect sketch family. The AlphaSketch is created using the UpdatableThetaSketchBuilder().
- * See Alpha TCF and
- * Theta Sketch Framework
- */
- ALPHA(1, "Alpha", 3, 3),
-
- /**
- * The QuickSelect Sketch family is a member of the Theta Sketch Framework of sketches and
- * is the workhorse of the Theta Sketch Families and can be constructed for either on-heap or
- * off-heap operation.
- * The QuickSelect Sketch is created using the UpdatableThetaSketchBuilder().
- * See Quick Select TCF
- */
- QUICKSELECT(2, "QuickSelect", 3, 3),
-
- /**
- * The Compact Sketch family is a member of the Theta Sketch Framework of sketches.
- * They are read-only and cannot be updated, but can participate in any of the Set Operations.
- * The compact sketches are never created directly with a constructor or builder.
- * Instead they are created as a result of the compact()
- * method of an UpdatableThetaSketch or as a result of a getSketchSamples() of a ThetaSetOperation.
- */
- COMPACT(3, "Compact", 1, 3),
-
- /**
- * The Union family is an operation for the Theta Sketch Framework of sketches.
- * The Union is constructed using the ThetaSetOperationBuilder().
- */
- UNION(4, "Union", 4, 4),
-
- /**
- * The Intersection family is an operation for the Theta Sketch Framework of sketches.
- * The ThetaIntersection is constructed using the ThetaSetOperationBuilder().
- */
- INTERSECTION(5, "Intersection", 3, 3),
-
- /**
- * The A and not B family is an operation for the Theta Sketch Framework of sketches.
- * The ThetaAnotB operation is constructed using the ThetaSetOperationBuilder().
- */
- A_NOT_B(6, "AnotB", 3, 3),
-
- /**
- * The HLL family of sketches. (Not part of TSF.)
- */
- HLL(7, "HLL", 1, 1),
-
- /**
- * The Quantiles family of sketches. (Not part of TSF.)
- */
- QUANTILES(8, "QUANTILES", 1, 2),
-
- /**
- * The Tuple family of sketches is a large family of sketches that are extensions of the
- * Theta Sketch Framework.
- */
- TUPLE(9, "TUPLE", 1, 3),
-
- /**
- * The Frequency family of sketches. (Not part of TSF.)
- */
- FREQUENCY(10, "FREQUENCY", 1, 4),
-
- /**
- * The Reservoir family of sketches. (Not part of TSF.)
- */
- RESERVOIR(11, "RESERVOIR", 1, 2),
-
- /**
- * The reservoir sampling family of Union operations. (Not part of TSF.)
- */
- RESERVOIR_UNION(12, "RESERVOIR_UNION", 1, 1),
-
- /**
- * The VarOpt family of sketches. (Not part of TSF.)
- */
- VAROPT(13, "VAROPT", 1, 4),
-
- /**
- * The VarOpt family of sketches. (Not part of TSF.)
- */
- VAROPT_UNION(14, "VAROPT_UNION", 1, 4),
-
- /**
- * KLL quantiles sketch
- */
- KLL(15, "KLL", 1, 2),
-
- /**
- * Compressed Probabilistic Counting (CPC) Sketch
- */
- CPC(16, "CPC", 1, 5),
-
- /**
- * Relative Error Quantiles Sketch
- */
- REQ(17, "REQ", 1, 2),
-
- /**
- * CountMin Sketch
- */
- COUNTMIN(18, "COUNTMIN", 2, 2),
-
- /**
- * Exact and Bounded, Probability Proportional to Size (EBPPS)
- */
- EBPPS(19, "EBPPS", 1, 5),
-
- /**
- * t-Digest for estimating quantiles and ranks
- */
- TDIGEST(20, "TDigest", 1, 2),
-
- /**
- * Bloom Filter
- */
- BLOOMFILTER(21, "BLOOMFILTER", 3, 4);
-
- private static final Map Note: If both segments are on-heap and not read-only, it can be determined if they were derived from
- * the same backing memory (array). However, this is not always possible off-heap. Because of this asymmetry, this definition
- * of "isSameResource" is confined to the existence of an overlap. Note: If both segments are on-heap and not read-only, it can be determined if they were derived from
- * the same backing memory (array). However, this is not always possible off-heap. Because of this asymmetry, this definition
- * of "isSameResource" is confined to the existence of an overlap. Note that the detail message associated with cause is not automatically incorporated
- * in this runtime exception's detail message. The detail message is saved for later retrieval by the Throwable.getMessage() method. The cause is not initialized, and may subsequently be initialized by a call to
- * Throwable.initCause(java.lang.Throwable).
- */
- public SketchesReadOnlyException() {
- super("Write operation attempted on a read-only class.");
- }
-
- /**
- * Constructs a new runtime exception with the specified detail message. The cause is not
- * initialized, and may subsequently be initialized by a call to
- * Throwable.initCause(java.lang.Throwable).
- *
- * @param message the detail message. The detail message is saved for later retrieval by the
- * Throwable.getMessage() method.
- */
- public SketchesReadOnlyException(final String message) {
- super(message);
- }
-}
diff --git a/src/main/java/org/apache/datasketches/common/SketchesStateException.java b/src/main/java/org/apache/datasketches/common/SketchesStateException.java
deleted file mode 100644
index b3d3df90e..000000000
--- a/src/main/java/org/apache/datasketches/common/SketchesStateException.java
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-/**
- * Illegal State Exception class for the library
- *
- * @author Lee Rhodes
- */
-public class SketchesStateException extends SketchesException {
- private static final long serialVersionUID = 1L;
-
- //other constructors to be added as needed.
-
- /**
- * Constructs a new runtime exception with the specified detail message. The cause is not
- * initialized, and may subsequently be initialized by a call to
- * Throwable.initCause(java.lang.Throwable).
- *
- * @param message the detail message. The detail message is saved for later retrieval by the
- * Throwable.getMessage() method.
- */
- public SketchesStateException(final String message) {
- super(message);
- }
-}
diff --git a/src/main/java/org/apache/datasketches/common/SpecialValueLayouts.java b/src/main/java/org/apache/datasketches/common/SpecialValueLayouts.java
deleted file mode 100644
index 292d5617f..000000000
--- a/src/main/java/org/apache/datasketches/common/SpecialValueLayouts.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-import java.lang.foreign.ValueLayout;
-import java.nio.ByteOrder;
-
-/**
- * Value Layouts for Non-native Endianness
- */
-public final class SpecialValueLayouts {
-
- private SpecialValueLayouts() { }
-
- /**
- * The static final for NON ByteOrder.nativeOrder().
- */
- public static final ByteOrder NON_NATIVE_BYTE_ORDER =
- (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN) ? ByteOrder.BIG_ENDIAN : ByteOrder.LITTLE_ENDIAN;
-
- //Non-Native Endian Layouts
-
- /**
- * The static final for NON ByteOrder.nativeOrder() char.
- */
- public static final ValueLayout.OfChar JAVA_CHAR_UNALIGNED_NON_NATIVE =
- ValueLayout.JAVA_CHAR_UNALIGNED.withOrder(NON_NATIVE_BYTE_ORDER);
-
- /**
- * The static final for NON ByteOrder.nativeOrder() double.
- */
- public static final ValueLayout.OfDouble JAVA_DOUBLE_UNALIGNED_NON_NATIVE =
- ValueLayout.JAVA_DOUBLE_UNALIGNED.withOrder(NON_NATIVE_BYTE_ORDER);
-
- /**
- * The static final for NON ByteOrder.nativeOrder() float.
- */
- public static final ValueLayout.OfFloat JAVA_FLOAT_UNALIGNED_NON_NATIVE =
- ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(NON_NATIVE_BYTE_ORDER);
-
- /**
- * The static final for NON ByteOrder.nativeOrder() int.
- */
- public static final ValueLayout.OfInt JAVA_INT_UNALIGNED_NON_NATIVE =
- ValueLayout.JAVA_INT_UNALIGNED.withOrder(NON_NATIVE_BYTE_ORDER);
-
- /**
- * The static final for NON ByteOrder.nativeOrder() long.
- */
- public static final ValueLayout.OfLong JAVA_LONG_UNALIGNED_NON_NATIVE =
- ValueLayout.JAVA_LONG_UNALIGNED.withOrder(NON_NATIVE_BYTE_ORDER);
-
- /**
- * The static final for NON ByteOrder.nativeOrder() short.
- */
- public static final ValueLayout.OfShort JAVA_SHORT_UNALIGNED_NON_NATIVE =
- ValueLayout.JAVA_SHORT_UNALIGNED.withOrder(NON_NATIVE_BYTE_ORDER);
-
- //Big-Endian Layouts
-
- /**
- * The static final for ByteOrder.BIG_ENDIAN char.
- */
- public static final ValueLayout.OfChar JAVA_CHAR_UNALIGNED_BIG_ENDIAN =
- ValueLayout.JAVA_CHAR_UNALIGNED.withOrder(ByteOrder.BIG_ENDIAN);
-
- /**
- * The static final for ByteOrder.BIG_ENDIAN double.
- */
- public static final ValueLayout.OfDouble JAVA_DOUBLE_UNALIGNED_BIG_ENDIAN =
- ValueLayout.JAVA_DOUBLE_UNALIGNED.withOrder(ByteOrder.BIG_ENDIAN);
-
- /**
- * The static final for ByteOrder.BIG_ENDIAN float.
- */
- public static final ValueLayout.OfFloat JAVA_FLOAT_UNALIGNED_BIG_ENDIAN =
- ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(ByteOrder.BIG_ENDIAN);
-
- /**
- * The static final for ByteOrder.BIG_ENDIAN int.
- */
- public static final ValueLayout.OfInt JAVA_INT_UNALIGNED_BIG_ENDIAN =
- ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.BIG_ENDIAN);
-
- /**
- * The static final for ByteOrder.BIG_ENDIAN long.
- */
- public static final ValueLayout.OfLong JAVA_LONG_UNALIGNED_BIG_ENDIAN =
- ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.BIG_ENDIAN);
-
- /**
- * The static final for ByteOrder.BIG_ENDIAN short.
- */
- public static final ValueLayout.OfShort JAVA_SHORT_UNALIGNED_BIG_ENDIAN =
- ValueLayout.JAVA_SHORT_UNALIGNED.withOrder(ByteOrder.BIG_ENDIAN);
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/SuppressFBWarnings.java b/src/main/java/org/apache/datasketches/common/SuppressFBWarnings.java
deleted file mode 100644
index c80421182..000000000
--- a/src/main/java/org/apache/datasketches/common/SuppressFBWarnings.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-import java.lang.annotation.Retention;
-import java.lang.annotation.RetentionPolicy;
-
-/**
- * Used to suppress SpotBug warnings.
- *
- * @author Lee Rhodes
- */
-@Retention(RetentionPolicy.CLASS)
-public @interface SuppressFBWarnings {
-
- /**
- * A list of comma-separated, quoted SpotBugs warnings that are to be suppressed in the associated
- * annotated element. The value can be a bug category, kind or pattern.
- * @return list of relevant bug descriptors
- */
- String[] value() default {};
-
- /**
- * Optional explanation for the suppression.
- * @return explanation
- */
- String justification() default "";
-}
-
diff --git a/src/main/java/org/apache/datasketches/common/Util.java b/src/main/java/org/apache/datasketches/common/Util.java
deleted file mode 100644
index 7d70323e1..000000000
--- a/src/main/java/org/apache/datasketches/common/Util.java
+++ /dev/null
@@ -1,993 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-import static java.lang.Math.ceil;
-import static java.lang.Math.floor;
-import static java.lang.Math.log;
-import static java.lang.Math.pow;
-import static java.lang.Math.round;
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.hash.MurmurHash3.hash;
-
-import java.lang.foreign.MemorySegment;
-import java.nio.ByteOrder;
-import java.util.Comparator;
-
-/**
- * Common utility functions.
- *
- * @author Lee Rhodes
- */
-@SuppressWarnings("unchecked")
-public final class Util {
-
- static {
- if (ByteOrder.nativeOrder() != ByteOrder.LITTLE_ENDIAN) {
- throw new SketchesNotSupportedException("Machine Native Endianness must be LITTLE_ENDIAN.");
- }
- }
-
- /**
- * The java line separator character as a String.
- */
- public static final String LS = System.getProperty("line.separator");
-
- /**
- * The tab character
- */
- public static final char TAB = '\t';
-
- /**
- * The natural logarithm of 2.0.
- */
- public static final double LOG2 = log(2.0);
-
- /**
- * The inverse golden ratio as an unsigned long.
- */
- public static final long INVERSE_GOLDEN_U64 = 0x9e3779b97f4a7c13L;
-
- /**
- * The inverse golden ratio as a fraction.
- * This has more precision than using the formula: (Math.sqrt(5.0) - 1.0) / 2.0.
- */
- public static final double INVERSE_GOLDEN = 0.6180339887498949025;
-
- /**
- * Long.MAX_VALUE as a double.
- */
- public static final double LONG_MAX_VALUE_AS_DOUBLE = Long.MAX_VALUE;
-
- /**
- * The seed 9001 used in the sketch update methods is a prime number that
- * was chosen very early on in experimental testing. Choosing a seed is somewhat arbitrary, and
- * the author cannot prove that this particular seed is somehow superior to other seeds. There
- * was some early Internet discussion that a seed of 0 did not produce as clean avalanche diagrams
- * as non-zero seeds, but this may have been more related to the MurmurHash2 release, which did
- * have some issues. As far as the author can determine, MurmurHash3 does not have these problems.
- *
- * In order to perform set operations on two sketches it is critical that the same hash
- * function and seed are identical for both sketches, otherwise the assumed 1:1 relationship
- * between the original source key value and the hashed bit string would be violated. Once
- * you have developed a history of stored sketches you are stuck with it.
- *
- * WARNING: This seed is used internally by library sketches in different
- * packages and thus must be declared public. However, this seed value must not be used by library
- * users with the MurmurHash3 function. It should be viewed as existing for exclusive, private
- * use by the library.
- *
- * See Default Update Seed
- */
- public static final long DEFAULT_UPDATE_SEED = 9001L;
-
- private Util() {}
-
- //Byte Conversions
-
- /**
- * Returns an int extracted from a Little-Endian byte array.
- * @param arr the given byte array
- * @return an int extracted from a Little-Endian byte array.
- */
- public static int bytesToInt(final byte[] arr) {
- return (arr[3] << 24)
- | ((arr[2] & 0xff) << 16)
- | ((arr[1] & 0xff) << 8)
- | (arr[0] & 0xff);
- }
-
- /**
- * Returns a long extracted from a Little-Endian byte array.
- * @param arr the given byte array
- * @return a long extracted from a Little-Endian byte array.
- */
- public static long bytesToLong(final byte[] arr) {
- return ((long)arr[7] << 56)
- | (((long)arr[6] & 0xff) << 48)
- | (((long)arr[5] & 0xff) << 40)
- | (((long)arr[4] & 0xff) << 32)
- | (((long)arr[3] & 0xff) << 24)
- | (((long)arr[2] & 0xff) << 16)
- | (((long)arr[1] & 0xff) << 8)
- | ((long)arr[0] & 0xff);
- }
-
- /**
- * Returns a Little-Endian byte array extracted from the given int.
- * @param v the given int
- * @param arr a given array of 4 bytes that will be returned with the data
- * @return a Little-Endian byte array extracted from the given int.
- */
- public static byte[] intToBytes(final int v, final byte[] arr) {
- arr[3] = (byte) (v >>> 24);
- arr[2] = (byte) (v >>> 16);
- arr[1] = (byte) (v >>> 8);
- arr[0] = (byte) v;
- return arr;
- }
-
- /**
- * Returns a Little-Endian byte array extracted from the given long.
- * @param v the given long
- * @param arr a given array of 8 bytes that will be returned with the data
- * @return a Little-Endian byte array extracted from the given long.
- */
- public static byte[] longToBytes(final long v, final byte[] arr) {
- arr[7] = (byte) (v >>> 56);
- arr[6] = (byte) (v >>> 48);
- arr[5] = (byte) (v >>> 40);
- arr[4] = (byte) (v >>> 32);
- arr[3] = (byte) (v >>> 24);
- arr[2] = (byte) (v >>> 16);
- arr[1] = (byte) (v >>> 8);
- arr[0] = (byte) v;
- return arr;
- }
-
- //Byte array conversions
-
- static long[] convertToLongArray(final byte[] byteArr, final boolean littleEndian) {
- final int len = byteArr.length;
- final long[] longArr = new long[(len / 8) + ((len % 8) != 0 ? 1 : 0)];
- int off = 0;
- int longArrIdx = 0;
- while (off < len) {
- final int rem = Math.min(len - 1 - off, 7);
- long tgt = 0;
- if (littleEndian) {
- for (int j = off + rem, k = 0; j >= off; --j, k++) {
- tgt |= (byteArr[j] & 0XFFL) << (k * 8);
- }
- } else { //BE
- for (int j = off + rem, k = rem; j >= off; --j, k--) {
- tgt |= (byteArr[j] & 0XFFL) << (k * 8);
- }
- }
- off += 8;
- longArr[longArrIdx++] = tgt;
- }
- return longArr;
- }
-
- //String Related
-
- /**
- * Returns a string of spaced hex bytes in Big-Endian order.
- * @param v the given long
- * @return string of spaced hex bytes in Big-Endian order.
- */
- public static String longToHexBytes(final long v) {
- final long mask = 0XFFL;
- final StringBuilder sb = new StringBuilder();
- for (int i = 8; i-- > 0; ) {
- final String s = Long.toHexString((v >>> (i * 8)) & mask);
- sb.append(zeroPad(s, 2)).append(" ");
- }
- return sb.toString();
- }
-
- /**
- * Returns a string view of a byte array
- * @param arr the given byte array
- * @param signed set true if you want the byte values signed.
- * @param littleEndian set true if you want Little-Endian order
- * @param sep the separator string between bytes
- * @return a string view of a byte array
- */
- public static String bytesToString(
- final byte[] arr, final boolean signed, final boolean littleEndian, final String sep) {
- final StringBuilder sb = new StringBuilder();
- final int mask = signed ? 0XFFFFFFFF : 0XFF;
- final int arrLen = arr.length;
- if (littleEndian) {
- for (int i = 0; i < (arrLen - 1); i++) {
- sb.append(arr[i] & mask).append(sep);
- }
- sb.append(arr[arrLen - 1] & mask);
- } else {
- for (int i = arrLen; i-- > 1; ) {
- sb.append(arr[i] & mask).append(sep);
- }
- sb.append(arr[0] & mask);
- }
- return sb.toString();
- }
-
- /**
- * Returns the given time in nanoseconds formatted as Sec.mSec_uSec_nSec
- * @param nS the given nanoseconds
- * @return the given time in nanoseconds formatted as Sec.mSec_uSec_nSec
- */
- public static String nanoSecToString(final long nS) {
- final long rem_nS = (long)(nS % 1000.0);
- final long rem_uS = (long)((nS / 1000.0) % 1000.0);
- final long rem_mS = (long)((nS / 1000000.0) % 1000.0);
- final long sec = (long)(nS / 1000000000.0);
- final String nSstr = zeroPad(Long.toString(rem_nS), 3);
- final String uSstr = zeroPad(Long.toString(rem_uS), 3);
- final String mSstr = zeroPad(Long.toString(rem_mS), 3);
- return String.format("%d.%3s_%3s_%3s", sec, mSstr, uSstr, nSstr);
- }
-
- /**
- * Returns the given time in milliseconds formatted as Hours:Min:Sec.mSec
- * @param mS the given milliseconds
- * @return the given time in milliseconds formatted as Hours:Min:Sec.mSec
- */
- public static String milliSecToString(final long mS) {
- final long rem_mS = (long)(mS % 1000.0);
- final long rem_sec = (long)((mS / 1000.0) % 60.0);
- final long rem_min = (long)((mS / 60000.0) % 60.0);
- final long hr = (long)(mS / 3600000.0);
- final String mSstr = zeroPad(Long.toString(rem_mS), 3);
- final String secStr = zeroPad(Long.toString(rem_sec), 2);
- final String minStr = zeroPad(Long.toString(rem_min), 2);
- return String.format("%d:%2s:%2s.%3s", hr, minStr, secStr, mSstr);
- }
-
- /**
- * Prepend the given string with zeros. If the given string is equal or greater than the given
- * field length, it will be returned without modification.
- * @param s the given string
- * @param fieldLength desired total field length including the given string
- * @return the given string prepended with zeros.
- */
- public static String zeroPad(final String s, final int fieldLength) {
- return characterPad(s, fieldLength, '0', false);
- }
-
- /**
- * Prepend or postpend the given string with the given character to fill the given field length.
- * If the given string is equal to or greater than the given field length, it will be returned
- * without modification.
- * @param s the given string
- * @param fieldLength the desired field length
- * @param padChar the desired pad character
- * @param postpend if true append the pacCharacters to the end of the string.
- * @return prepended or postpended given string with the given character to fill the given field length.
- */
- public static String characterPad(final String s, final int fieldLength, final char padChar, final boolean postpend) {
- final int sLen = s.length();
- if (sLen < fieldLength) {
- final char[] cArr = new char[fieldLength - sLen];
- java.util.Arrays.fill(cArr, padChar);
- final String addstr = String.valueOf(cArr);
- return (postpend) ? s.concat(addstr) : addstr.concat(s);
- }
- return s;
- }
-
- //Memory byte alignment
-
- /**
- * Checks if parameter v is a multiple of 8 and greater than zero.
- * @param v The parameter to check
- * @param argName This name will be part of the error message if the check fails.
- */
- public static void checkIfMultipleOf8AndGT0(final long v, final String argName) {
- if (((v & 0X7L) == 0L) && (v > 0L)) {
- return;
- }
- throw new SketchesArgumentException("The value of the parameter \"" + argName
- + "\" must be a positive multiple of 8 and greater than zero: " + v);
- }
-
- /**
- * Returns true if v is a multiple of 8 and greater than zero
- * @param v The parameter to check
- * @return true if v is a multiple of 8 and greater than zero
- */
- public static boolean isMultipleOf8AndGT0(final long v) {
- return ((v & 0X7L) == 0L) && (v > 0L);
- }
-
- //Powers of 2 or powers of base related
-
- /**
- * Returns true if given long argument is exactly a positive power of 2.
- *
- * @param n The input argument.
- * @return true if argument is exactly a positive power of 2.
- */
- public static boolean isPowerOf2(final long n) {
- return (n > 0) && ((n & (n - 1L)) == 0); //or (n > 0) && ((n & -n) == n)
- }
-
- /**
- * Checks the given long argument if it is a positive integer power of 2.
- * If not, it throws an exception with the user supplied local argument name, if not null.
- * @param n The input long argument must be a positive integer power of 2.
- * @param argName Used in the thrown exception. It may be null.
- * @throws SketchesArgumentException if not a positive integer power of 2.
- */
- public static void checkIfPowerOf2(final long n, String argName) {
- if (isPowerOf2(n)) { return; }
- argName = (argName == null) ? "" : argName;
- throw new SketchesArgumentException("The value of the argument \"" + argName + "\""
- + " must be a positive integer power of 2: " + n);
- }
-
- /**
- * Computes the int ceiling power of 2 within the range [1, 2^30]. This is the smallest positive power
- * of 2 that is equal to or greater than the given n and a positive integer.
- *
- * For:
- * For:
- * For:
- * For:
- * The formula is: baseceiling(logbase(x)) The formula is: basefloor(logbase(x)) For example, assume a sequence of integers from 1 to 1000. The largest value has
- * four decimal digits. Convert the entire sequence of strings to the form " 1" to "1000".
- * When these strings are sorted they will be in numerical sequence: " 1", " 2", ... "1000". If aligned is true, the returned MemorySegment will be constructed from a long[] array,
- * and, as a result, it will have a memory alignment of 8 bytes.
- * If the requested capacity is not exactly divisible by eight, the returned size
- * will be rolled up to the next multiple of eight bytes. If aligned is false, the returned MemorySegment will be constructed from a byte[] array,
- * and have a memory alignment of 1 byte.
- *
- * @param capacityBytes The new capacity being requested. It must not be negative and cannot exceed Integer.MAX_VALUE.
- * @param aligned if true, the new heap segment will have an alignment of 8 bytes, otherwise the alignment will be 1 byte.
- * @return a new MemorySegment with the requested capacity and alignment.
- */
- public static MemorySegment alignedHeapSegment(final int capacityBytes, final boolean aligned) {
- if (aligned) {
- final int lenLongs = capacityBytes >>> 3;
- final long[] array = ((capacityBytes & 0x7) == 0)
- ? new long[lenLongs]
- : new long[lenLongs + 1];
- return MemorySegment.ofArray(array);
- }
- return MemorySegment.ofArray(new byte[capacityBytes]);
- }
-
- /**
- * Sets the bits defined by the bitMask
- * @param seg the given MemorySegment
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param bitMask the bits set to one will be set
- */
- public static void setBits(final MemorySegment seg, final long offsetBytes, final byte bitMask) {
- final byte b = seg.get(JAVA_BYTE, offsetBytes);
- seg.set(JAVA_BYTE, offsetBytes, (byte)(b | bitMask));
- }
-
- /**
- * Computes and checks the 16-bit seed hash from the given long seed.
- * The seed hash may not be zero in order to maintain compatibility with older serialized
- * versions that did not have this concept.
- * @param seed See Update Hash Seed
- * @return the seed hash.
- */
- public static short computeSeedHash(final long seed) {
- final long[] seedArr = {seed};
- final short seedHash = (short)(hash(seedArr, 0L)[0] & 0xFFFFL);
- if (seedHash == 0) {
- throw new SketchesArgumentException(
- "The given seed: " + seed + " produced a seedHash of zero. "
- + "You must choose a different seed.");
- }
- return seedHash;
- }
-
- /**
- * Check if the two seed hashes are equal. If not, throw an SketchesArgumentException.
- * @param seedHashA the seedHash A
- * @param seedHashB the seedHash B
- * @return seedHashA if they are equal
- */
- public static short checkSeedHashes(final short seedHashA, final short seedHashB) {
- if (seedHashA != seedHashB) {
- throw new SketchesArgumentException(
- "Incompatible Seed Hashes. " + Integer.toHexString(seedHashA & 0XFFFF)
- + ", " + Integer.toHexString(seedHashB & 0XFFFF));
- }
- return seedHashA;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/package-info.java b/src/main/java/org/apache/datasketches/common/package-info.java
deleted file mode 100644
index 0fbaab76d..000000000
--- a/src/main/java/org/apache/datasketches/common/package-info.java
+++ /dev/null
@@ -1,23 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * This package is for common classes that may be used across all the sketch families.
- */
-package org.apache.datasketches.common;
diff --git a/src/main/java/org/apache/datasketches/common/positional/PositionInvariantsException.java b/src/main/java/org/apache/datasketches/common/positional/PositionInvariantsException.java
deleted file mode 100644
index ef327a564..000000000
--- a/src/main/java/org/apache/datasketches/common/positional/PositionInvariantsException.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common.positional;
-
-/**
- * Position operation violation.
- *
- * @author Lee Rhodes
- */
-public class PositionInvariantsException extends RuntimeException {
- private static final long serialVersionUID = 1L;
-
- /**
- * The associated position operation violated one of the positional invariants.
- *
- * The invariants equation is {@code 0 <= start <= position <= end <= capacity}. A Count-Min sketch is a probabilistic data structure that estimates the frequency of items in a large data stream using a
- * fixed amount of memory. It uses a 2D array of counters and multiple hash functions to map each item to a position in every row.
- * To estimate an item's frequency, it takes the minimum count from all the positions it hashes to, which provides an overestimate
- * that is guaranteed to be greater than or equal to the true count. A Count sketch is a type of dimensionality reduction that is particularly efficient in statistics, machine learning and
- * algorithms. It was invented by Moses Charikar, Kevin Chen and Martin Farach-Colton in an effort to speed up the AMS Sketch by
- * Alon, Matias and Szegedy for approximating the frequency moments of streams[4] (these calculations require counting of the
- * number of occurrences for the distinct elements of the stream). ~/ocaml-4.03.0/bin/ocamlopt -o generateHuffmanCodes columnProbabilities.ml generateHuffmanCodes.ml
- *
- * ./generateHuffmanCodes > raw-encoding-tables.c
- *
- * Some manual cutting and pasting was then done to transfer the contents
- * of that file into this one.
- *
- * Only the encoding tables are defined by this file. The decoding tables
- * (which are exact inverses) are created at library startup time by the function
- * makeDecodingTable (), which is defined in fm85Compression.c.
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class CompressionData {
-
- private static byte[] makeInversePermutation(final byte[]encodePermu) {
- final int length = encodePermu.length;
- final byte[] inverse = new byte[length];
-
- for (int i = 0; i < length; i++) {
- inverse[encodePermu[i]] = (byte) i;
- }
- for (int i = 0; i < length; i++) {
- assert ((encodePermu[inverse[i]] & 0XFF) == i);
- }
- return inverse;
- }
-
- /**
- * Given an encoding table that maps unsigned bytes to codewords
- * of length at most 12, this builds a size-4096 decoding table.
- *
- * The second argument is typically 256, but can be other values such as 65.
- * @param encodingTable unsigned
- * @param numByteValues size of encoding table
- * @return one segment of the decoding table
- */
- private static short[] makeDecodingTable(final short[] encodingTable, final int numByteValues) {
- final short[] decodingTable = new short[4096];
-
- for (int byteValue = 0; byteValue < numByteValues; byteValue++) {
- final int encodingEntry = encodingTable[byteValue] & 0xFFFF;
- final int codeValue = encodingEntry & 0xfff;
- final int codeLength = encodingEntry >> 12;
- final int decodingEntry = (codeLength << 8) | byteValue;
- final int garbageLength = 12 - codeLength;
- final int numCopies = 1 << garbageLength;
-
- for (int garbageBits = 0; garbageBits < numCopies; garbageBits++) {
- final int extendedCodeValue = codeValue | (garbageBits << codeLength);
- decodingTable[extendedCodeValue & 0xfff] = (short) decodingEntry;
- }
- }
- return (decodingTable);
- }
-
- /**
- * These short arrays are being treated as unsigned
- * @param decodingTable unsigned
- * @param encodingTable unsigned
- */
- static void validateDecodingTable(final short[] decodingTable, final short[] encodingTable) {
- for (int decodeThis = 0; decodeThis < 4096; decodeThis++) {
- final int tmpD = decodingTable[decodeThis] & 0xFFFF;
- final int decodedByte = tmpD & 0xff;
- final int decodedLength = tmpD >> 8;
-
- final int tmpE = encodingTable[decodedByte] & 0xFFFF;
- final int encodedBitpattern = tmpE & 0xfff;
- final int encodedLength = tmpE >> 12;
-
- // encodedBitpattern++; // uncomment this line to force failure when testing this method
- // encodedLength++; // uncomment this line to force failure when testing this method
-
- assert (decodedLength == encodedLength)
- : "deLen: " + decodedLength + ", enLen: " + encodedLength;
- assert (encodedBitpattern == (decodeThis & ((1 << decodedLength) - 1)));
- }
- }
-
- private static void makeTheDecodingTables() {
- lengthLimitedUnaryDecodingTable65 = makeDecodingTable(lengthLimitedUnaryEncodingTable65, 65);
- validateDecodingTable(lengthLimitedUnaryDecodingTable65, lengthLimitedUnaryEncodingTable65);
-
- for (int i = 0; i < (16 + 6); i++) {
- decodingTablesForHighEntropyByte[i] = makeDecodingTable(encodingTablesForHighEntropyByte[i], 256);
- validateDecodingTable(decodingTablesForHighEntropyByte[i], encodingTablesForHighEntropyByte[i]);
- }
-
- for (int i = 0; i < 16; i++) {
- columnPermutationsForDecoding[i] = makeInversePermutation(columnPermutationsForEncoding[i]);
- }
- }
-
- /**
- * These decoding tables are created at library startup time by inverting the encoding tables.
- * Sixteen tables for the steady state (chosen based on the "phase" of C/K).
- * Six more tables for the gradual transition between warmup mode and the steady state.
- */
- static short[][] decodingTablesForHighEntropyByte = new short[22][];
-
- /**
- * Sixteen Encoding Tables for the Steady State.
- */
- static short[][] encodingTablesForHighEntropyByte = new short[][] //[22][256]
- {
- // (table 0 of 22) (steady 0 of 16) (phase = 0.031250000 = 1.0 / 32.0)
- // entropy: 4.4619200780464778333
- // avg_length: 4.5415773046232610355; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x9017, // ( 9, 23) 0
- (short) 0x5009, // ( 5, 9) 1
- (short) 0x7033, // ( 7, 51) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0x9117, // ( 9, 279) 4
- (short) 0x5019, // ( 5, 25) 5
- (short) 0x7073, // ( 7, 115) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xa177, // (10, 375) 8
- (short) 0x601d, // ( 6, 29) 9
- (short) 0x803b, // ( 8, 59) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xa377, // (10, 887) 12
- (short) 0x5005, // ( 5, 5) 13
- (short) 0x80bb, // ( 8, 187) 14
- (short) 0x3006, // ( 3, 6) 15
- (short) 0xb0cf, // (11, 207) 16
- (short) 0x700b, // ( 7, 11) 17
- (short) 0xa0f7, // (10, 247) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xb4cf, // (11, 1231) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0x9097, // ( 9, 151) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xc4af, // (12, 1199) 24
- (short) 0x807b, // ( 8, 123) 25
- (short) 0xa2f7, // (10, 759) 26
- (short) 0x603d, // ( 6, 61) 27
- (short) 0xccaf, // (12, 3247) 28
- (short) 0x80fb, // ( 8, 251) 29
- (short) 0xa1f7, // (10, 503) 30
- (short) 0x6003, // ( 6, 3) 31
- (short) 0xc2af, // (12, 687) 32
- (short) 0x8007, // ( 8, 7) 33
- (short) 0xb2cf, // (11, 719) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xcaaf, // (12, 2735) 36
- (short) 0x8087, // ( 8, 135) 37
- (short) 0xa3f7, // (10, 1015) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xc6af, // (12, 1711) 40
- (short) 0x9197, // ( 9, 407) 41
- (short) 0xceaf, // (12, 3759) 42
- (short) 0x702b, // ( 7, 43) 43
- (short) 0xc1af, // (12, 431) 44
- (short) 0x9057, // ( 9, 87) 45
- (short) 0xb6cf, // (11, 1743) 46
- (short) 0x706b, // ( 7, 107) 47
- (short) 0xc9af, // (12, 2479) 48
- (short) 0xa00f, // (10, 15) 49
- (short) 0xc5af, // (12, 1455) 50
- (short) 0x8047, // ( 8, 71) 51
- (short) 0xcdaf, // (12, 3503) 52
- (short) 0xa20f, // (10, 527) 53
- (short) 0xc3af, // (12, 943) 54
- (short) 0x80c7, // ( 8, 199) 55
- (short) 0xcbaf, // (12, 2991) 56
- (short) 0xb1cf, // (11, 463) 57
- (short) 0xc7af, // (12, 1967) 58
- (short) 0x9157, // ( 9, 343) 59
- (short) 0xcfaf, // (12, 4015) 60
- (short) 0xb5cf, // (11, 1487) 61
- (short) 0xc06f, // (12, 111) 62
- (short) 0x90d7, // ( 9, 215) 63
- (short) 0xc86f, // (12, 2159) 64
- (short) 0x91d7, // ( 9, 471) 65
- (short) 0xc46f, // (12, 1135) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xcc6f, // (12, 3183) 68
- (short) 0x9037, // ( 9, 55) 69
- (short) 0xb3cf, // (11, 975) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc26f, // (12, 623) 72
- (short) 0xa10f, // (10, 271) 73
- (short) 0xca6f, // (12, 2671) 74
- (short) 0x8027, // ( 8, 39) 75
- (short) 0xc66f, // (12, 1647) 76
- (short) 0xa30f, // (10, 783) 77
- (short) 0xce6f, // (12, 3695) 78
- (short) 0x80a7, // ( 8, 167) 79
- (short) 0xc16f, // (12, 367) 80
- (short) 0xb7cf, // (11, 1999) 81
- (short) 0xc96f, // (12, 2415) 82
- (short) 0x9137, // ( 9, 311) 83
- (short) 0xc56f, // (12, 1391) 84
- (short) 0xb02f, // (11, 47) 85
- (short) 0xcd6f, // (12, 3439) 86
- (short) 0x90b7, // ( 9, 183) 87
- (short) 0xc36f, // (12, 879) 88
- (short) 0xcb6f, // (12, 2927) 89
- (short) 0xc76f, // (12, 1903) 90
- (short) 0xa08f, // (10, 143) 91
- (short) 0xcf6f, // (12, 3951) 92
- (short) 0xc0ef, // (12, 239) 93
- (short) 0xc8ef, // (12, 2287) 94
- (short) 0xa28f, // (10, 655) 95
- (short) 0xc4ef, // (12, 1263) 96
- (short) 0xccef, // (12, 3311) 97
- (short) 0xc2ef, // (12, 751) 98
- (short) 0xa18f, // (10, 399) 99
- (short) 0xcaef, // (12, 2799) 100
- (short) 0xc6ef, // (12, 1775) 101
- (short) 0xceef, // (12, 3823) 102
- (short) 0xa38f, // (10, 911) 103
- (short) 0xc1ef, // (12, 495) 104
- (short) 0xc9ef, // (12, 2543) 105
- (short) 0xc5ef, // (12, 1519) 106
- (short) 0xb42f, // (11, 1071) 107
- (short) 0xcdef, // (12, 3567) 108
- (short) 0xc3ef, // (12, 1007) 109
- (short) 0xcbef, // (12, 3055) 110
- (short) 0xb22f, // (11, 559) 111
- (short) 0xc7ef, // (12, 2031) 112
- (short) 0xcfef, // (12, 4079) 113
- (short) 0xc01f, // (12, 31) 114
- (short) 0xc81f, // (12, 2079) 115
- (short) 0xc41f, // (12, 1055) 116
- (short) 0xcc1f, // (12, 3103) 117
- (short) 0xc21f, // (12, 543) 118
- (short) 0xca1f, // (12, 2591) 119
- (short) 0xc61f, // (12, 1567) 120
- (short) 0xce1f, // (12, 3615) 121
- (short) 0xc11f, // (12, 287) 122
- (short) 0xc91f, // (12, 2335) 123
- (short) 0xc51f, // (12, 1311) 124
- (short) 0xcd1f, // (12, 3359) 125
- (short) 0xc31f, // (12, 799) 126
- (short) 0xcb1f, // (12, 2847) 127
- (short) 0xc71f, // (12, 1823) 128
- (short) 0xa04f, // (10, 79) 129
- (short) 0xcf1f, // (12, 3871) 130
- (short) 0x8067, // ( 8, 103) 131
- (short) 0xc09f, // (12, 159) 132
- (short) 0xa24f, // (10, 591) 133
- (short) 0xc89f, // (12, 2207) 134
- (short) 0x80e7, // ( 8, 231) 135
- (short) 0xc49f, // (12, 1183) 136
- (short) 0xb62f, // (11, 1583) 137
- (short) 0xcc9f, // (12, 3231) 138
- (short) 0x91b7, // ( 9, 439) 139
- (short) 0xc29f, // (12, 671) 140
- (short) 0xb12f, // (11, 303) 141
- (short) 0xca9f, // (12, 2719) 142
- (short) 0x9077, // ( 9, 119) 143
- (short) 0xc69f, // (12, 1695) 144
- (short) 0xce9f, // (12, 3743) 145
- (short) 0xc19f, // (12, 415) 146
- (short) 0xa14f, // (10, 335) 147
- (short) 0xc99f, // (12, 2463) 148
- (short) 0xc59f, // (12, 1439) 149
- (short) 0xcd9f, // (12, 3487) 150
- (short) 0xa34f, // (10, 847) 151
- (short) 0xc39f, // (12, 927) 152
- (short) 0xcb9f, // (12, 2975) 153
- (short) 0xc79f, // (12, 1951) 154
- (short) 0xb52f, // (11, 1327) 155
- (short) 0xcf9f, // (12, 3999) 156
- (short) 0xc05f, // (12, 95) 157
- (short) 0xc85f, // (12, 2143) 158
- (short) 0xb32f, // (11, 815) 159
- (short) 0xc45f, // (12, 1119) 160
- (short) 0xcc5f, // (12, 3167) 161
- (short) 0xc25f, // (12, 607) 162
- (short) 0xb72f, // (11, 1839) 163
- (short) 0xca5f, // (12, 2655) 164
- (short) 0xc65f, // (12, 1631) 165
- (short) 0xce5f, // (12, 3679) 166
- (short) 0xb0af, // (11, 175) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 1 of 22) (steady 1 of 16) (phase = 0.093750000 = 3.0 / 32.0)
- // entropy: 4.4574755684414029133
- // avg_length: 4.5336306265208552446; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0xa177, // (10, 375) 0
- (short) 0x5009, // ( 5, 9) 1
- (short) 0x803b, // ( 8, 59) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0x9017, // ( 9, 23) 4
- (short) 0x5019, // ( 5, 25) 5
- (short) 0x700b, // ( 7, 11) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xb34f, // (11, 847) 8
- (short) 0x601d, // ( 6, 29) 9
- (short) 0x9117, // ( 9, 279) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xa377, // (10, 887) 12
- (short) 0x603d, // ( 6, 61) 13
- (short) 0x80bb, // ( 8, 187) 14
- (short) 0x3006, // ( 3, 6) 15
- (short) 0xc4af, // (12, 1199) 16
- (short) 0x704b, // ( 7, 75) 17
- (short) 0xa0f7, // (10, 247) 18
- (short) 0x5005, // ( 5, 5) 19
- (short) 0xb74f, // (11, 1871) 20
- (short) 0x702b, // ( 7, 43) 21
- (short) 0x9097, // ( 9, 151) 22
- (short) 0x5015, // ( 5, 21) 23
- (short) 0xccaf, // (12, 3247) 24
- (short) 0x807b, // ( 8, 123) 25
- (short) 0xb0cf, // (11, 207) 26
- (short) 0x6003, // ( 6, 3) 27
- (short) 0xc2af, // (12, 687) 28
- (short) 0x80fb, // ( 8, 251) 29
- (short) 0xa2f7, // (10, 759) 30
- (short) 0x500d, // ( 5, 13) 31
- (short) 0xcaaf, // (12, 2735) 32
- (short) 0x8007, // ( 8, 7) 33
- (short) 0xb4cf, // (11, 1231) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xc6af, // (12, 1711) 36
- (short) 0x8087, // ( 8, 135) 37
- (short) 0xa1f7, // (10, 503) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xceaf, // (12, 3759) 40
- (short) 0x9197, // ( 9, 407) 41
- (short) 0xc1af, // (12, 431) 42
- (short) 0x706b, // ( 7, 107) 43
- (short) 0xc9af, // (12, 2479) 44
- (short) 0x9057, // ( 9, 87) 45
- (short) 0xb2cf, // (11, 719) 46
- (short) 0x6033, // ( 6, 51) 47
- (short) 0xc5af, // (12, 1455) 48
- (short) 0xa3f7, // (10, 1015) 49
- (short) 0xcdaf, // (12, 3503) 50
- (short) 0x8047, // ( 8, 71) 51
- (short) 0xc3af, // (12, 943) 52
- (short) 0xa00f, // (10, 15) 53
- (short) 0xcbaf, // (12, 2991) 54
- (short) 0x80c7, // ( 8, 199) 55
- (short) 0xc7af, // (12, 1967) 56
- (short) 0xb6cf, // (11, 1743) 57
- (short) 0xcfaf, // (12, 4015) 58
- (short) 0x9157, // ( 9, 343) 59
- (short) 0xc06f, // (12, 111) 60
- (short) 0xb1cf, // (11, 463) 61
- (short) 0xc86f, // (12, 2159) 62
- (short) 0x90d7, // ( 9, 215) 63
- (short) 0xc46f, // (12, 1135) 64
- (short) 0x91d7, // ( 9, 471) 65
- (short) 0xcc6f, // (12, 3183) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xc26f, // (12, 623) 68
- (short) 0x9037, // ( 9, 55) 69
- (short) 0xb5cf, // (11, 1487) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xca6f, // (12, 2671) 72
- (short) 0xa20f, // (10, 527) 73
- (short) 0xc66f, // (12, 1647) 74
- (short) 0x8027, // ( 8, 39) 75
- (short) 0xce6f, // (12, 3695) 76
- (short) 0xa10f, // (10, 271) 77
- (short) 0xc16f, // (12, 367) 78
- (short) 0x80a7, // ( 8, 167) 79
- (short) 0xc96f, // (12, 2415) 80
- (short) 0xb3cf, // (11, 975) 81
- (short) 0xc56f, // (12, 1391) 82
- (short) 0x9137, // ( 9, 311) 83
- (short) 0xcd6f, // (12, 3439) 84
- (short) 0xb7cf, // (11, 1999) 85
- (short) 0xc36f, // (12, 879) 86
- (short) 0x90b7, // ( 9, 183) 87
- (short) 0xcb6f, // (12, 2927) 88
- (short) 0xc76f, // (12, 1903) 89
- (short) 0xcf6f, // (12, 3951) 90
- (short) 0xa30f, // (10, 783) 91
- (short) 0xc0ef, // (12, 239) 92
- (short) 0xc8ef, // (12, 2287) 93
- (short) 0xc4ef, // (12, 1263) 94
- (short) 0xa08f, // (10, 143) 95
- (short) 0xccef, // (12, 3311) 96
- (short) 0xc2ef, // (12, 751) 97
- (short) 0xcaef, // (12, 2799) 98
- (short) 0xa28f, // (10, 655) 99
- (short) 0xc6ef, // (12, 1775) 100
- (short) 0xceef, // (12, 3823) 101
- (short) 0xc1ef, // (12, 495) 102
- (short) 0xa18f, // (10, 399) 103
- (short) 0xc9ef, // (12, 2543) 104
- (short) 0xc5ef, // (12, 1519) 105
- (short) 0xcdef, // (12, 3567) 106
- (short) 0xb02f, // (11, 47) 107
- (short) 0xc3ef, // (12, 1007) 108
- (short) 0xcbef, // (12, 3055) 109
- (short) 0xc7ef, // (12, 2031) 110
- (short) 0xb42f, // (11, 1071) 111
- (short) 0xcfef, // (12, 4079) 112
- (short) 0xc01f, // (12, 31) 113
- (short) 0xc81f, // (12, 2079) 114
- (short) 0xc41f, // (12, 1055) 115
- (short) 0xcc1f, // (12, 3103) 116
- (short) 0xc21f, // (12, 543) 117
- (short) 0xca1f, // (12, 2591) 118
- (short) 0xc61f, // (12, 1567) 119
- (short) 0xce1f, // (12, 3615) 120
- (short) 0xc11f, // (12, 287) 121
- (short) 0xc91f, // (12, 2335) 122
- (short) 0xc51f, // (12, 1311) 123
- (short) 0xcd1f, // (12, 3359) 124
- (short) 0xc31f, // (12, 799) 125
- (short) 0xcb1f, // (12, 2847) 126
- (short) 0xc71f, // (12, 1823) 127
- (short) 0xcf1f, // (12, 3871) 128
- (short) 0xa38f, // (10, 911) 129
- (short) 0xc09f, // (12, 159) 130
- (short) 0x8067, // ( 8, 103) 131
- (short) 0xc89f, // (12, 2207) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xc49f, // (12, 1183) 134
- (short) 0x80e7, // ( 8, 231) 135
- (short) 0xcc9f, // (12, 3231) 136
- (short) 0xb22f, // (11, 559) 137
- (short) 0xc29f, // (12, 671) 138
- (short) 0x91b7, // ( 9, 439) 139
- (short) 0xca9f, // (12, 2719) 140
- (short) 0xb62f, // (11, 1583) 141
- (short) 0xc69f, // (12, 1695) 142
- (short) 0x9077, // ( 9, 119) 143
- (short) 0xce9f, // (12, 3743) 144
- (short) 0xc19f, // (12, 415) 145
- (short) 0xc99f, // (12, 2463) 146
- (short) 0xa24f, // (10, 591) 147
- (short) 0xc59f, // (12, 1439) 148
- (short) 0xcd9f, // (12, 3487) 149
- (short) 0xc39f, // (12, 927) 150
- (short) 0xa14f, // (10, 335) 151
- (short) 0xcb9f, // (12, 2975) 152
- (short) 0xc79f, // (12, 1951) 153
- (short) 0xcf9f, // (12, 3999) 154
- (short) 0xb12f, // (11, 303) 155
- (short) 0xc05f, // (12, 95) 156
- (short) 0xc85f, // (12, 2143) 157
- (short) 0xc45f, // (12, 1119) 158
- (short) 0xb52f, // (11, 1327) 159
- (short) 0xcc5f, // (12, 3167) 160
- (short) 0xc25f, // (12, 607) 161
- (short) 0xca5f, // (12, 2655) 162
- (short) 0xb32f, // (11, 815) 163
- (short) 0xc65f, // (12, 1631) 164
- (short) 0xce5f, // (12, 3679) 165
- (short) 0xc15f, // (12, 351) 166
- (short) 0xb72f, // (11, 1839) 167
- (short) 0xc95f, // (12, 2399) 168
- (short) 0xc55f, // (12, 1375) 169
- (short) 0xcd5f, // (12, 3423) 170
- (short) 0xc35f, // (12, 863) 171
- (short) 0xcb5f, // (12, 2911) 172
- (short) 0xc75f, // (12, 1887) 173
- (short) 0xcf5f, // (12, 3935) 174
- (short) 0xb0af, // (11, 175) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 2 of 22) (steady 2 of 16) (phase = 0.156250000 = 5.0 / 32.0)
- // entropy: 4.4520619712441886762
- // avg_length: 4.5253989110544479146; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0xa177, // (10, 375) 0
- (short) 0x5009, // ( 5, 9) 1
- (short) 0x803b, // ( 8, 59) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0xa377, // (10, 887) 4
- (short) 0x5019, // ( 5, 25) 5
- (short) 0x80bb, // ( 8, 187) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xb34f, // (11, 847) 8
- (short) 0x601d, // ( 6, 29) 9
- (short) 0x9057, // ( 9, 87) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xb74f, // (11, 1871) 12
- (short) 0x603d, // ( 6, 61) 13
- (short) 0x807b, // ( 8, 123) 14
- (short) 0x3006, // ( 3, 6) 15
- (short) 0xc72f, // (12, 1839) 16
- (short) 0x700b, // ( 7, 11) 17
- (short) 0xa0f7, // (10, 247) 18
- (short) 0x5005, // ( 5, 5) 19
- (short) 0xcf2f, // (12, 3887) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0xa2f7, // (10, 759) 22
- (short) 0x5015, // ( 5, 21) 23
- (short) 0xc0af, // (12, 175) 24
- (short) 0x80fb, // ( 8, 251) 25
- (short) 0xb0cf, // (11, 207) 26
- (short) 0x6003, // ( 6, 3) 27
- (short) 0xc8af, // (12, 2223) 28
- (short) 0x8007, // ( 8, 7) 29
- (short) 0xa1f7, // (10, 503) 30
- (short) 0x500d, // ( 5, 13) 31
- (short) 0xc4af, // (12, 1199) 32
- (short) 0x8087, // ( 8, 135) 33
- (short) 0xb4cf, // (11, 1231) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xccaf, // (12, 3247) 36
- (short) 0x8047, // ( 8, 71) 37
- (short) 0xb2cf, // (11, 719) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xc2af, // (12, 687) 40
- (short) 0x9157, // ( 9, 343) 41
- (short) 0xcaaf, // (12, 2735) 42
- (short) 0x702b, // ( 7, 43) 43
- (short) 0xc6af, // (12, 1711) 44
- (short) 0x90d7, // ( 9, 215) 45
- (short) 0xceaf, // (12, 3759) 46
- (short) 0x6033, // ( 6, 51) 47
- (short) 0xc1af, // (12, 431) 48
- (short) 0xa3f7, // (10, 1015) 49
- (short) 0xc9af, // (12, 2479) 50
- (short) 0x80c7, // ( 8, 199) 51
- (short) 0xc5af, // (12, 1455) 52
- (short) 0xa00f, // (10, 15) 53
- (short) 0xcdaf, // (12, 3503) 54
- (short) 0x8027, // ( 8, 39) 55
- (short) 0xc3af, // (12, 943) 56
- (short) 0xb6cf, // (11, 1743) 57
- (short) 0xcbaf, // (12, 2991) 58
- (short) 0x91d7, // ( 9, 471) 59
- (short) 0xc7af, // (12, 1967) 60
- (short) 0xb1cf, // (11, 463) 61
- (short) 0xcfaf, // (12, 4015) 62
- (short) 0x80a7, // ( 8, 167) 63
- (short) 0xc06f, // (12, 111) 64
- (short) 0x9037, // ( 9, 55) 65
- (short) 0xc86f, // (12, 2159) 66
- (short) 0x706b, // ( 7, 107) 67
- (short) 0xc46f, // (12, 1135) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xcc6f, // (12, 3183) 70
- (short) 0x701b, // ( 7, 27) 71
- (short) 0xc26f, // (12, 623) 72
- (short) 0xa20f, // (10, 527) 73
- (short) 0xca6f, // (12, 2671) 74
- (short) 0x8067, // ( 8, 103) 75
- (short) 0xc66f, // (12, 1647) 76
- (short) 0xa10f, // (10, 271) 77
- (short) 0xce6f, // (12, 3695) 78
- (short) 0x705b, // ( 7, 91) 79
- (short) 0xc16f, // (12, 367) 80
- (short) 0xb5cf, // (11, 1487) 81
- (short) 0xc96f, // (12, 2415) 82
- (short) 0x90b7, // ( 9, 183) 83
- (short) 0xc56f, // (12, 1391) 84
- (short) 0xb3cf, // (11, 975) 85
- (short) 0xcd6f, // (12, 3439) 86
- (short) 0x91b7, // ( 9, 439) 87
- (short) 0xc36f, // (12, 879) 88
- (short) 0xcb6f, // (12, 2927) 89
- (short) 0xc76f, // (12, 1903) 90
- (short) 0xa30f, // (10, 783) 91
- (short) 0xcf6f, // (12, 3951) 92
- (short) 0xc0ef, // (12, 239) 93
- (short) 0xc8ef, // (12, 2287) 94
- (short) 0xa08f, // (10, 143) 95
- (short) 0xc4ef, // (12, 1263) 96
- (short) 0xccef, // (12, 3311) 97
- (short) 0xc2ef, // (12, 751) 98
- (short) 0xa28f, // (10, 655) 99
- (short) 0xcaef, // (12, 2799) 100
- (short) 0xc6ef, // (12, 1775) 101
- (short) 0xceef, // (12, 3823) 102
- (short) 0xa18f, // (10, 399) 103
- (short) 0xc1ef, // (12, 495) 104
- (short) 0xc9ef, // (12, 2543) 105
- (short) 0xc5ef, // (12, 1519) 106
- (short) 0xb7cf, // (11, 1999) 107
- (short) 0xcdef, // (12, 3567) 108
- (short) 0xc3ef, // (12, 1007) 109
- (short) 0xcbef, // (12, 3055) 110
- (short) 0xb02f, // (11, 47) 111
- (short) 0xc7ef, // (12, 2031) 112
- (short) 0xcfef, // (12, 4079) 113
- (short) 0xc01f, // (12, 31) 114
- (short) 0xc81f, // (12, 2079) 115
- (short) 0xc41f, // (12, 1055) 116
- (short) 0xcc1f, // (12, 3103) 117
- (short) 0xc21f, // (12, 543) 118
- (short) 0xca1f, // (12, 2591) 119
- (short) 0xc61f, // (12, 1567) 120
- (short) 0xce1f, // (12, 3615) 121
- (short) 0xc11f, // (12, 287) 122
- (short) 0xc91f, // (12, 2335) 123
- (short) 0xc51f, // (12, 1311) 124
- (short) 0xcd1f, // (12, 3359) 125
- (short) 0xc31f, // (12, 799) 126
- (short) 0xcb1f, // (12, 2847) 127
- (short) 0xc71f, // (12, 1823) 128
- (short) 0xa38f, // (10, 911) 129
- (short) 0xcf1f, // (12, 3871) 130
- (short) 0x80e7, // ( 8, 231) 131
- (short) 0xc09f, // (12, 159) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xc89f, // (12, 2207) 134
- (short) 0x8017, // ( 8, 23) 135
- (short) 0xc49f, // (12, 1183) 136
- (short) 0xb42f, // (11, 1071) 137
- (short) 0xcc9f, // (12, 3231) 138
- (short) 0x9077, // ( 9, 119) 139
- (short) 0xc29f, // (12, 671) 140
- (short) 0xb22f, // (11, 559) 141
- (short) 0xca9f, // (12, 2719) 142
- (short) 0x8097, // ( 8, 151) 143
- (short) 0xc69f, // (12, 1695) 144
- (short) 0xce9f, // (12, 3743) 145
- (short) 0xc19f, // (12, 415) 146
- (short) 0xa24f, // (10, 591) 147
- (short) 0xc99f, // (12, 2463) 148
- (short) 0xc59f, // (12, 1439) 149
- (short) 0xcd9f, // (12, 3487) 150
- (short) 0xa14f, // (10, 335) 151
- (short) 0xc39f, // (12, 927) 152
- (short) 0xcb9f, // (12, 2975) 153
- (short) 0xc79f, // (12, 1951) 154
- (short) 0xb62f, // (11, 1583) 155
- (short) 0xcf9f, // (12, 3999) 156
- (short) 0xc05f, // (12, 95) 157
- (short) 0xc85f, // (12, 2143) 158
- (short) 0xb12f, // (11, 303) 159
- (short) 0xc45f, // (12, 1119) 160
- (short) 0xcc5f, // (12, 3167) 161
- (short) 0xc25f, // (12, 607) 162
- (short) 0xb52f, // (11, 1327) 163
- (short) 0xca5f, // (12, 2655) 164
- (short) 0xc65f, // (12, 1631) 165
- (short) 0xce5f, // (12, 3679) 166
- (short) 0xb32f, // (11, 815) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 3 of 22) (steady 3 of 16) (phase = 0.218750000 = 7.0 / 32.0)
- // entropy: 4.4457680500675866853
- // avg_length: 4.5181192844586535173; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0xb24f, // (11, 591) 0
- (short) 0x601d, // ( 6, 29) 1
- (short) 0x9097, // ( 9, 151) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0xa1f7, // (10, 503) 4
- (short) 0x5005, // ( 5, 5) 5
- (short) 0x807b, // ( 8, 123) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xc52f, // (12, 1327) 8
- (short) 0x603d, // ( 6, 61) 9
- (short) 0x9197, // ( 9, 407) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xb64f, // (11, 1615) 12
- (short) 0x6003, // ( 6, 3) 13
- (short) 0x9057, // ( 9, 87) 14
- (short) 0x3006, // ( 3, 6) 15
- (short) 0xcd2f, // (12, 3375) 16
- (short) 0x80fb, // ( 8, 251) 17
- (short) 0xb14f, // (11, 335) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xc32f, // (12, 815) 20
- (short) 0x702b, // ( 7, 43) 21
- (short) 0xa3f7, // (10, 1015) 22
- (short) 0x4009, // ( 4, 9) 23
- (short) 0xcb2f, // (12, 2863) 24
- (short) 0x8007, // ( 8, 7) 25
- (short) 0xb54f, // (11, 1359) 26
- (short) 0x6023, // ( 6, 35) 27
- (short) 0xc72f, // (12, 1839) 28
- (short) 0x8087, // ( 8, 135) 29
- (short) 0xb34f, // (11, 847) 30
- (short) 0x500d, // ( 5, 13) 31
- (short) 0xcf2f, // (12, 3887) 32
- (short) 0x9157, // ( 9, 343) 33
- (short) 0xc0af, // (12, 175) 34
- (short) 0x6013, // ( 6, 19) 35
- (short) 0xc8af, // (12, 2223) 36
- (short) 0x8047, // ( 8, 71) 37
- (short) 0xb74f, // (11, 1871) 38
- (short) 0x6033, // ( 6, 51) 39
- (short) 0xc4af, // (12, 1199) 40
- (short) 0x90d7, // ( 9, 215) 41
- (short) 0xccaf, // (12, 3247) 42
- (short) 0x706b, // ( 7, 107) 43
- (short) 0xc2af, // (12, 687) 44
- (short) 0x91d7, // ( 9, 471) 45
- (short) 0xcaaf, // (12, 2735) 46
- (short) 0x600b, // ( 6, 11) 47
- (short) 0xc6af, // (12, 1711) 48
- (short) 0xb0cf, // (11, 207) 49
- (short) 0xceaf, // (12, 3759) 50
- (short) 0x80c7, // ( 8, 199) 51
- (short) 0xc1af, // (12, 431) 52
- (short) 0xa00f, // (10, 15) 53
- (short) 0xc9af, // (12, 2479) 54
- (short) 0x8027, // ( 8, 39) 55
- (short) 0xc5af, // (12, 1455) 56
- (short) 0xb4cf, // (11, 1231) 57
- (short) 0xcdaf, // (12, 3503) 58
- (short) 0x9037, // ( 9, 55) 59
- (short) 0xc3af, // (12, 943) 60
- (short) 0xb2cf, // (11, 719) 61
- (short) 0xcbaf, // (12, 2991) 62
- (short) 0x80a7, // ( 8, 167) 63
- (short) 0xc7af, // (12, 1967) 64
- (short) 0xa20f, // (10, 527) 65
- (short) 0xcfaf, // (12, 4015) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xc06f, // (12, 111) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xc86f, // (12, 2159) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc46f, // (12, 1135) 72
- (short) 0xb6cf, // (11, 1743) 73
- (short) 0xcc6f, // (12, 3183) 74
- (short) 0x8067, // ( 8, 103) 75
- (short) 0xc26f, // (12, 623) 76
- (short) 0xa10f, // (10, 271) 77
- (short) 0xca6f, // (12, 2671) 78
- (short) 0x703b, // ( 7, 59) 79
- (short) 0xc66f, // (12, 1647) 80
- (short) 0xce6f, // (12, 3695) 81
- (short) 0xc16f, // (12, 367) 82
- (short) 0x90b7, // ( 9, 183) 83
- (short) 0xc96f, // (12, 2415) 84
- (short) 0xb1cf, // (11, 463) 85
- (short) 0xc56f, // (12, 1391) 86
- (short) 0x91b7, // ( 9, 439) 87
- (short) 0xcd6f, // (12, 3439) 88
- (short) 0xc36f, // (12, 879) 89
- (short) 0xcb6f, // (12, 2927) 90
- (short) 0xa30f, // (10, 783) 91
- (short) 0xc76f, // (12, 1903) 92
- (short) 0xcf6f, // (12, 3951) 93
- (short) 0xc0ef, // (12, 239) 94
- (short) 0x9077, // ( 9, 119) 95
- (short) 0xc8ef, // (12, 2287) 96
- (short) 0xc4ef, // (12, 1263) 97
- (short) 0xccef, // (12, 3311) 98
- (short) 0xa08f, // (10, 143) 99
- (short) 0xc2ef, // (12, 751) 100
- (short) 0xcaef, // (12, 2799) 101
- (short) 0xc6ef, // (12, 1775) 102
- (short) 0xa28f, // (10, 655) 103
- (short) 0xceef, // (12, 3823) 104
- (short) 0xc1ef, // (12, 495) 105
- (short) 0xc9ef, // (12, 2543) 106
- (short) 0xb5cf, // (11, 1487) 107
- (short) 0xc5ef, // (12, 1519) 108
- (short) 0xcdef, // (12, 3567) 109
- (short) 0xc3ef, // (12, 1007) 110
- (short) 0xb3cf, // (11, 975) 111
- (short) 0xcbef, // (12, 3055) 112
- (short) 0xc7ef, // (12, 2031) 113
- (short) 0xcfef, // (12, 4079) 114
- (short) 0xc01f, // (12, 31) 115
- (short) 0xc81f, // (12, 2079) 116
- (short) 0xc41f, // (12, 1055) 117
- (short) 0xcc1f, // (12, 3103) 118
- (short) 0xc21f, // (12, 543) 119
- (short) 0xca1f, // (12, 2591) 120
- (short) 0xc61f, // (12, 1567) 121
- (short) 0xce1f, // (12, 3615) 122
- (short) 0xc11f, // (12, 287) 123
- (short) 0xc91f, // (12, 2335) 124
- (short) 0xc51f, // (12, 1311) 125
- (short) 0xcd1f, // (12, 3359) 126
- (short) 0xc31f, // (12, 799) 127
- (short) 0xcb1f, // (12, 2847) 128
- (short) 0xb7cf, // (11, 1999) 129
- (short) 0xc71f, // (12, 1823) 130
- (short) 0x80e7, // ( 8, 231) 131
- (short) 0xcf1f, // (12, 3871) 132
- (short) 0xa18f, // (10, 399) 133
- (short) 0xc09f, // (12, 159) 134
- (short) 0x8017, // ( 8, 23) 135
- (short) 0xc89f, // (12, 2207) 136
- (short) 0xc49f, // (12, 1183) 137
- (short) 0xcc9f, // (12, 3231) 138
- (short) 0x9177, // ( 9, 375) 139
- (short) 0xc29f, // (12, 671) 140
- (short) 0xb02f, // (11, 47) 141
- (short) 0xca9f, // (12, 2719) 142
- (short) 0x90f7, // ( 9, 247) 143
- (short) 0xc69f, // (12, 1695) 144
- (short) 0xce9f, // (12, 3743) 145
- (short) 0xc19f, // (12, 415) 146
- (short) 0xa38f, // (10, 911) 147
- (short) 0xc99f, // (12, 2463) 148
- (short) 0xc59f, // (12, 1439) 149
- (short) 0xcd9f, // (12, 3487) 150
- (short) 0xa04f, // (10, 79) 151
- (short) 0xc39f, // (12, 927) 152
- (short) 0xcb9f, // (12, 2975) 153
- (short) 0xc79f, // (12, 1951) 154
- (short) 0xb42f, // (11, 1071) 155
- (short) 0xcf9f, // (12, 3999) 156
- (short) 0xc05f, // (12, 95) 157
- (short) 0xc85f, // (12, 2143) 158
- (short) 0xb22f, // (11, 559) 159
- (short) 0xc45f, // (12, 1119) 160
- (short) 0xcc5f, // (12, 3167) 161
- (short) 0xc25f, // (12, 607) 162
- (short) 0xb62f, // (11, 1583) 163
- (short) 0xca5f, // (12, 2655) 164
- (short) 0xc65f, // (12, 1631) 165
- (short) 0xce5f, // (12, 3679) 166
- (short) 0xb12f, // (11, 303) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 4 of 22) (steady 4 of 16) (phase = 0.281250000 = 9.0 / 32.0)
- // entropy: 4.4386754570568340839
- // avg_length: 4.5071584786605640716; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0xb24f, // (11, 591) 0
- (short) 0x601d, // ( 6, 29) 1
- (short) 0x9057, // ( 9, 87) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0xb64f, // (11, 1615) 4
- (short) 0x5005, // ( 5, 5) 5
- (short) 0x807b, // ( 8, 123) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xc32f, // (12, 815) 8
- (short) 0x700b, // ( 7, 11) 9
- (short) 0xa0f7, // (10, 247) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xb14f, // (11, 335) 12
- (short) 0x603d, // ( 6, 61) 13
- (short) 0x9157, // ( 9, 343) 14
- (short) 0x3006, // ( 3, 6) 15
- (short) 0xcb2f, // (12, 2863) 16
- (short) 0x80fb, // ( 8, 251) 17
- (short) 0xb54f, // (11, 1359) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xc72f, // (12, 1839) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0xa2f7, // (10, 759) 22
- (short) 0x4009, // ( 4, 9) 23
- (short) 0xcf2f, // (12, 3887) 24
- (short) 0x8007, // ( 8, 7) 25
- (short) 0xb34f, // (11, 847) 26
- (short) 0x6003, // ( 6, 3) 27
- (short) 0xc0af, // (12, 175) 28
- (short) 0x8087, // ( 8, 135) 29
- (short) 0xb74f, // (11, 1871) 30
- (short) 0x500d, // ( 5, 13) 31
- (short) 0xc8af, // (12, 2223) 32
- (short) 0x90d7, // ( 9, 215) 33
- (short) 0xc4af, // (12, 1199) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xccaf, // (12, 3247) 36
- (short) 0x8047, // ( 8, 71) 37
- (short) 0xb0cf, // (11, 207) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xc2af, // (12, 687) 40
- (short) 0xa1f7, // (10, 503) 41
- (short) 0xcaaf, // (12, 2735) 42
- (short) 0x702b, // ( 7, 43) 43
- (short) 0xc6af, // (12, 1711) 44
- (short) 0x91d7, // ( 9, 471) 45
- (short) 0xceaf, // (12, 3759) 46
- (short) 0x6033, // ( 6, 51) 47
- (short) 0xc1af, // (12, 431) 48
- (short) 0xb4cf, // (11, 1231) 49
- (short) 0xc9af, // (12, 2479) 50
- (short) 0x80c7, // ( 8, 199) 51
- (short) 0xc5af, // (12, 1455) 52
- (short) 0xa3f7, // (10, 1015) 53
- (short) 0xcdaf, // (12, 3503) 54
- (short) 0x706b, // ( 7, 107) 55
- (short) 0xc3af, // (12, 943) 56
- (short) 0xb2cf, // (11, 719) 57
- (short) 0xcbaf, // (12, 2991) 58
- (short) 0x9037, // ( 9, 55) 59
- (short) 0xc7af, // (12, 1967) 60
- (short) 0xb6cf, // (11, 1743) 61
- (short) 0xcfaf, // (12, 4015) 62
- (short) 0x8027, // ( 8, 39) 63
- (short) 0xc06f, // (12, 111) 64
- (short) 0xa00f, // (10, 15) 65
- (short) 0xc86f, // (12, 2159) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xc46f, // (12, 1135) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xcc6f, // (12, 3183) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc26f, // (12, 623) 72
- (short) 0xb1cf, // (11, 463) 73
- (short) 0xca6f, // (12, 2671) 74
- (short) 0x80a7, // ( 8, 167) 75
- (short) 0xc66f, // (12, 1647) 76
- (short) 0xa20f, // (10, 527) 77
- (short) 0xce6f, // (12, 3695) 78
- (short) 0x703b, // ( 7, 59) 79
- (short) 0xc16f, // (12, 367) 80
- (short) 0xc96f, // (12, 2415) 81
- (short) 0xc56f, // (12, 1391) 82
- (short) 0x90b7, // ( 9, 183) 83
- (short) 0xcd6f, // (12, 3439) 84
- (short) 0xb5cf, // (11, 1487) 85
- (short) 0xc36f, // (12, 879) 86
- (short) 0x8067, // ( 8, 103) 87
- (short) 0xcb6f, // (12, 2927) 88
- (short) 0xc76f, // (12, 1903) 89
- (short) 0xcf6f, // (12, 3951) 90
- (short) 0xa10f, // (10, 271) 91
- (short) 0xc0ef, // (12, 239) 92
- (short) 0xc8ef, // (12, 2287) 93
- (short) 0xc4ef, // (12, 1263) 94
- (short) 0x91b7, // ( 9, 439) 95
- (short) 0xccef, // (12, 3311) 96
- (short) 0xc2ef, // (12, 751) 97
- (short) 0xcaef, // (12, 2799) 98
- (short) 0xa30f, // (10, 783) 99
- (short) 0xc6ef, // (12, 1775) 100
- (short) 0xceef, // (12, 3823) 101
- (short) 0xc1ef, // (12, 495) 102
- (short) 0xa08f, // (10, 143) 103
- (short) 0xc9ef, // (12, 2543) 104
- (short) 0xc5ef, // (12, 1519) 105
- (short) 0xcdef, // (12, 3567) 106
- (short) 0xb3cf, // (11, 975) 107
- (short) 0xc3ef, // (12, 1007) 108
- (short) 0xcbef, // (12, 3055) 109
- (short) 0xc7ef, // (12, 2031) 110
- (short) 0xa28f, // (10, 655) 111
- (short) 0xcfef, // (12, 4079) 112
- (short) 0xc01f, // (12, 31) 113
- (short) 0xc81f, // (12, 2079) 114
- (short) 0xc41f, // (12, 1055) 115
- (short) 0xcc1f, // (12, 3103) 116
- (short) 0xc21f, // (12, 543) 117
- (short) 0xca1f, // (12, 2591) 118
- (short) 0xb7cf, // (11, 1999) 119
- (short) 0xc61f, // (12, 1567) 120
- (short) 0xce1f, // (12, 3615) 121
- (short) 0xc11f, // (12, 287) 122
- (short) 0xc91f, // (12, 2335) 123
- (short) 0xc51f, // (12, 1311) 124
- (short) 0xcd1f, // (12, 3359) 125
- (short) 0xc31f, // (12, 799) 126
- (short) 0xcb1f, // (12, 2847) 127
- (short) 0xc71f, // (12, 1823) 128
- (short) 0xb02f, // (11, 47) 129
- (short) 0xcf1f, // (12, 3871) 130
- (short) 0x80e7, // ( 8, 231) 131
- (short) 0xc09f, // (12, 159) 132
- (short) 0xa18f, // (10, 399) 133
- (short) 0xc89f, // (12, 2207) 134
- (short) 0x8017, // ( 8, 23) 135
- (short) 0xc49f, // (12, 1183) 136
- (short) 0xcc9f, // (12, 3231) 137
- (short) 0xc29f, // (12, 671) 138
- (short) 0x9077, // ( 9, 119) 139
- (short) 0xca9f, // (12, 2719) 140
- (short) 0xb42f, // (11, 1071) 141
- (short) 0xc69f, // (12, 1695) 142
- (short) 0x8097, // ( 8, 151) 143
- (short) 0xce9f, // (12, 3743) 144
- (short) 0xc19f, // (12, 415) 145
- (short) 0xc99f, // (12, 2463) 146
- (short) 0xa38f, // (10, 911) 147
- (short) 0xc59f, // (12, 1439) 148
- (short) 0xcd9f, // (12, 3487) 149
- (short) 0xc39f, // (12, 927) 150
- (short) 0x9177, // ( 9, 375) 151
- (short) 0xcb9f, // (12, 2975) 152
- (short) 0xc79f, // (12, 1951) 153
- (short) 0xcf9f, // (12, 3999) 154
- (short) 0xb22f, // (11, 559) 155
- (short) 0xc05f, // (12, 95) 156
- (short) 0xc85f, // (12, 2143) 157
- (short) 0xc45f, // (12, 1119) 158
- (short) 0xa04f, // (10, 79) 159
- (short) 0xcc5f, // (12, 3167) 160
- (short) 0xc25f, // (12, 607) 161
- (short) 0xca5f, // (12, 2655) 162
- (short) 0xb62f, // (11, 1583) 163
- (short) 0xc65f, // (12, 1631) 164
- (short) 0xce5f, // (12, 3679) 165
- (short) 0xc15f, // (12, 351) 166
- (short) 0xb12f, // (11, 303) 167
- (short) 0xc95f, // (12, 2399) 168
- (short) 0xc55f, // (12, 1375) 169
- (short) 0xcd5f, // (12, 3423) 170
- (short) 0xc35f, // (12, 863) 171
- (short) 0xcb5f, // (12, 2911) 172
- (short) 0xc75f, // (12, 1887) 173
- (short) 0xcf5f, // (12, 3935) 174
- (short) 0xb52f, // (11, 1327) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 5 of 22) (steady 5 of 16) (phase = 0.343750000 = 11.0 / 32.0)
- // entropy: 4.4308578632493116345
- // avg_length: 4.4996166821663301505; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0xc12f, // (12, 303) 0
- (short) 0x601d, // ( 6, 29) 1
- (short) 0x9057, // ( 9, 87) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0xb14f, // (11, 335) 4
- (short) 0x5005, // ( 5, 5) 5
- (short) 0x807b, // ( 8, 123) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xc92f, // (12, 2351) 8
- (short) 0x700b, // ( 7, 11) 9
- (short) 0xa1f7, // (10, 503) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xc52f, // (12, 1327) 12
- (short) 0x603d, // ( 6, 61) 13
- (short) 0x9157, // ( 9, 343) 14
- (short) 0x3006, // ( 3, 6) 15
- (short) 0xcd2f, // (12, 3375) 16
- (short) 0x80fb, // ( 8, 251) 17
- (short) 0xb54f, // (11, 1359) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xc32f, // (12, 815) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0xa3f7, // (10, 1015) 22
- (short) 0x4009, // ( 4, 9) 23
- (short) 0xcb2f, // (12, 2863) 24
- (short) 0x8007, // ( 8, 7) 25
- (short) 0xc72f, // (12, 1839) 26
- (short) 0x6003, // ( 6, 3) 27
- (short) 0xcf2f, // (12, 3887) 28
- (short) 0x8087, // ( 8, 135) 29
- (short) 0xb34f, // (11, 847) 30
- (short) 0x500d, // ( 5, 13) 31
- (short) 0xc0af, // (12, 175) 32
- (short) 0x90d7, // ( 9, 215) 33
- (short) 0xc8af, // (12, 2223) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xc4af, // (12, 1199) 36
- (short) 0x8047, // ( 8, 71) 37
- (short) 0xb74f, // (11, 1871) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xccaf, // (12, 3247) 40
- (short) 0xa00f, // (10, 15) 41
- (short) 0xc2af, // (12, 687) 42
- (short) 0x702b, // ( 7, 43) 43
- (short) 0xcaaf, // (12, 2735) 44
- (short) 0x91d7, // ( 9, 471) 45
- (short) 0xc6af, // (12, 1711) 46
- (short) 0x6033, // ( 6, 51) 47
- (short) 0xceaf, // (12, 3759) 48
- (short) 0xb0cf, // (11, 207) 49
- (short) 0xc1af, // (12, 431) 50
- (short) 0x80c7, // ( 8, 199) 51
- (short) 0xc9af, // (12, 2479) 52
- (short) 0xa20f, // (10, 527) 53
- (short) 0xc5af, // (12, 1455) 54
- (short) 0x706b, // ( 7, 107) 55
- (short) 0xcdaf, // (12, 3503) 56
- (short) 0xc3af, // (12, 943) 57
- (short) 0xcbaf, // (12, 2991) 58
- (short) 0x9037, // ( 9, 55) 59
- (short) 0xc7af, // (12, 1967) 60
- (short) 0xb4cf, // (11, 1231) 61
- (short) 0xcfaf, // (12, 4015) 62
- (short) 0x8027, // ( 8, 39) 63
- (short) 0xc06f, // (12, 111) 64
- (short) 0xa10f, // (10, 271) 65
- (short) 0xc86f, // (12, 2159) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xc46f, // (12, 1135) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xcc6f, // (12, 3183) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc26f, // (12, 623) 72
- (short) 0xb2cf, // (11, 719) 73
- (short) 0xca6f, // (12, 2671) 74
- (short) 0x80a7, // ( 8, 167) 75
- (short) 0xc66f, // (12, 1647) 76
- (short) 0xa30f, // (10, 783) 77
- (short) 0xce6f, // (12, 3695) 78
- (short) 0x703b, // ( 7, 59) 79
- (short) 0xc16f, // (12, 367) 80
- (short) 0xc96f, // (12, 2415) 81
- (short) 0xc56f, // (12, 1391) 82
- (short) 0x90b7, // ( 9, 183) 83
- (short) 0xcd6f, // (12, 3439) 84
- (short) 0xb6cf, // (11, 1743) 85
- (short) 0xc36f, // (12, 879) 86
- (short) 0x8067, // ( 8, 103) 87
- (short) 0xcb6f, // (12, 2927) 88
- (short) 0xc76f, // (12, 1903) 89
- (short) 0xcf6f, // (12, 3951) 90
- (short) 0xa08f, // (10, 143) 91
- (short) 0xc0ef, // (12, 239) 92
- (short) 0xc8ef, // (12, 2287) 93
- (short) 0xc4ef, // (12, 1263) 94
- (short) 0x91b7, // ( 9, 439) 95
- (short) 0xccef, // (12, 3311) 96
- (short) 0xc2ef, // (12, 751) 97
- (short) 0xcaef, // (12, 2799) 98
- (short) 0xa28f, // (10, 655) 99
- (short) 0xc6ef, // (12, 1775) 100
- (short) 0xceef, // (12, 3823) 101
- (short) 0xc1ef, // (12, 495) 102
- (short) 0x9077, // ( 9, 119) 103
- (short) 0xc9ef, // (12, 2543) 104
- (short) 0xc5ef, // (12, 1519) 105
- (short) 0xcdef, // (12, 3567) 106
- (short) 0xb1cf, // (11, 463) 107
- (short) 0xc3ef, // (12, 1007) 108
- (short) 0xcbef, // (12, 3055) 109
- (short) 0xc7ef, // (12, 2031) 110
- (short) 0xa18f, // (10, 399) 111
- (short) 0xcfef, // (12, 4079) 112
- (short) 0xc01f, // (12, 31) 113
- (short) 0xc81f, // (12, 2079) 114
- (short) 0xc41f, // (12, 1055) 115
- (short) 0xcc1f, // (12, 3103) 116
- (short) 0xc21f, // (12, 543) 117
- (short) 0xca1f, // (12, 2591) 118
- (short) 0xb5cf, // (11, 1487) 119
- (short) 0xc61f, // (12, 1567) 120
- (short) 0xce1f, // (12, 3615) 121
- (short) 0xc11f, // (12, 287) 122
- (short) 0xc91f, // (12, 2335) 123
- (short) 0xc51f, // (12, 1311) 124
- (short) 0xcd1f, // (12, 3359) 125
- (short) 0xc31f, // (12, 799) 126
- (short) 0xcb1f, // (12, 2847) 127
- (short) 0xc71f, // (12, 1823) 128
- (short) 0xb3cf, // (11, 975) 129
- (short) 0xcf1f, // (12, 3871) 130
- (short) 0x80e7, // ( 8, 231) 131
- (short) 0xc09f, // (12, 159) 132
- (short) 0xa38f, // (10, 911) 133
- (short) 0xc89f, // (12, 2207) 134
- (short) 0x8017, // ( 8, 23) 135
- (short) 0xc49f, // (12, 1183) 136
- (short) 0xcc9f, // (12, 3231) 137
- (short) 0xc29f, // (12, 671) 138
- (short) 0x9177, // ( 9, 375) 139
- (short) 0xca9f, // (12, 2719) 140
- (short) 0xb7cf, // (11, 1999) 141
- (short) 0xc69f, // (12, 1695) 142
- (short) 0x8097, // ( 8, 151) 143
- (short) 0xce9f, // (12, 3743) 144
- (short) 0xc19f, // (12, 415) 145
- (short) 0xc99f, // (12, 2463) 146
- (short) 0xa04f, // (10, 79) 147
- (short) 0xc59f, // (12, 1439) 148
- (short) 0xcd9f, // (12, 3487) 149
- (short) 0xc39f, // (12, 927) 150
- (short) 0x90f7, // ( 9, 247) 151
- (short) 0xcb9f, // (12, 2975) 152
- (short) 0xc79f, // (12, 1951) 153
- (short) 0xcf9f, // (12, 3999) 154
- (short) 0xb02f, // (11, 47) 155
- (short) 0xc05f, // (12, 95) 156
- (short) 0xc85f, // (12, 2143) 157
- (short) 0xc45f, // (12, 1119) 158
- (short) 0xa24f, // (10, 591) 159
- (short) 0xcc5f, // (12, 3167) 160
- (short) 0xc25f, // (12, 607) 161
- (short) 0xca5f, // (12, 2655) 162
- (short) 0xb42f, // (11, 1071) 163
- (short) 0xc65f, // (12, 1631) 164
- (short) 0xce5f, // (12, 3679) 165
- (short) 0xc15f, // (12, 351) 166
- (short) 0xb22f, // (11, 559) 167
- (short) 0xc95f, // (12, 2399) 168
- (short) 0xc55f, // (12, 1375) 169
- (short) 0xcd5f, // (12, 3423) 170
- (short) 0xc35f, // (12, 863) 171
- (short) 0xcb5f, // (12, 2911) 172
- (short) 0xc75f, // (12, 1887) 173
- (short) 0xcf5f, // (12, 3935) 174
- (short) 0xb62f, // (11, 1583) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 6 of 22) (steady 6 of 16) (phase = 0.406250000 = 13.0 / 32.0)
- // entropy: 4.4310364988500126060
- // avg_length: 4.5051134111084252254; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x601d, // ( 6, 29) 0
- (short) 0x3002, // ( 3, 2) 1
- (short) 0x603d, // ( 6, 61) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x700b, // ( 7, 11) 4
- (short) 0x4001, // ( 4, 1) 5
- (short) 0x6003, // ( 6, 3) 6
- (short) 0x3006, // ( 3, 6) 7
- (short) 0x807b, // ( 8, 123) 8
- (short) 0x5005, // ( 5, 5) 9
- (short) 0x704b, // ( 7, 75) 10
- (short) 0x4009, // ( 4, 9) 11
- (short) 0x9097, // ( 9, 151) 12
- (short) 0x6023, // ( 6, 35) 13
- (short) 0x80fb, // ( 8, 251) 14
- (short) 0x5015, // ( 5, 21) 15
- (short) 0x9197, // ( 9, 407) 16
- (short) 0x6013, // ( 6, 19) 17
- (short) 0x8007, // ( 8, 7) 18
- (short) 0x500d, // ( 5, 13) 19
- (short) 0xa0f7, // (10, 247) 20
- (short) 0x702b, // ( 7, 43) 21
- (short) 0x9057, // ( 9, 87) 22
- (short) 0x6033, // ( 6, 51) 23
- (short) 0xb14f, // (11, 335) 24
- (short) 0x8087, // ( 8, 135) 25
- (short) 0xa2f7, // (10, 759) 26
- (short) 0x706b, // ( 7, 107) 27
- (short) 0xb54f, // (11, 1359) 28
- (short) 0x9157, // ( 9, 343) 29
- (short) 0xa1f7, // (10, 503) 30
- (short) 0x8047, // ( 8, 71) 31
- (short) 0xa3f7, // (10, 1015) 32
- (short) 0x701b, // ( 7, 27) 33
- (short) 0x90d7, // ( 9, 215) 34
- (short) 0x705b, // ( 7, 91) 35
- (short) 0xb34f, // (11, 847) 36
- (short) 0x80c7, // ( 8, 199) 37
- (short) 0xa00f, // (10, 15) 38
- (short) 0x703b, // ( 7, 59) 39
- (short) 0xc32f, // (12, 815) 40
- (short) 0x91d7, // ( 9, 471) 41
- (short) 0xb74f, // (11, 1871) 42
- (short) 0x8027, // ( 8, 39) 43
- (short) 0xcb2f, // (12, 2863) 44
- (short) 0xa20f, // (10, 527) 45
- (short) 0xb0cf, // (11, 207) 46
- (short) 0x9037, // ( 9, 55) 47
- (short) 0xc72f, // (12, 1839) 48
- (short) 0xa10f, // (10, 271) 49
- (short) 0xcf2f, // (12, 3887) 50
- (short) 0x9137, // ( 9, 311) 51
- (short) 0xc0af, // (12, 175) 52
- (short) 0xb4cf, // (11, 1231) 53
- (short) 0xc8af, // (12, 2223) 54
- (short) 0xa30f, // (10, 783) 55
- (short) 0xc4af, // (12, 1199) 56
- (short) 0xccaf, // (12, 3247) 57
- (short) 0xc2af, // (12, 687) 58
- (short) 0xb2cf, // (11, 719) 59
- (short) 0xcaaf, // (12, 2735) 60
- (short) 0xc6af, // (12, 1711) 61
- (short) 0xceaf, // (12, 3759) 62
- (short) 0xb6cf, // (11, 1743) 63
- (short) 0xb1cf, // (11, 463) 64
- (short) 0x80a7, // ( 8, 167) 65
- (short) 0xa08f, // (10, 143) 66
- (short) 0x8067, // ( 8, 103) 67
- (short) 0xc1af, // (12, 431) 68
- (short) 0x90b7, // ( 9, 183) 69
- (short) 0xb5cf, // (11, 1487) 70
- (short) 0x80e7, // ( 8, 231) 71
- (short) 0xc9af, // (12, 2479) 72
- (short) 0xa28f, // (10, 655) 73
- (short) 0xc5af, // (12, 1455) 74
- (short) 0x91b7, // ( 9, 439) 75
- (short) 0xcdaf, // (12, 3503) 76
- (short) 0xb3cf, // (11, 975) 77
- (short) 0xc3af, // (12, 943) 78
- (short) 0xa18f, // (10, 399) 79
- (short) 0xcbaf, // (12, 2991) 80
- (short) 0xb7cf, // (11, 1999) 81
- (short) 0xc7af, // (12, 1967) 82
- (short) 0xa38f, // (10, 911) 83
- (short) 0xcfaf, // (12, 4015) 84
- (short) 0xc06f, // (12, 111) 85
- (short) 0xc86f, // (12, 2159) 86
- (short) 0xb02f, // (11, 47) 87
- (short) 0xc46f, // (12, 1135) 88
- (short) 0xcc6f, // (12, 3183) 89
- (short) 0xc26f, // (12, 623) 90
- (short) 0xca6f, // (12, 2671) 91
- (short) 0xc66f, // (12, 1647) 92
- (short) 0xce6f, // (12, 3695) 93
- (short) 0xc16f, // (12, 367) 94
- (short) 0xc96f, // (12, 2415) 95
- (short) 0xc56f, // (12, 1391) 96
- (short) 0xcd6f, // (12, 3439) 97
- (short) 0xc36f, // (12, 879) 98
- (short) 0xb42f, // (11, 1071) 99
- (short) 0xcb6f, // (12, 2927) 100
- (short) 0xc76f, // (12, 1903) 101
- (short) 0xcf6f, // (12, 3951) 102
- (short) 0xc0ef, // (12, 239) 103
- (short) 0xc8ef, // (12, 2287) 104
- (short) 0xc4ef, // (12, 1263) 105
- (short) 0xccef, // (12, 3311) 106
- (short) 0xc2ef, // (12, 751) 107
- (short) 0xcaef, // (12, 2799) 108
- (short) 0xc6ef, // (12, 1775) 109
- (short) 0xceef, // (12, 3823) 110
- (short) 0xc1ef, // (12, 495) 111
- (short) 0xc9ef, // (12, 2543) 112
- (short) 0xc5ef, // (12, 1519) 113
- (short) 0xcdef, // (12, 3567) 114
- (short) 0xc3ef, // (12, 1007) 115
- (short) 0xcbef, // (12, 3055) 116
- (short) 0xc7ef, // (12, 2031) 117
- (short) 0xcfef, // (12, 4079) 118
- (short) 0xc01f, // (12, 31) 119
- (short) 0xc81f, // (12, 2079) 120
- (short) 0xc41f, // (12, 1055) 121
- (short) 0xcc1f, // (12, 3103) 122
- (short) 0xc21f, // (12, 543) 123
- (short) 0xca1f, // (12, 2591) 124
- (short) 0xc61f, // (12, 1567) 125
- (short) 0xce1f, // (12, 3615) 126
- (short) 0xc11f, // (12, 287) 127
- (short) 0xc91f, // (12, 2335) 128
- (short) 0x9077, // ( 9, 119) 129
- (short) 0xb22f, // (11, 559) 130
- (short) 0x8017, // ( 8, 23) 131
- (short) 0xc51f, // (12, 1311) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xcd1f, // (12, 3359) 134
- (short) 0x9177, // ( 9, 375) 135
- (short) 0xc31f, // (12, 799) 136
- (short) 0xb62f, // (11, 1583) 137
- (short) 0xcb1f, // (12, 2847) 138
- (short) 0xa24f, // (10, 591) 139
- (short) 0xc71f, // (12, 1823) 140
- (short) 0xcf1f, // (12, 3871) 141
- (short) 0xc09f, // (12, 159) 142
- (short) 0xb12f, // (11, 303) 143
- (short) 0xc89f, // (12, 2207) 144
- (short) 0xc49f, // (12, 1183) 145
- (short) 0xcc9f, // (12, 3231) 146
- (short) 0xb52f, // (11, 1327) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 7 of 22) (steady 7 of 16) (phase = 0.468750000 = 15.0 / 32.0)
- // entropy: 4.4417871821766841123
- // avg_length: 4.5206419191518980583; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x700b, // ( 7, 11) 0
- (short) 0x3002, // ( 3, 2) 1
- (short) 0x601d, // ( 6, 29) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x704b, // ( 7, 75) 4
- (short) 0x4001, // ( 4, 1) 5
- (short) 0x603d, // ( 6, 61) 6
- (short) 0x3006, // ( 3, 6) 7
- (short) 0x8007, // ( 8, 7) 8
- (short) 0x5005, // ( 5, 5) 9
- (short) 0x702b, // ( 7, 43) 10
- (short) 0x4009, // ( 4, 9) 11
- (short) 0x9097, // ( 9, 151) 12
- (short) 0x6003, // ( 6, 3) 13
- (short) 0x8087, // ( 8, 135) 14
- (short) 0x5015, // ( 5, 21) 15
- (short) 0x9197, // ( 9, 407) 16
- (short) 0x6023, // ( 6, 35) 17
- (short) 0x8047, // ( 8, 71) 18
- (short) 0x500d, // ( 5, 13) 19
- (short) 0xa0f7, // (10, 247) 20
- (short) 0x706b, // ( 7, 107) 21
- (short) 0x9057, // ( 9, 87) 22
- (short) 0x6013, // ( 6, 19) 23
- (short) 0xb14f, // (11, 335) 24
- (short) 0x80c7, // ( 8, 199) 25
- (short) 0xa2f7, // (10, 759) 26
- (short) 0x701b, // ( 7, 27) 27
- (short) 0xc52f, // (12, 1327) 28
- (short) 0x9157, // ( 9, 343) 29
- (short) 0xb54f, // (11, 1359) 30
- (short) 0x8027, // ( 8, 39) 31
- (short) 0xa1f7, // (10, 503) 32
- (short) 0x705b, // ( 7, 91) 33
- (short) 0x90d7, // ( 9, 215) 34
- (short) 0x6033, // ( 6, 51) 35
- (short) 0xb34f, // (11, 847) 36
- (short) 0x80a7, // ( 8, 167) 37
- (short) 0xa3f7, // (10, 1015) 38
- (short) 0x703b, // ( 7, 59) 39
- (short) 0xcd2f, // (12, 3375) 40
- (short) 0x91d7, // ( 9, 471) 41
- (short) 0xb74f, // (11, 1871) 42
- (short) 0x8067, // ( 8, 103) 43
- (short) 0xc32f, // (12, 815) 44
- (short) 0xa00f, // (10, 15) 45
- (short) 0xcb2f, // (12, 2863) 46
- (short) 0x9037, // ( 9, 55) 47
- (short) 0xc72f, // (12, 1839) 48
- (short) 0xa20f, // (10, 527) 49
- (short) 0xcf2f, // (12, 3887) 50
- (short) 0x9137, // ( 9, 311) 51
- (short) 0xc0af, // (12, 175) 52
- (short) 0xb0cf, // (11, 207) 53
- (short) 0xc8af, // (12, 2223) 54
- (short) 0xa10f, // (10, 271) 55
- (short) 0xc4af, // (12, 1199) 56
- (short) 0xccaf, // (12, 3247) 57
- (short) 0xc2af, // (12, 687) 58
- (short) 0xb4cf, // (11, 1231) 59
- (short) 0xcaaf, // (12, 2735) 60
- (short) 0xc6af, // (12, 1711) 61
- (short) 0xceaf, // (12, 3759) 62
- (short) 0xb2cf, // (11, 719) 63
- (short) 0xb6cf, // (11, 1743) 64
- (short) 0x80e7, // ( 8, 231) 65
- (short) 0xa30f, // (10, 783) 66
- (short) 0x707b, // ( 7, 123) 67
- (short) 0xc1af, // (12, 431) 68
- (short) 0x90b7, // ( 9, 183) 69
- (short) 0xb1cf, // (11, 463) 70
- (short) 0x8017, // ( 8, 23) 71
- (short) 0xc9af, // (12, 2479) 72
- (short) 0xa08f, // (10, 143) 73
- (short) 0xc5af, // (12, 1455) 74
- (short) 0x91b7, // ( 9, 439) 75
- (short) 0xcdaf, // (12, 3503) 76
- (short) 0xb5cf, // (11, 1487) 77
- (short) 0xc3af, // (12, 943) 78
- (short) 0xa28f, // (10, 655) 79
- (short) 0xcbaf, // (12, 2991) 80
- (short) 0xb3cf, // (11, 975) 81
- (short) 0xc7af, // (12, 1967) 82
- (short) 0xa18f, // (10, 399) 83
- (short) 0xcfaf, // (12, 4015) 84
- (short) 0xc06f, // (12, 111) 85
- (short) 0xc86f, // (12, 2159) 86
- (short) 0xb7cf, // (11, 1999) 87
- (short) 0xc46f, // (12, 1135) 88
- (short) 0xcc6f, // (12, 3183) 89
- (short) 0xc26f, // (12, 623) 90
- (short) 0xca6f, // (12, 2671) 91
- (short) 0xc66f, // (12, 1647) 92
- (short) 0xce6f, // (12, 3695) 93
- (short) 0xc16f, // (12, 367) 94
- (short) 0xc96f, // (12, 2415) 95
- (short) 0xc56f, // (12, 1391) 96
- (short) 0xcd6f, // (12, 3439) 97
- (short) 0xc36f, // (12, 879) 98
- (short) 0xb02f, // (11, 47) 99
- (short) 0xcb6f, // (12, 2927) 100
- (short) 0xc76f, // (12, 1903) 101
- (short) 0xcf6f, // (12, 3951) 102
- (short) 0xc0ef, // (12, 239) 103
- (short) 0xc8ef, // (12, 2287) 104
- (short) 0xc4ef, // (12, 1263) 105
- (short) 0xccef, // (12, 3311) 106
- (short) 0xc2ef, // (12, 751) 107
- (short) 0xcaef, // (12, 2799) 108
- (short) 0xc6ef, // (12, 1775) 109
- (short) 0xceef, // (12, 3823) 110
- (short) 0xc1ef, // (12, 495) 111
- (short) 0xc9ef, // (12, 2543) 112
- (short) 0xc5ef, // (12, 1519) 113
- (short) 0xcdef, // (12, 3567) 114
- (short) 0xc3ef, // (12, 1007) 115
- (short) 0xcbef, // (12, 3055) 116
- (short) 0xc7ef, // (12, 2031) 117
- (short) 0xcfef, // (12, 4079) 118
- (short) 0xc01f, // (12, 31) 119
- (short) 0xc81f, // (12, 2079) 120
- (short) 0xc41f, // (12, 1055) 121
- (short) 0xcc1f, // (12, 3103) 122
- (short) 0xc21f, // (12, 543) 123
- (short) 0xca1f, // (12, 2591) 124
- (short) 0xc61f, // (12, 1567) 125
- (short) 0xce1f, // (12, 3615) 126
- (short) 0xc11f, // (12, 287) 127
- (short) 0xc91f, // (12, 2335) 128
- (short) 0xa38f, // (10, 911) 129
- (short) 0xb42f, // (11, 1071) 130
- (short) 0x9077, // ( 9, 119) 131
- (short) 0xc51f, // (12, 1311) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xcd1f, // (12, 3359) 134
- (short) 0x9177, // ( 9, 375) 135
- (short) 0xc31f, // (12, 799) 136
- (short) 0xb22f, // (11, 559) 137
- (short) 0xcb1f, // (12, 2847) 138
- (short) 0xa24f, // (10, 591) 139
- (short) 0xc71f, // (12, 1823) 140
- (short) 0xcf1f, // (12, 3871) 141
- (short) 0xc09f, // (12, 159) 142
- (short) 0xb62f, // (11, 1583) 143
- (short) 0xc89f, // (12, 2207) 144
- (short) 0xc49f, // (12, 1183) 145
- (short) 0xcc9f, // (12, 3231) 146
- (short) 0xb12f, // (11, 303) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 8 of 22) (steady 8 of 16) (phase = 0.531250000 = 17.0 / 32.0)
- // entropy: 4.4505873338397474726
- // avg_length: 4.5270058771550303334; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x7033, // ( 7, 51) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x601d, // ( 6, 29) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x7073, // ( 7, 115) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x603d, // ( 6, 61) 6
- (short) 0x3002, // ( 3, 2) 7
- (short) 0x807b, // ( 8, 123) 8
- (short) 0x5005, // ( 5, 5) 9
- (short) 0x700b, // ( 7, 11) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0x9097, // ( 9, 151) 12
- (short) 0x5015, // ( 5, 21) 13
- (short) 0x80fb, // ( 8, 251) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xa0f7, // (10, 247) 16
- (short) 0x6003, // ( 6, 3) 17
- (short) 0x8007, // ( 8, 7) 18
- (short) 0x500d, // ( 5, 13) 19
- (short) 0xa2f7, // (10, 759) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0x9197, // ( 9, 407) 22
- (short) 0x6023, // ( 6, 35) 23
- (short) 0xb34f, // (11, 847) 24
- (short) 0x8087, // ( 8, 135) 25
- (short) 0xa1f7, // (10, 503) 26
- (short) 0x702b, // ( 7, 43) 27
- (short) 0xb74f, // (11, 1871) 28
- (short) 0x8047, // ( 8, 71) 29
- (short) 0xa3f7, // (10, 1015) 30
- (short) 0x706b, // ( 7, 107) 31
- (short) 0xb0cf, // (11, 207) 32
- (short) 0x701b, // ( 7, 27) 33
- (short) 0x9057, // ( 9, 87) 34
- (short) 0x6013, // ( 6, 19) 35
- (short) 0xb4cf, // (11, 1231) 36
- (short) 0x80c7, // ( 8, 199) 37
- (short) 0xa00f, // (10, 15) 38
- (short) 0x705b, // ( 7, 91) 39
- (short) 0xc72f, // (12, 1839) 40
- (short) 0x9157, // ( 9, 343) 41
- (short) 0xb2cf, // (11, 719) 42
- (short) 0x8027, // ( 8, 39) 43
- (short) 0xcf2f, // (12, 3887) 44
- (short) 0x90d7, // ( 9, 215) 45
- (short) 0xb6cf, // (11, 1743) 46
- (short) 0x80a7, // ( 8, 167) 47
- (short) 0xc0af, // (12, 175) 48
- (short) 0xa20f, // (10, 527) 49
- (short) 0xc8af, // (12, 2223) 50
- (short) 0x91d7, // ( 9, 471) 51
- (short) 0xc4af, // (12, 1199) 52
- (short) 0xa10f, // (10, 271) 53
- (short) 0xccaf, // (12, 3247) 54
- (short) 0x9037, // ( 9, 55) 55
- (short) 0xc2af, // (12, 687) 56
- (short) 0xcaaf, // (12, 2735) 57
- (short) 0xc6af, // (12, 1711) 58
- (short) 0xb1cf, // (11, 463) 59
- (short) 0xceaf, // (12, 3759) 60
- (short) 0xc1af, // (12, 431) 61
- (short) 0xc9af, // (12, 2479) 62
- (short) 0xb5cf, // (11, 1487) 63
- (short) 0xc5af, // (12, 1455) 64
- (short) 0x8067, // ( 8, 103) 65
- (short) 0xa30f, // (10, 783) 66
- (short) 0x703b, // ( 7, 59) 67
- (short) 0xcdaf, // (12, 3503) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xb3cf, // (11, 975) 70
- (short) 0x80e7, // ( 8, 231) 71
- (short) 0xc3af, // (12, 943) 72
- (short) 0xa08f, // (10, 143) 73
- (short) 0xcbaf, // (12, 2991) 74
- (short) 0x90b7, // ( 9, 183) 75
- (short) 0xc7af, // (12, 1967) 76
- (short) 0xa28f, // (10, 655) 77
- (short) 0xcfaf, // (12, 4015) 78
- (short) 0x91b7, // ( 9, 439) 79
- (short) 0xc06f, // (12, 111) 80
- (short) 0xb7cf, // (11, 1999) 81
- (short) 0xc86f, // (12, 2159) 82
- (short) 0xa18f, // (10, 399) 83
- (short) 0xc46f, // (12, 1135) 84
- (short) 0xb02f, // (11, 47) 85
- (short) 0xcc6f, // (12, 3183) 86
- (short) 0xa38f, // (10, 911) 87
- (short) 0xc26f, // (12, 623) 88
- (short) 0xca6f, // (12, 2671) 89
- (short) 0xc66f, // (12, 1647) 90
- (short) 0xce6f, // (12, 3695) 91
- (short) 0xc16f, // (12, 367) 92
- (short) 0xc96f, // (12, 2415) 93
- (short) 0xc56f, // (12, 1391) 94
- (short) 0xcd6f, // (12, 3439) 95
- (short) 0xc36f, // (12, 879) 96
- (short) 0xcb6f, // (12, 2927) 97
- (short) 0xc76f, // (12, 1903) 98
- (short) 0xb42f, // (11, 1071) 99
- (short) 0xcf6f, // (12, 3951) 100
- (short) 0xc0ef, // (12, 239) 101
- (short) 0xc8ef, // (12, 2287) 102
- (short) 0xb22f, // (11, 559) 103
- (short) 0xc4ef, // (12, 1263) 104
- (short) 0xccef, // (12, 3311) 105
- (short) 0xc2ef, // (12, 751) 106
- (short) 0xcaef, // (12, 2799) 107
- (short) 0xc6ef, // (12, 1775) 108
- (short) 0xceef, // (12, 3823) 109
- (short) 0xc1ef, // (12, 495) 110
- (short) 0xc9ef, // (12, 2543) 111
- (short) 0xc5ef, // (12, 1519) 112
- (short) 0xcdef, // (12, 3567) 113
- (short) 0xc3ef, // (12, 1007) 114
- (short) 0xcbef, // (12, 3055) 115
- (short) 0xc7ef, // (12, 2031) 116
- (short) 0xcfef, // (12, 4079) 117
- (short) 0xc01f, // (12, 31) 118
- (short) 0xc81f, // (12, 2079) 119
- (short) 0xc41f, // (12, 1055) 120
- (short) 0xcc1f, // (12, 3103) 121
- (short) 0xc21f, // (12, 543) 122
- (short) 0xca1f, // (12, 2591) 123
- (short) 0xc61f, // (12, 1567) 124
- (short) 0xce1f, // (12, 3615) 125
- (short) 0xc11f, // (12, 287) 126
- (short) 0xc91f, // (12, 2335) 127
- (short) 0xc51f, // (12, 1311) 128
- (short) 0x9077, // ( 9, 119) 129
- (short) 0xcd1f, // (12, 3359) 130
- (short) 0x8017, // ( 8, 23) 131
- (short) 0xc31f, // (12, 799) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xcb1f, // (12, 2847) 134
- (short) 0x9177, // ( 9, 375) 135
- (short) 0xc71f, // (12, 1823) 136
- (short) 0xb62f, // (11, 1583) 137
- (short) 0xcf1f, // (12, 3871) 138
- (short) 0xa24f, // (10, 591) 139
- (short) 0xc09f, // (12, 159) 140
- (short) 0xb12f, // (11, 303) 141
- (short) 0xc89f, // (12, 2207) 142
- (short) 0xa14f, // (10, 335) 143
- (short) 0xc49f, // (12, 1183) 144
- (short) 0xcc9f, // (12, 3231) 145
- (short) 0xc29f, // (12, 671) 146
- (short) 0xb52f, // (11, 1327) 147
- (short) 0xca9f, // (12, 2719) 148
- (short) 0xc69f, // (12, 1695) 149
- (short) 0xce9f, // (12, 3743) 150
- (short) 0xb32f, // (11, 815) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 9 of 22) (steady 9 of 16) (phase = 0.593750000 = 19.0 / 32.0)
- // entropy: 4.4575203029748040606
- // avg_length: 4.5315465600684730063; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x7033, // ( 7, 51) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x601d, // ( 6, 29) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x7073, // ( 7, 115) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x603d, // ( 6, 61) 6
- (short) 0x3002, // ( 3, 2) 7
- (short) 0x9097, // ( 9, 151) 8
- (short) 0x5005, // ( 5, 5) 9
- (short) 0x700b, // ( 7, 11) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0x9197, // ( 9, 407) 12
- (short) 0x6003, // ( 6, 3) 13
- (short) 0x807b, // ( 8, 123) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xa0f7, // (10, 247) 16
- (short) 0x6023, // ( 6, 35) 17
- (short) 0x80fb, // ( 8, 251) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xa2f7, // (10, 759) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0x9057, // ( 9, 87) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xb34f, // (11, 847) 24
- (short) 0x8007, // ( 8, 7) 25
- (short) 0xa1f7, // (10, 503) 26
- (short) 0x702b, // ( 7, 43) 27
- (short) 0xc72f, // (12, 1839) 28
- (short) 0x8087, // ( 8, 135) 29
- (short) 0xa3f7, // (10, 1015) 30
- (short) 0x706b, // ( 7, 107) 31
- (short) 0xb74f, // (11, 1871) 32
- (short) 0x701b, // ( 7, 27) 33
- (short) 0x9157, // ( 9, 343) 34
- (short) 0x6013, // ( 6, 19) 35
- (short) 0xb0cf, // (11, 207) 36
- (short) 0x8047, // ( 8, 71) 37
- (short) 0xa00f, // (10, 15) 38
- (short) 0x705b, // ( 7, 91) 39
- (short) 0xcf2f, // (12, 3887) 40
- (short) 0x90d7, // ( 9, 215) 41
- (short) 0xb4cf, // (11, 1231) 42
- (short) 0x80c7, // ( 8, 199) 43
- (short) 0xc0af, // (12, 175) 44
- (short) 0x91d7, // ( 9, 471) 45
- (short) 0xb2cf, // (11, 719) 46
- (short) 0x8027, // ( 8, 39) 47
- (short) 0xc8af, // (12, 2223) 48
- (short) 0xa20f, // (10, 527) 49
- (short) 0xc4af, // (12, 1199) 50
- (short) 0x9037, // ( 9, 55) 51
- (short) 0xccaf, // (12, 3247) 52
- (short) 0xa10f, // (10, 271) 53
- (short) 0xc2af, // (12, 687) 54
- (short) 0x9137, // ( 9, 311) 55
- (short) 0xcaaf, // (12, 2735) 56
- (short) 0xc6af, // (12, 1711) 57
- (short) 0xceaf, // (12, 3759) 58
- (short) 0xa30f, // (10, 783) 59
- (short) 0xc1af, // (12, 431) 60
- (short) 0xc9af, // (12, 2479) 61
- (short) 0xc5af, // (12, 1455) 62
- (short) 0xb6cf, // (11, 1743) 63
- (short) 0xcdaf, // (12, 3503) 64
- (short) 0x80a7, // ( 8, 167) 65
- (short) 0xb1cf, // (11, 463) 66
- (short) 0x703b, // ( 7, 59) 67
- (short) 0xc3af, // (12, 943) 68
- (short) 0x90b7, // ( 9, 183) 69
- (short) 0xb5cf, // (11, 1487) 70
- (short) 0x8067, // ( 8, 103) 71
- (short) 0xcbaf, // (12, 2991) 72
- (short) 0xa08f, // (10, 143) 73
- (short) 0xc7af, // (12, 1967) 74
- (short) 0x91b7, // ( 9, 439) 75
- (short) 0xcfaf, // (12, 4015) 76
- (short) 0xa28f, // (10, 655) 77
- (short) 0xc06f, // (12, 111) 78
- (short) 0x9077, // ( 9, 119) 79
- (short) 0xc86f, // (12, 2159) 80
- (short) 0xb3cf, // (11, 975) 81
- (short) 0xc46f, // (12, 1135) 82
- (short) 0xa18f, // (10, 399) 83
- (short) 0xcc6f, // (12, 3183) 84
- (short) 0xb7cf, // (11, 1999) 85
- (short) 0xc26f, // (12, 623) 86
- (short) 0xa38f, // (10, 911) 87
- (short) 0xca6f, // (12, 2671) 88
- (short) 0xc66f, // (12, 1647) 89
- (short) 0xce6f, // (12, 3695) 90
- (short) 0xb02f, // (11, 47) 91
- (short) 0xc16f, // (12, 367) 92
- (short) 0xc96f, // (12, 2415) 93
- (short) 0xc56f, // (12, 1391) 94
- (short) 0xcd6f, // (12, 3439) 95
- (short) 0xc36f, // (12, 879) 96
- (short) 0xcb6f, // (12, 2927) 97
- (short) 0xc76f, // (12, 1903) 98
- (short) 0xb42f, // (11, 1071) 99
- (short) 0xcf6f, // (12, 3951) 100
- (short) 0xc0ef, // (12, 239) 101
- (short) 0xc8ef, // (12, 2287) 102
- (short) 0xb22f, // (11, 559) 103
- (short) 0xc4ef, // (12, 1263) 104
- (short) 0xccef, // (12, 3311) 105
- (short) 0xc2ef, // (12, 751) 106
- (short) 0xcaef, // (12, 2799) 107
- (short) 0xc6ef, // (12, 1775) 108
- (short) 0xceef, // (12, 3823) 109
- (short) 0xc1ef, // (12, 495) 110
- (short) 0xc9ef, // (12, 2543) 111
- (short) 0xc5ef, // (12, 1519) 112
- (short) 0xcdef, // (12, 3567) 113
- (short) 0xc3ef, // (12, 1007) 114
- (short) 0xcbef, // (12, 3055) 115
- (short) 0xc7ef, // (12, 2031) 116
- (short) 0xcfef, // (12, 4079) 117
- (short) 0xc01f, // (12, 31) 118
- (short) 0xc81f, // (12, 2079) 119
- (short) 0xc41f, // (12, 1055) 120
- (short) 0xcc1f, // (12, 3103) 121
- (short) 0xc21f, // (12, 543) 122
- (short) 0xca1f, // (12, 2591) 123
- (short) 0xc61f, // (12, 1567) 124
- (short) 0xce1f, // (12, 3615) 125
- (short) 0xc11f, // (12, 287) 126
- (short) 0xc91f, // (12, 2335) 127
- (short) 0xc51f, // (12, 1311) 128
- (short) 0x9177, // ( 9, 375) 129
- (short) 0xcd1f, // (12, 3359) 130
- (short) 0x80e7, // ( 8, 231) 131
- (short) 0xc31f, // (12, 799) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xcb1f, // (12, 2847) 134
- (short) 0x8017, // ( 8, 23) 135
- (short) 0xc71f, // (12, 1823) 136
- (short) 0xb62f, // (11, 1583) 137
- (short) 0xcf1f, // (12, 3871) 138
- (short) 0xa24f, // (10, 591) 139
- (short) 0xc09f, // (12, 159) 140
- (short) 0xb12f, // (11, 303) 141
- (short) 0xc89f, // (12, 2207) 142
- (short) 0xa14f, // (10, 335) 143
- (short) 0xc49f, // (12, 1183) 144
- (short) 0xcc9f, // (12, 3231) 145
- (short) 0xc29f, // (12, 671) 146
- (short) 0xb52f, // (11, 1327) 147
- (short) 0xca9f, // (12, 2719) 148
- (short) 0xc69f, // (12, 1695) 149
- (short) 0xce9f, // (12, 3743) 150
- (short) 0xb32f, // (11, 815) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 10 of 22) (steady 10 of 16) (phase = 0.656250000 = 21.0 / 32.0)
- // entropy: 4.4626765653088611430
- // avg_length: 4.5373141251902122661; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x700b, // ( 7, 11) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x601d, // ( 6, 29) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x807b, // ( 8, 123) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x603d, // ( 6, 61) 6
- (short) 0x3002, // ( 3, 2) 7
- (short) 0x9017, // ( 9, 23) 8
- (short) 0x5005, // ( 5, 5) 9
- (short) 0x704b, // ( 7, 75) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0x9117, // ( 9, 279) 12
- (short) 0x6003, // ( 6, 3) 13
- (short) 0x80fb, // ( 8, 251) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xa177, // (10, 375) 16
- (short) 0x6023, // ( 6, 35) 17
- (short) 0x9097, // ( 9, 151) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xa377, // (10, 887) 20
- (short) 0x702b, // ( 7, 43) 21
- (short) 0x9197, // ( 9, 407) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xb34f, // (11, 847) 24
- (short) 0x8007, // ( 8, 7) 25
- (short) 0xa0f7, // (10, 247) 26
- (short) 0x706b, // ( 7, 107) 27
- (short) 0xc0af, // (12, 175) 28
- (short) 0x8087, // ( 8, 135) 29
- (short) 0xa2f7, // (10, 759) 30
- (short) 0x701b, // ( 7, 27) 31
- (short) 0xb74f, // (11, 1871) 32
- (short) 0x8047, // ( 8, 71) 33
- (short) 0xa1f7, // (10, 503) 34
- (short) 0x6013, // ( 6, 19) 35
- (short) 0xb0cf, // (11, 207) 36
- (short) 0x80c7, // ( 8, 199) 37
- (short) 0xa3f7, // (10, 1015) 38
- (short) 0x6033, // ( 6, 51) 39
- (short) 0xc8af, // (12, 2223) 40
- (short) 0x9057, // ( 9, 87) 41
- (short) 0xb4cf, // (11, 1231) 42
- (short) 0x8027, // ( 8, 39) 43
- (short) 0xc4af, // (12, 1199) 44
- (short) 0x9157, // ( 9, 343) 45
- (short) 0xb2cf, // (11, 719) 46
- (short) 0x80a7, // ( 8, 167) 47
- (short) 0xccaf, // (12, 3247) 48
- (short) 0xa00f, // (10, 15) 49
- (short) 0xc2af, // (12, 687) 50
- (short) 0x90d7, // ( 9, 215) 51
- (short) 0xcaaf, // (12, 2735) 52
- (short) 0xa20f, // (10, 527) 53
- (short) 0xc6af, // (12, 1711) 54
- (short) 0x91d7, // ( 9, 471) 55
- (short) 0xceaf, // (12, 3759) 56
- (short) 0xb6cf, // (11, 1743) 57
- (short) 0xc1af, // (12, 431) 58
- (short) 0xa10f, // (10, 271) 59
- (short) 0xc9af, // (12, 2479) 60
- (short) 0xc5af, // (12, 1455) 61
- (short) 0xcdaf, // (12, 3503) 62
- (short) 0xa30f, // (10, 783) 63
- (short) 0xc3af, // (12, 943) 64
- (short) 0x9037, // ( 9, 55) 65
- (short) 0xb1cf, // (11, 463) 66
- (short) 0x705b, // ( 7, 91) 67
- (short) 0xcbaf, // (12, 2991) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xb5cf, // (11, 1487) 70
- (short) 0x703b, // ( 7, 59) 71
- (short) 0xc7af, // (12, 1967) 72
- (short) 0xa08f, // (10, 143) 73
- (short) 0xcfaf, // (12, 4015) 74
- (short) 0x90b7, // ( 9, 183) 75
- (short) 0xc06f, // (12, 111) 76
- (short) 0xa28f, // (10, 655) 77
- (short) 0xc86f, // (12, 2159) 78
- (short) 0x91b7, // ( 9, 439) 79
- (short) 0xc46f, // (12, 1135) 80
- (short) 0xb3cf, // (11, 975) 81
- (short) 0xcc6f, // (12, 3183) 82
- (short) 0xa18f, // (10, 399) 83
- (short) 0xc26f, // (12, 623) 84
- (short) 0xb7cf, // (11, 1999) 85
- (short) 0xca6f, // (12, 2671) 86
- (short) 0xa38f, // (10, 911) 87
- (short) 0xc66f, // (12, 1647) 88
- (short) 0xce6f, // (12, 3695) 89
- (short) 0xc16f, // (12, 367) 90
- (short) 0xb02f, // (11, 47) 91
- (short) 0xc96f, // (12, 2415) 92
- (short) 0xc56f, // (12, 1391) 93
- (short) 0xcd6f, // (12, 3439) 94
- (short) 0xb42f, // (11, 1071) 95
- (short) 0xc36f, // (12, 879) 96
- (short) 0xcb6f, // (12, 2927) 97
- (short) 0xc76f, // (12, 1903) 98
- (short) 0xb22f, // (11, 559) 99
- (short) 0xcf6f, // (12, 3951) 100
- (short) 0xc0ef, // (12, 239) 101
- (short) 0xc8ef, // (12, 2287) 102
- (short) 0xb62f, // (11, 1583) 103
- (short) 0xc4ef, // (12, 1263) 104
- (short) 0xccef, // (12, 3311) 105
- (short) 0xc2ef, // (12, 751) 106
- (short) 0xcaef, // (12, 2799) 107
- (short) 0xc6ef, // (12, 1775) 108
- (short) 0xceef, // (12, 3823) 109
- (short) 0xc1ef, // (12, 495) 110
- (short) 0xc9ef, // (12, 2543) 111
- (short) 0xc5ef, // (12, 1519) 112
- (short) 0xcdef, // (12, 3567) 113
- (short) 0xc3ef, // (12, 1007) 114
- (short) 0xcbef, // (12, 3055) 115
- (short) 0xc7ef, // (12, 2031) 116
- (short) 0xcfef, // (12, 4079) 117
- (short) 0xc01f, // (12, 31) 118
- (short) 0xc81f, // (12, 2079) 119
- (short) 0xc41f, // (12, 1055) 120
- (short) 0xcc1f, // (12, 3103) 121
- (short) 0xc21f, // (12, 543) 122
- (short) 0xca1f, // (12, 2591) 123
- (short) 0xc61f, // (12, 1567) 124
- (short) 0xce1f, // (12, 3615) 125
- (short) 0xc11f, // (12, 287) 126
- (short) 0xc91f, // (12, 2335) 127
- (short) 0xc51f, // (12, 1311) 128
- (short) 0x9077, // ( 9, 119) 129
- (short) 0xcd1f, // (12, 3359) 130
- (short) 0x8067, // ( 8, 103) 131
- (short) 0xc31f, // (12, 799) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xcb1f, // (12, 2847) 134
- (short) 0x80e7, // ( 8, 231) 135
- (short) 0xc71f, // (12, 1823) 136
- (short) 0xb12f, // (11, 303) 137
- (short) 0xcf1f, // (12, 3871) 138
- (short) 0xa24f, // (10, 591) 139
- (short) 0xc09f, // (12, 159) 140
- (short) 0xb52f, // (11, 1327) 141
- (short) 0xc89f, // (12, 2207) 142
- (short) 0xa14f, // (10, 335) 143
- (short) 0xc49f, // (12, 1183) 144
- (short) 0xcc9f, // (12, 3231) 145
- (short) 0xc29f, // (12, 671) 146
- (short) 0xb32f, // (11, 815) 147
- (short) 0xca9f, // (12, 2719) 148
- (short) 0xc69f, // (12, 1695) 149
- (short) 0xce9f, // (12, 3743) 150
- (short) 0xb72f, // (11, 1839) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 11 of 22) (steady 11 of 16) (phase = 0.718750000 = 23.0 / 32.0)
- // entropy: 4.4661524304421691411
- // avg_length: 4.5443750890419041255; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x803b, // ( 8, 59) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x601d, // ( 6, 29) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x80bb, // ( 8, 187) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x603d, // ( 6, 61) 6
- (short) 0x3002, // ( 3, 2) 7
- (short) 0x9017, // ( 9, 23) 8
- (short) 0x5005, // ( 5, 5) 9
- (short) 0x807b, // ( 8, 123) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0x9117, // ( 9, 279) 12
- (short) 0x6003, // ( 6, 3) 13
- (short) 0x80fb, // ( 8, 251) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xa177, // (10, 375) 16
- (short) 0x6023, // ( 6, 35) 17
- (short) 0x9097, // ( 9, 151) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xa377, // (10, 887) 20
- (short) 0x702b, // ( 7, 43) 21
- (short) 0x9197, // ( 9, 407) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xb34f, // (11, 847) 24
- (short) 0x8007, // ( 8, 7) 25
- (short) 0xa0f7, // (10, 247) 26
- (short) 0x6013, // ( 6, 19) 27
- (short) 0xc0af, // (12, 175) 28
- (short) 0x8087, // ( 8, 135) 29
- (short) 0xa2f7, // (10, 759) 30
- (short) 0x706b, // ( 7, 107) 31
- (short) 0xb74f, // (11, 1871) 32
- (short) 0x8047, // ( 8, 71) 33
- (short) 0xa1f7, // (10, 503) 34
- (short) 0x6033, // ( 6, 51) 35
- (short) 0xb0cf, // (11, 207) 36
- (short) 0x80c7, // ( 8, 199) 37
- (short) 0xa3f7, // (10, 1015) 38
- (short) 0x600b, // ( 6, 11) 39
- (short) 0xc8af, // (12, 2223) 40
- (short) 0x9057, // ( 9, 87) 41
- (short) 0xb4cf, // (11, 1231) 42
- (short) 0x8027, // ( 8, 39) 43
- (short) 0xc4af, // (12, 1199) 44
- (short) 0x9157, // ( 9, 343) 45
- (short) 0xb2cf, // (11, 719) 46
- (short) 0x80a7, // ( 8, 167) 47
- (short) 0xccaf, // (12, 3247) 48
- (short) 0xa00f, // (10, 15) 49
- (short) 0xc2af, // (12, 687) 50
- (short) 0x90d7, // ( 9, 215) 51
- (short) 0xcaaf, // (12, 2735) 52
- (short) 0xa20f, // (10, 527) 53
- (short) 0xc6af, // (12, 1711) 54
- (short) 0x91d7, // ( 9, 471) 55
- (short) 0xceaf, // (12, 3759) 56
- (short) 0xb6cf, // (11, 1743) 57
- (short) 0xc1af, // (12, 431) 58
- (short) 0xa10f, // (10, 271) 59
- (short) 0xc9af, // (12, 2479) 60
- (short) 0xc5af, // (12, 1455) 61
- (short) 0xcdaf, // (12, 3503) 62
- (short) 0xa30f, // (10, 783) 63
- (short) 0xc3af, // (12, 943) 64
- (short) 0x9037, // ( 9, 55) 65
- (short) 0xb1cf, // (11, 463) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xcbaf, // (12, 2991) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xb5cf, // (11, 1487) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc7af, // (12, 1967) 72
- (short) 0xa08f, // (10, 143) 73
- (short) 0xcfaf, // (12, 4015) 74
- (short) 0x90b7, // ( 9, 183) 75
- (short) 0xc06f, // (12, 111) 76
- (short) 0xa28f, // (10, 655) 77
- (short) 0xc86f, // (12, 2159) 78
- (short) 0x91b7, // ( 9, 439) 79
- (short) 0xc46f, // (12, 1135) 80
- (short) 0xb3cf, // (11, 975) 81
- (short) 0xcc6f, // (12, 3183) 82
- (short) 0xa18f, // (10, 399) 83
- (short) 0xc26f, // (12, 623) 84
- (short) 0xb7cf, // (11, 1999) 85
- (short) 0xca6f, // (12, 2671) 86
- (short) 0xa38f, // (10, 911) 87
- (short) 0xc66f, // (12, 1647) 88
- (short) 0xce6f, // (12, 3695) 89
- (short) 0xc16f, // (12, 367) 90
- (short) 0xb02f, // (11, 47) 91
- (short) 0xc96f, // (12, 2415) 92
- (short) 0xc56f, // (12, 1391) 93
- (short) 0xcd6f, // (12, 3439) 94
- (short) 0xb42f, // (11, 1071) 95
- (short) 0xc36f, // (12, 879) 96
- (short) 0xcb6f, // (12, 2927) 97
- (short) 0xc76f, // (12, 1903) 98
- (short) 0xb22f, // (11, 559) 99
- (short) 0xcf6f, // (12, 3951) 100
- (short) 0xc0ef, // (12, 239) 101
- (short) 0xc8ef, // (12, 2287) 102
- (short) 0xb62f, // (11, 1583) 103
- (short) 0xc4ef, // (12, 1263) 104
- (short) 0xccef, // (12, 3311) 105
- (short) 0xc2ef, // (12, 751) 106
- (short) 0xcaef, // (12, 2799) 107
- (short) 0xc6ef, // (12, 1775) 108
- (short) 0xceef, // (12, 3823) 109
- (short) 0xc1ef, // (12, 495) 110
- (short) 0xc9ef, // (12, 2543) 111
- (short) 0xc5ef, // (12, 1519) 112
- (short) 0xcdef, // (12, 3567) 113
- (short) 0xc3ef, // (12, 1007) 114
- (short) 0xcbef, // (12, 3055) 115
- (short) 0xc7ef, // (12, 2031) 116
- (short) 0xcfef, // (12, 4079) 117
- (short) 0xc01f, // (12, 31) 118
- (short) 0xc81f, // (12, 2079) 119
- (short) 0xc41f, // (12, 1055) 120
- (short) 0xcc1f, // (12, 3103) 121
- (short) 0xc21f, // (12, 543) 122
- (short) 0xca1f, // (12, 2591) 123
- (short) 0xc61f, // (12, 1567) 124
- (short) 0xce1f, // (12, 3615) 125
- (short) 0xc11f, // (12, 287) 126
- (short) 0xc91f, // (12, 2335) 127
- (short) 0xc51f, // (12, 1311) 128
- (short) 0xa04f, // (10, 79) 129
- (short) 0xcd1f, // (12, 3359) 130
- (short) 0x8067, // ( 8, 103) 131
- (short) 0xc31f, // (12, 799) 132
- (short) 0xa24f, // (10, 591) 133
- (short) 0xcb1f, // (12, 2847) 134
- (short) 0x80e7, // ( 8, 231) 135
- (short) 0xc71f, // (12, 1823) 136
- (short) 0xb12f, // (11, 303) 137
- (short) 0xcf1f, // (12, 3871) 138
- (short) 0x9077, // ( 9, 119) 139
- (short) 0xc09f, // (12, 159) 140
- (short) 0xb52f, // (11, 1327) 141
- (short) 0xc89f, // (12, 2207) 142
- (short) 0xa14f, // (10, 335) 143
- (short) 0xc49f, // (12, 1183) 144
- (short) 0xcc9f, // (12, 3231) 145
- (short) 0xc29f, // (12, 671) 146
- (short) 0xb32f, // (11, 815) 147
- (short) 0xca9f, // (12, 2719) 148
- (short) 0xc69f, // (12, 1695) 149
- (short) 0xce9f, // (12, 3743) 150
- (short) 0xb72f, // (11, 1839) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 12 of 22) (steady 12 of 16) (phase = 0.781250000 = 25.0 / 32.0)
- // entropy: 4.4680486273043946710
- // avg_length: 4.5521643785256946657; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x807b, // ( 8, 123) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x601d, // ( 6, 29) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x80fb, // ( 8, 251) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x700b, // ( 7, 11) 6
- (short) 0x3002, // ( 3, 2) 7
- (short) 0x9097, // ( 9, 151) 8
- (short) 0x5005, // ( 5, 5) 9
- (short) 0x8007, // ( 8, 7) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0x9197, // ( 9, 407) 12
- (short) 0x603d, // ( 6, 61) 13
- (short) 0x8087, // ( 8, 135) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xa177, // (10, 375) 16
- (short) 0x704b, // ( 7, 75) 17
- (short) 0x9057, // ( 9, 87) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xb34f, // (11, 847) 20
- (short) 0x702b, // ( 7, 43) 21
- (short) 0x9157, // ( 9, 343) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xc72f, // (12, 1839) 24
- (short) 0x8047, // ( 8, 71) 25
- (short) 0xa377, // (10, 887) 26
- (short) 0x6003, // ( 6, 3) 27
- (short) 0xcf2f, // (12, 3887) 28
- (short) 0x80c7, // ( 8, 199) 29
- (short) 0xa0f7, // (10, 247) 30
- (short) 0x6023, // ( 6, 35) 31
- (short) 0xc0af, // (12, 175) 32
- (short) 0x8027, // ( 8, 39) 33
- (short) 0xa2f7, // (10, 759) 34
- (short) 0x6013, // ( 6, 19) 35
- (short) 0xc8af, // (12, 2223) 36
- (short) 0x80a7, // ( 8, 167) 37
- (short) 0xa1f7, // (10, 503) 38
- (short) 0x6033, // ( 6, 51) 39
- (short) 0xc4af, // (12, 1199) 40
- (short) 0x90d7, // ( 9, 215) 41
- (short) 0xb74f, // (11, 1871) 42
- (short) 0x706b, // ( 7, 107) 43
- (short) 0xccaf, // (12, 3247) 44
- (short) 0x91d7, // ( 9, 471) 45
- (short) 0xb0cf, // (11, 207) 46
- (short) 0x701b, // ( 7, 27) 47
- (short) 0xc2af, // (12, 687) 48
- (short) 0xa3f7, // (10, 1015) 49
- (short) 0xcaaf, // (12, 2735) 50
- (short) 0x9037, // ( 9, 55) 51
- (short) 0xc6af, // (12, 1711) 52
- (short) 0xa00f, // (10, 15) 53
- (short) 0xceaf, // (12, 3759) 54
- (short) 0x9137, // ( 9, 311) 55
- (short) 0xc1af, // (12, 431) 56
- (short) 0xb4cf, // (11, 1231) 57
- (short) 0xc9af, // (12, 2479) 58
- (short) 0xa20f, // (10, 527) 59
- (short) 0xc5af, // (12, 1455) 60
- (short) 0xb2cf, // (11, 719) 61
- (short) 0xcdaf, // (12, 3503) 62
- (short) 0xa10f, // (10, 271) 63
- (short) 0xc3af, // (12, 943) 64
- (short) 0x90b7, // ( 9, 183) 65
- (short) 0xb6cf, // (11, 1743) 66
- (short) 0x705b, // ( 7, 91) 67
- (short) 0xcbaf, // (12, 2991) 68
- (short) 0x91b7, // ( 9, 439) 69
- (short) 0xb1cf, // (11, 463) 70
- (short) 0x703b, // ( 7, 59) 71
- (short) 0xc7af, // (12, 1967) 72
- (short) 0xa30f, // (10, 783) 73
- (short) 0xcfaf, // (12, 4015) 74
- (short) 0x8067, // ( 8, 103) 75
- (short) 0xc06f, // (12, 111) 76
- (short) 0xa08f, // (10, 143) 77
- (short) 0xc86f, // (12, 2159) 78
- (short) 0x9077, // ( 9, 119) 79
- (short) 0xc46f, // (12, 1135) 80
- (short) 0xb5cf, // (11, 1487) 81
- (short) 0xcc6f, // (12, 3183) 82
- (short) 0xa28f, // (10, 655) 83
- (short) 0xc26f, // (12, 623) 84
- (short) 0xb3cf, // (11, 975) 85
- (short) 0xca6f, // (12, 2671) 86
- (short) 0xa18f, // (10, 399) 87
- (short) 0xc66f, // (12, 1647) 88
- (short) 0xce6f, // (12, 3695) 89
- (short) 0xc16f, // (12, 367) 90
- (short) 0xb7cf, // (11, 1999) 91
- (short) 0xc96f, // (12, 2415) 92
- (short) 0xc56f, // (12, 1391) 93
- (short) 0xcd6f, // (12, 3439) 94
- (short) 0xb02f, // (11, 47) 95
- (short) 0xc36f, // (12, 879) 96
- (short) 0xcb6f, // (12, 2927) 97
- (short) 0xc76f, // (12, 1903) 98
- (short) 0xb42f, // (11, 1071) 99
- (short) 0xcf6f, // (12, 3951) 100
- (short) 0xc0ef, // (12, 239) 101
- (short) 0xc8ef, // (12, 2287) 102
- (short) 0xb22f, // (11, 559) 103
- (short) 0xc4ef, // (12, 1263) 104
- (short) 0xccef, // (12, 3311) 105
- (short) 0xc2ef, // (12, 751) 106
- (short) 0xcaef, // (12, 2799) 107
- (short) 0xc6ef, // (12, 1775) 108
- (short) 0xceef, // (12, 3823) 109
- (short) 0xc1ef, // (12, 495) 110
- (short) 0xc9ef, // (12, 2543) 111
- (short) 0xc5ef, // (12, 1519) 112
- (short) 0xcdef, // (12, 3567) 113
- (short) 0xc3ef, // (12, 1007) 114
- (short) 0xcbef, // (12, 3055) 115
- (short) 0xc7ef, // (12, 2031) 116
- (short) 0xcfef, // (12, 4079) 117
- (short) 0xc01f, // (12, 31) 118
- (short) 0xc81f, // (12, 2079) 119
- (short) 0xc41f, // (12, 1055) 120
- (short) 0xcc1f, // (12, 3103) 121
- (short) 0xc21f, // (12, 543) 122
- (short) 0xca1f, // (12, 2591) 123
- (short) 0xc61f, // (12, 1567) 124
- (short) 0xce1f, // (12, 3615) 125
- (short) 0xc11f, // (12, 287) 126
- (short) 0xc91f, // (12, 2335) 127
- (short) 0xc51f, // (12, 1311) 128
- (short) 0xa38f, // (10, 911) 129
- (short) 0xcd1f, // (12, 3359) 130
- (short) 0x80e7, // ( 8, 231) 131
- (short) 0xc31f, // (12, 799) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xcb1f, // (12, 2847) 134
- (short) 0x8017, // ( 8, 23) 135
- (short) 0xc71f, // (12, 1823) 136
- (short) 0xb62f, // (11, 1583) 137
- (short) 0xcf1f, // (12, 3871) 138
- (short) 0xa24f, // (10, 591) 139
- (short) 0xc09f, // (12, 159) 140
- (short) 0xb12f, // (11, 303) 141
- (short) 0xc89f, // (12, 2207) 142
- (short) 0xa14f, // (10, 335) 143
- (short) 0xc49f, // (12, 1183) 144
- (short) 0xcc9f, // (12, 3231) 145
- (short) 0xc29f, // (12, 671) 146
- (short) 0xb52f, // (11, 1327) 147
- (short) 0xca9f, // (12, 2719) 148
- (short) 0xc69f, // (12, 1695) 149
- (short) 0xce9f, // (12, 3743) 150
- (short) 0xb32f, // (11, 815) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 13 of 22) (steady 13 of 16) (phase = 0.843750000 = 27.0 / 32.0)
- // entropy: 4.4684687952964843305
- // avg_length: 4.5509169030369793774; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x803b, // ( 8, 59) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x7033, // ( 7, 51) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0x80bb, // ( 8, 187) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x7073, // ( 7, 115) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xa0f7, // (10, 247) 8
- (short) 0x601d, // ( 6, 29) 9
- (short) 0x807b, // ( 8, 123) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xa2f7, // (10, 759) 12
- (short) 0x5005, // ( 5, 5) 13
- (short) 0x80fb, // ( 8, 251) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xb34f, // (11, 847) 16
- (short) 0x700b, // ( 7, 11) 17
- (short) 0x9057, // ( 9, 87) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xb74f, // (11, 1871) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0x9157, // ( 9, 343) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xc72f, // (12, 1839) 24
- (short) 0x8007, // ( 8, 7) 25
- (short) 0xa1f7, // (10, 503) 26
- (short) 0x603d, // ( 6, 61) 27
- (short) 0xcf2f, // (12, 3887) 28
- (short) 0x8087, // ( 8, 135) 29
- (short) 0xa3f7, // (10, 1015) 30
- (short) 0x6003, // ( 6, 3) 31
- (short) 0xc0af, // (12, 175) 32
- (short) 0x8047, // ( 8, 71) 33
- (short) 0xa00f, // (10, 15) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xc8af, // (12, 2223) 36
- (short) 0x80c7, // ( 8, 199) 37
- (short) 0xa20f, // (10, 527) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xc4af, // (12, 1199) 40
- (short) 0x90d7, // ( 9, 215) 41
- (short) 0xb0cf, // (11, 207) 42
- (short) 0x702b, // ( 7, 43) 43
- (short) 0xccaf, // (12, 3247) 44
- (short) 0x91d7, // ( 9, 471) 45
- (short) 0xb4cf, // (11, 1231) 46
- (short) 0x706b, // ( 7, 107) 47
- (short) 0xc2af, // (12, 687) 48
- (short) 0xa10f, // (10, 271) 49
- (short) 0xcaaf, // (12, 2735) 50
- (short) 0x8027, // ( 8, 39) 51
- (short) 0xc6af, // (12, 1711) 52
- (short) 0xa30f, // (10, 783) 53
- (short) 0xceaf, // (12, 3759) 54
- (short) 0x80a7, // ( 8, 167) 55
- (short) 0xc1af, // (12, 431) 56
- (short) 0xb2cf, // (11, 719) 57
- (short) 0xc9af, // (12, 2479) 58
- (short) 0xa08f, // (10, 143) 59
- (short) 0xc5af, // (12, 1455) 60
- (short) 0xb6cf, // (11, 1743) 61
- (short) 0xcdaf, // (12, 3503) 62
- (short) 0xa28f, // (10, 655) 63
- (short) 0xc3af, // (12, 943) 64
- (short) 0x9037, // ( 9, 55) 65
- (short) 0xb1cf, // (11, 463) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xcbaf, // (12, 2991) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xb5cf, // (11, 1487) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc7af, // (12, 1967) 72
- (short) 0xa18f, // (10, 399) 73
- (short) 0xcfaf, // (12, 4015) 74
- (short) 0x8067, // ( 8, 103) 75
- (short) 0xc06f, // (12, 111) 76
- (short) 0xa38f, // (10, 911) 77
- (short) 0xc86f, // (12, 2159) 78
- (short) 0x80e7, // ( 8, 231) 79
- (short) 0xc46f, // (12, 1135) 80
- (short) 0xb3cf, // (11, 975) 81
- (short) 0xcc6f, // (12, 3183) 82
- (short) 0x90b7, // ( 9, 183) 83
- (short) 0xc26f, // (12, 623) 84
- (short) 0xb7cf, // (11, 1999) 85
- (short) 0xca6f, // (12, 2671) 86
- (short) 0x91b7, // ( 9, 439) 87
- (short) 0xc66f, // (12, 1647) 88
- (short) 0xce6f, // (12, 3695) 89
- (short) 0xc16f, // (12, 367) 90
- (short) 0xb02f, // (11, 47) 91
- (short) 0xc96f, // (12, 2415) 92
- (short) 0xc56f, // (12, 1391) 93
- (short) 0xcd6f, // (12, 3439) 94
- (short) 0xb42f, // (11, 1071) 95
- (short) 0xc36f, // (12, 879) 96
- (short) 0xcb6f, // (12, 2927) 97
- (short) 0xc76f, // (12, 1903) 98
- (short) 0xb22f, // (11, 559) 99
- (short) 0xcf6f, // (12, 3951) 100
- (short) 0xc0ef, // (12, 239) 101
- (short) 0xc8ef, // (12, 2287) 102
- (short) 0xb62f, // (11, 1583) 103
- (short) 0xc4ef, // (12, 1263) 104
- (short) 0xccef, // (12, 3311) 105
- (short) 0xc2ef, // (12, 751) 106
- (short) 0xcaef, // (12, 2799) 107
- (short) 0xc6ef, // (12, 1775) 108
- (short) 0xceef, // (12, 3823) 109
- (short) 0xc1ef, // (12, 495) 110
- (short) 0xc9ef, // (12, 2543) 111
- (short) 0xc5ef, // (12, 1519) 112
- (short) 0xcdef, // (12, 3567) 113
- (short) 0xc3ef, // (12, 1007) 114
- (short) 0xcbef, // (12, 3055) 115
- (short) 0xc7ef, // (12, 2031) 116
- (short) 0xcfef, // (12, 4079) 117
- (short) 0xc01f, // (12, 31) 118
- (short) 0xc81f, // (12, 2079) 119
- (short) 0xc41f, // (12, 1055) 120
- (short) 0xcc1f, // (12, 3103) 121
- (short) 0xc21f, // (12, 543) 122
- (short) 0xca1f, // (12, 2591) 123
- (short) 0xc61f, // (12, 1567) 124
- (short) 0xce1f, // (12, 3615) 125
- (short) 0xc11f, // (12, 287) 126
- (short) 0xc91f, // (12, 2335) 127
- (short) 0xc51f, // (12, 1311) 128
- (short) 0xa04f, // (10, 79) 129
- (short) 0xcd1f, // (12, 3359) 130
- (short) 0x8017, // ( 8, 23) 131
- (short) 0xc31f, // (12, 799) 132
- (short) 0xa24f, // (10, 591) 133
- (short) 0xcb1f, // (12, 2847) 134
- (short) 0x8097, // ( 8, 151) 135
- (short) 0xc71f, // (12, 1823) 136
- (short) 0xb12f, // (11, 303) 137
- (short) 0xcf1f, // (12, 3871) 138
- (short) 0x9077, // ( 9, 119) 139
- (short) 0xc09f, // (12, 159) 140
- (short) 0xb52f, // (11, 1327) 141
- (short) 0xc89f, // (12, 2207) 142
- (short) 0x9177, // ( 9, 375) 143
- (short) 0xc49f, // (12, 1183) 144
- (short) 0xcc9f, // (12, 3231) 145
- (short) 0xc29f, // (12, 671) 146
- (short) 0xb32f, // (11, 815) 147
- (short) 0xca9f, // (12, 2719) 148
- (short) 0xc69f, // (12, 1695) 149
- (short) 0xce9f, // (12, 3743) 150
- (short) 0xa14f, // (10, 335) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 14 of 22) (steady 14 of 16) (phase = 0.906250000 = 29.0 / 32.0)
- // entropy: 4.4675179140944036860
- // avg_length: 4.5477235350841240802; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x9017, // ( 9, 23) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x7033, // ( 7, 51) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0x9117, // ( 9, 279) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x7073, // ( 7, 115) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xa177, // (10, 375) 8
- (short) 0x601d, // ( 6, 29) 9
- (short) 0x803b, // ( 8, 59) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xa377, // (10, 887) 12
- (short) 0x5005, // ( 5, 5) 13
- (short) 0x80bb, // ( 8, 187) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xb0cf, // (11, 207) 16
- (short) 0x700b, // ( 7, 11) 17
- (short) 0x9097, // ( 9, 151) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xb4cf, // (11, 1231) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0x9197, // ( 9, 407) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xc4af, // (12, 1199) 24
- (short) 0x807b, // ( 8, 123) 25
- (short) 0xa0f7, // (10, 247) 26
- (short) 0x603d, // ( 6, 61) 27
- (short) 0xccaf, // (12, 3247) 28
- (short) 0x80fb, // ( 8, 251) 29
- (short) 0xa2f7, // (10, 759) 30
- (short) 0x6003, // ( 6, 3) 31
- (short) 0xc2af, // (12, 687) 32
- (short) 0x8007, // ( 8, 7) 33
- (short) 0xa1f7, // (10, 503) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xcaaf, // (12, 2735) 36
- (short) 0x8087, // ( 8, 135) 37
- (short) 0xa3f7, // (10, 1015) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xc6af, // (12, 1711) 40
- (short) 0x9057, // ( 9, 87) 41
- (short) 0xb2cf, // (11, 719) 42
- (short) 0x702b, // ( 7, 43) 43
- (short) 0xceaf, // (12, 3759) 44
- (short) 0x9157, // ( 9, 343) 45
- (short) 0xb6cf, // (11, 1743) 46
- (short) 0x706b, // ( 7, 107) 47
- (short) 0xc1af, // (12, 431) 48
- (short) 0xa00f, // (10, 15) 49
- (short) 0xc9af, // (12, 2479) 50
- (short) 0x8047, // ( 8, 71) 51
- (short) 0xc5af, // (12, 1455) 52
- (short) 0xa20f, // (10, 527) 53
- (short) 0xcdaf, // (12, 3503) 54
- (short) 0x80c7, // ( 8, 199) 55
- (short) 0xc3af, // (12, 943) 56
- (short) 0xb1cf, // (11, 463) 57
- (short) 0xcbaf, // (12, 2991) 58
- (short) 0xa10f, // (10, 271) 59
- (short) 0xc7af, // (12, 1967) 60
- (short) 0xb5cf, // (11, 1487) 61
- (short) 0xcfaf, // (12, 4015) 62
- (short) 0x90d7, // ( 9, 215) 63
- (short) 0xc06f, // (12, 111) 64
- (short) 0x91d7, // ( 9, 471) 65
- (short) 0xb3cf, // (11, 975) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xc86f, // (12, 2159) 68
- (short) 0x9037, // ( 9, 55) 69
- (short) 0xb7cf, // (11, 1999) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc46f, // (12, 1135) 72
- (short) 0xa30f, // (10, 783) 73
- (short) 0xcc6f, // (12, 3183) 74
- (short) 0x8027, // ( 8, 39) 75
- (short) 0xc26f, // (12, 623) 76
- (short) 0xa08f, // (10, 143) 77
- (short) 0xca6f, // (12, 2671) 78
- (short) 0x80a7, // ( 8, 167) 79
- (short) 0xc66f, // (12, 1647) 80
- (short) 0xb02f, // (11, 47) 81
- (short) 0xce6f, // (12, 3695) 82
- (short) 0x9137, // ( 9, 311) 83
- (short) 0xc16f, // (12, 367) 84
- (short) 0xb42f, // (11, 1071) 85
- (short) 0xc96f, // (12, 2415) 86
- (short) 0x90b7, // ( 9, 183) 87
- (short) 0xc56f, // (12, 1391) 88
- (short) 0xcd6f, // (12, 3439) 89
- (short) 0xc36f, // (12, 879) 90
- (short) 0xb22f, // (11, 559) 91
- (short) 0xcb6f, // (12, 2927) 92
- (short) 0xc76f, // (12, 1903) 93
- (short) 0xcf6f, // (12, 3951) 94
- (short) 0xa28f, // (10, 655) 95
- (short) 0xc0ef, // (12, 239) 96
- (short) 0xc8ef, // (12, 2287) 97
- (short) 0xc4ef, // (12, 1263) 98
- (short) 0xa18f, // (10, 399) 99
- (short) 0xccef, // (12, 3311) 100
- (short) 0xc2ef, // (12, 751) 101
- (short) 0xcaef, // (12, 2799) 102
- (short) 0xa38f, // (10, 911) 103
- (short) 0xc6ef, // (12, 1775) 104
- (short) 0xceef, // (12, 3823) 105
- (short) 0xc1ef, // (12, 495) 106
- (short) 0xc9ef, // (12, 2543) 107
- (short) 0xc5ef, // (12, 1519) 108
- (short) 0xcdef, // (12, 3567) 109
- (short) 0xc3ef, // (12, 1007) 110
- (short) 0xb62f, // (11, 1583) 111
- (short) 0xcbef, // (12, 3055) 112
- (short) 0xc7ef, // (12, 2031) 113
- (short) 0xcfef, // (12, 4079) 114
- (short) 0xc01f, // (12, 31) 115
- (short) 0xc81f, // (12, 2079) 116
- (short) 0xc41f, // (12, 1055) 117
- (short) 0xcc1f, // (12, 3103) 118
- (short) 0xc21f, // (12, 543) 119
- (short) 0xca1f, // (12, 2591) 120
- (short) 0xc61f, // (12, 1567) 121
- (short) 0xce1f, // (12, 3615) 122
- (short) 0xc11f, // (12, 287) 123
- (short) 0xc91f, // (12, 2335) 124
- (short) 0xc51f, // (12, 1311) 125
- (short) 0xcd1f, // (12, 3359) 126
- (short) 0xc31f, // (12, 799) 127
- (short) 0xcb1f, // (12, 2847) 128
- (short) 0xa04f, // (10, 79) 129
- (short) 0xc71f, // (12, 1823) 130
- (short) 0x8067, // ( 8, 103) 131
- (short) 0xcf1f, // (12, 3871) 132
- (short) 0xa24f, // (10, 591) 133
- (short) 0xc09f, // (12, 159) 134
- (short) 0x80e7, // ( 8, 231) 135
- (short) 0xc89f, // (12, 2207) 136
- (short) 0xb12f, // (11, 303) 137
- (short) 0xc49f, // (12, 1183) 138
- (short) 0x91b7, // ( 9, 439) 139
- (short) 0xcc9f, // (12, 3231) 140
- (short) 0xb52f, // (11, 1327) 141
- (short) 0xc29f, // (12, 671) 142
- (short) 0x9077, // ( 9, 119) 143
- (short) 0xca9f, // (12, 2719) 144
- (short) 0xc69f, // (12, 1695) 145
- (short) 0xce9f, // (12, 3743) 146
- (short) 0xa14f, // (10, 335) 147
- (short) 0xc19f, // (12, 415) 148
- (short) 0xc99f, // (12, 2463) 149
- (short) 0xc59f, // (12, 1439) 150
- (short) 0xa34f, // (10, 847) 151
- (short) 0xcd9f, // (12, 3487) 152
- (short) 0xc39f, // (12, 927) 153
- (short) 0xcb9f, // (12, 2975) 154
- (short) 0xc79f, // (12, 1951) 155
- (short) 0xcf9f, // (12, 3999) 156
- (short) 0xc05f, // (12, 95) 157
- (short) 0xc85f, // (12, 2143) 158
- (short) 0xb32f, // (11, 815) 159
- (short) 0xc45f, // (12, 1119) 160
- (short) 0xcc5f, // (12, 3167) 161
- (short) 0xc25f, // (12, 607) 162
- (short) 0xb72f, // (11, 1839) 163
- (short) 0xca5f, // (12, 2655) 164
- (short) 0xc65f, // (12, 1631) 165
- (short) 0xce5f, // (12, 3679) 166
- (short) 0xb0af, // (11, 175) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 15 of 22) (steady 15 of 16) (phase = 0.968750000 = 31.0 / 32.0)
- // entropy: 4.4653007097343397902
- // avg_length: 4.5480722016259509388; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x9017, // ( 9, 23) 0
- (short) 0x4006, // ( 4, 6) 1
- (short) 0x7033, // ( 7, 51) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0x9117, // ( 9, 279) 4
- (short) 0x400e, // ( 4, 14) 5
- (short) 0x7073, // ( 7, 115) 6
- (short) 0x2000, // ( 2, 0) 7
- (short) 0xa0f7, // (10, 247) 8
- (short) 0x601d, // ( 6, 29) 9
- (short) 0x803b, // ( 8, 59) 10
- (short) 0x4001, // ( 4, 1) 11
- (short) 0xa2f7, // (10, 759) 12
- (short) 0x5005, // ( 5, 5) 13
- (short) 0x80bb, // ( 8, 187) 14
- (short) 0x4009, // ( 4, 9) 15
- (short) 0xb0cf, // (11, 207) 16
- (short) 0x700b, // ( 7, 11) 17
- (short) 0x9097, // ( 9, 151) 18
- (short) 0x5015, // ( 5, 21) 19
- (short) 0xb4cf, // (11, 1231) 20
- (short) 0x704b, // ( 7, 75) 21
- (short) 0x9197, // ( 9, 407) 22
- (short) 0x500d, // ( 5, 13) 23
- (short) 0xc0af, // (12, 175) 24
- (short) 0x807b, // ( 8, 123) 25
- (short) 0xb2cf, // (11, 719) 26
- (short) 0x603d, // ( 6, 61) 27
- (short) 0xc8af, // (12, 2223) 28
- (short) 0x80fb, // ( 8, 251) 29
- (short) 0xa1f7, // (10, 503) 30
- (short) 0x6003, // ( 6, 3) 31
- (short) 0xc4af, // (12, 1199) 32
- (short) 0x8007, // ( 8, 7) 33
- (short) 0xb6cf, // (11, 1743) 34
- (short) 0x6023, // ( 6, 35) 35
- (short) 0xccaf, // (12, 3247) 36
- (short) 0x8087, // ( 8, 135) 37
- (short) 0xa3f7, // (10, 1015) 38
- (short) 0x6013, // ( 6, 19) 39
- (short) 0xc2af, // (12, 687) 40
- (short) 0x9057, // ( 9, 87) 41
- (short) 0xcaaf, // (12, 2735) 42
- (short) 0x702b, // ( 7, 43) 43
- (short) 0xc6af, // (12, 1711) 44
- (short) 0x9157, // ( 9, 343) 45
- (short) 0xb1cf, // (11, 463) 46
- (short) 0x706b, // ( 7, 107) 47
- (short) 0xceaf, // (12, 3759) 48
- (short) 0xa00f, // (10, 15) 49
- (short) 0xc1af, // (12, 431) 50
- (short) 0x8047, // ( 8, 71) 51
- (short) 0xc9af, // (12, 2479) 52
- (short) 0xa20f, // (10, 527) 53
- (short) 0xc5af, // (12, 1455) 54
- (short) 0x80c7, // ( 8, 199) 55
- (short) 0xcdaf, // (12, 3503) 56
- (short) 0xb5cf, // (11, 1487) 57
- (short) 0xc3af, // (12, 943) 58
- (short) 0x90d7, // ( 9, 215) 59
- (short) 0xcbaf, // (12, 2991) 60
- (short) 0xb3cf, // (11, 975) 61
- (short) 0xc7af, // (12, 1967) 62
- (short) 0x91d7, // ( 9, 471) 63
- (short) 0xcfaf, // (12, 4015) 64
- (short) 0x9037, // ( 9, 55) 65
- (short) 0xc06f, // (12, 111) 66
- (short) 0x701b, // ( 7, 27) 67
- (short) 0xc86f, // (12, 2159) 68
- (short) 0x9137, // ( 9, 311) 69
- (short) 0xb7cf, // (11, 1999) 70
- (short) 0x705b, // ( 7, 91) 71
- (short) 0xc46f, // (12, 1135) 72
- (short) 0xa10f, // (10, 271) 73
- (short) 0xcc6f, // (12, 3183) 74
- (short) 0x8027, // ( 8, 39) 75
- (short) 0xc26f, // (12, 623) 76
- (short) 0xa30f, // (10, 783) 77
- (short) 0xca6f, // (12, 2671) 78
- (short) 0x80a7, // ( 8, 167) 79
- (short) 0xc66f, // (12, 1647) 80
- (short) 0xb02f, // (11, 47) 81
- (short) 0xce6f, // (12, 3695) 82
- (short) 0x90b7, // ( 9, 183) 83
- (short) 0xc16f, // (12, 367) 84
- (short) 0xb42f, // (11, 1071) 85
- (short) 0xc96f, // (12, 2415) 86
- (short) 0x91b7, // ( 9, 439) 87
- (short) 0xc56f, // (12, 1391) 88
- (short) 0xcd6f, // (12, 3439) 89
- (short) 0xc36f, // (12, 879) 90
- (short) 0xa08f, // (10, 143) 91
- (short) 0xcb6f, // (12, 2927) 92
- (short) 0xc76f, // (12, 1903) 93
- (short) 0xcf6f, // (12, 3951) 94
- (short) 0xa28f, // (10, 655) 95
- (short) 0xc0ef, // (12, 239) 96
- (short) 0xc8ef, // (12, 2287) 97
- (short) 0xc4ef, // (12, 1263) 98
- (short) 0xa18f, // (10, 399) 99
- (short) 0xccef, // (12, 3311) 100
- (short) 0xc2ef, // (12, 751) 101
- (short) 0xcaef, // (12, 2799) 102
- (short) 0xa38f, // (10, 911) 103
- (short) 0xc6ef, // (12, 1775) 104
- (short) 0xceef, // (12, 3823) 105
- (short) 0xc1ef, // (12, 495) 106
- (short) 0xc9ef, // (12, 2543) 107
- (short) 0xc5ef, // (12, 1519) 108
- (short) 0xcdef, // (12, 3567) 109
- (short) 0xc3ef, // (12, 1007) 110
- (short) 0xb22f, // (11, 559) 111
- (short) 0xcbef, // (12, 3055) 112
- (short) 0xc7ef, // (12, 2031) 113
- (short) 0xcfef, // (12, 4079) 114
- (short) 0xc01f, // (12, 31) 115
- (short) 0xc81f, // (12, 2079) 116
- (short) 0xc41f, // (12, 1055) 117
- (short) 0xcc1f, // (12, 3103) 118
- (short) 0xc21f, // (12, 543) 119
- (short) 0xca1f, // (12, 2591) 120
- (short) 0xc61f, // (12, 1567) 121
- (short) 0xce1f, // (12, 3615) 122
- (short) 0xc11f, // (12, 287) 123
- (short) 0xc91f, // (12, 2335) 124
- (short) 0xc51f, // (12, 1311) 125
- (short) 0xcd1f, // (12, 3359) 126
- (short) 0xc31f, // (12, 799) 127
- (short) 0xcb1f, // (12, 2847) 128
- (short) 0xa04f, // (10, 79) 129
- (short) 0xc71f, // (12, 1823) 130
- (short) 0x8067, // ( 8, 103) 131
- (short) 0xcf1f, // (12, 3871) 132
- (short) 0xa24f, // (10, 591) 133
- (short) 0xc09f, // (12, 159) 134
- (short) 0x80e7, // ( 8, 231) 135
- (short) 0xc89f, // (12, 2207) 136
- (short) 0xb62f, // (11, 1583) 137
- (short) 0xc49f, // (12, 1183) 138
- (short) 0x9077, // ( 9, 119) 139
- (short) 0xcc9f, // (12, 3231) 140
- (short) 0xb12f, // (11, 303) 141
- (short) 0xc29f, // (12, 671) 142
- (short) 0x9177, // ( 9, 375) 143
- (short) 0xca9f, // (12, 2719) 144
- (short) 0xc69f, // (12, 1695) 145
- (short) 0xce9f, // (12, 3743) 146
- (short) 0xa14f, // (10, 335) 147
- (short) 0xc19f, // (12, 415) 148
- (short) 0xc99f, // (12, 2463) 149
- (short) 0xc59f, // (12, 1439) 150
- (short) 0xa34f, // (10, 847) 151
- (short) 0xcd9f, // (12, 3487) 152
- (short) 0xc39f, // (12, 927) 153
- (short) 0xcb9f, // (12, 2975) 154
- (short) 0xc79f, // (12, 1951) 155
- (short) 0xcf9f, // (12, 3999) 156
- (short) 0xc05f, // (12, 95) 157
- (short) 0xc85f, // (12, 2143) 158
- (short) 0xb52f, // (11, 1327) 159
- (short) 0xc45f, // (12, 1119) 160
- (short) 0xcc5f, // (12, 3167) 161
- (short) 0xc25f, // (12, 607) 162
- (short) 0xb32f, // (11, 815) 163
- (short) 0xca5f, // (12, 2655) 164
- (short) 0xc65f, // (12, 1631) 165
- (short) 0xce5f, // (12, 3679) 166
- (short) 0xb72f, // (11, 1839) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // Six Encoding Tables for the Midrange.
-
- // (table 16 of 22) (midrange 0 of 6) (c/k = 0.500000000 = 3.0 / 6.0)
- // entropy: 2.1627885076675394949
- // avg_length: 2.2704182849800043087; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x1000, // ( 1, 0) 0
- (short) 0x2001, // ( 2, 1) 1
- (short) 0x4003, // ( 4, 3) 2
- (short) 0x500b, // ( 5, 11) 3
- (short) 0x501b, // ( 5, 27) 4
- (short) 0x6007, // ( 6, 7) 5
- (short) 0x8057, // ( 8, 87) 6
- (short) 0x9077, // ( 9, 119) 7
- (short) 0x6027, // ( 6, 39) 8
- (short) 0x80d7, // ( 8, 215) 9
- (short) 0x9177, // ( 9, 375) 10
- (short) 0xa1f7, // (10, 503) 11
- (short) 0xa3f7, // (10, 1015) 12
- (short) 0xb08f, // (11, 143) 13
- (short) 0xc58f, // (12, 1423) 14
- (short) 0xcd8f, // (12, 3471) 15
- (short) 0x7017, // ( 7, 23) 16
- (short) 0x8037, // ( 8, 55) 17
- (short) 0xa00f, // (10, 15) 18
- (short) 0xb48f, // (11, 1167) 19
- (short) 0xb28f, // (11, 655) 20
- (short) 0xc38f, // (12, 911) 21
- (short) 0xcb8f, // (12, 2959) 22
- (short) 0xc78f, // (12, 1935) 23
- (short) 0xcf8f, // (12, 3983) 24
- (short) 0xc04f, // (12, 79) 25
- (short) 0xc84f, // (12, 2127) 26
- (short) 0xc44f, // (12, 1103) 27
- (short) 0xcc4f, // (12, 3151) 28
- (short) 0xc24f, // (12, 591) 29
- (short) 0xca4f, // (12, 2639) 30
- (short) 0xc64f, // (12, 1615) 31
- (short) 0x80b7, // ( 8, 183) 32
- (short) 0xa20f, // (10, 527) 33
- (short) 0xb68f, // (11, 1679) 34
- (short) 0xce4f, // (12, 3663) 35
- (short) 0xc14f, // (12, 335) 36
- (short) 0xc94f, // (12, 2383) 37
- (short) 0xc54f, // (12, 1359) 38
- (short) 0xcd4f, // (12, 3407) 39
- (short) 0xc34f, // (12, 847) 40
- (short) 0xcb4f, // (12, 2895) 41
- (short) 0xc74f, // (12, 1871) 42
- (short) 0xcf4f, // (12, 3919) 43
- (short) 0xc0cf, // (12, 207) 44
- (short) 0xc8cf, // (12, 2255) 45
- (short) 0xc4cf, // (12, 1231) 46
- (short) 0xcccf, // (12, 3279) 47
- (short) 0xc2cf, // (12, 719) 48
- (short) 0xcacf, // (12, 2767) 49
- (short) 0xc6cf, // (12, 1743) 50
- (short) 0xcecf, // (12, 3791) 51
- (short) 0xc1cf, // (12, 463) 52
- (short) 0xc9cf, // (12, 2511) 53
- (short) 0xc5cf, // (12, 1487) 54
- (short) 0xcdcf, // (12, 3535) 55
- (short) 0xc3cf, // (12, 975) 56
- (short) 0xcbcf, // (12, 3023) 57
- (short) 0xc7cf, // (12, 1999) 58
- (short) 0xcfcf, // (12, 4047) 59
- (short) 0xc02f, // (12, 47) 60
- (short) 0xc82f, // (12, 2095) 61
- (short) 0xc42f, // (12, 1071) 62
- (short) 0xcc2f, // (12, 3119) 63
- (short) 0x90f7, // ( 9, 247) 64
- (short) 0xa10f, // (10, 271) 65
- (short) 0xc22f, // (12, 559) 66
- (short) 0xca2f, // (12, 2607) 67
- (short) 0xc62f, // (12, 1583) 68
- (short) 0xce2f, // (12, 3631) 69
- (short) 0xc12f, // (12, 303) 70
- (short) 0xc92f, // (12, 2351) 71
- (short) 0xc52f, // (12, 1327) 72
- (short) 0xcd2f, // (12, 3375) 73
- (short) 0xc32f, // (12, 815) 74
- (short) 0xcb2f, // (12, 2863) 75
- (short) 0xc72f, // (12, 1839) 76
- (short) 0xcf2f, // (12, 3887) 77
- (short) 0xc0af, // (12, 175) 78
- (short) 0xc8af, // (12, 2223) 79
- (short) 0xc4af, // (12, 1199) 80
- (short) 0xccaf, // (12, 3247) 81
- (short) 0xc2af, // (12, 687) 82
- (short) 0xcaaf, // (12, 2735) 83
- (short) 0xc6af, // (12, 1711) 84
- (short) 0xceaf, // (12, 3759) 85
- (short) 0xc1af, // (12, 431) 86
- (short) 0xc9af, // (12, 2479) 87
- (short) 0xc5af, // (12, 1455) 88
- (short) 0xcdaf, // (12, 3503) 89
- (short) 0xc3af, // (12, 943) 90
- (short) 0xcbaf, // (12, 2991) 91
- (short) 0xc7af, // (12, 1967) 92
- (short) 0xcfaf, // (12, 4015) 93
- (short) 0xc06f, // (12, 111) 94
- (short) 0xc86f, // (12, 2159) 95
- (short) 0xc46f, // (12, 1135) 96
- (short) 0xcc6f, // (12, 3183) 97
- (short) 0xc26f, // (12, 623) 98
- (short) 0xca6f, // (12, 2671) 99
- (short) 0xc66f, // (12, 1647) 100
- (short) 0xce6f, // (12, 3695) 101
- (short) 0xc16f, // (12, 367) 102
- (short) 0xc96f, // (12, 2415) 103
- (short) 0xc56f, // (12, 1391) 104
- (short) 0xcd6f, // (12, 3439) 105
- (short) 0xc36f, // (12, 879) 106
- (short) 0xcb6f, // (12, 2927) 107
- (short) 0xc76f, // (12, 1903) 108
- (short) 0xcf6f, // (12, 3951) 109
- (short) 0xc0ef, // (12, 239) 110
- (short) 0xc8ef, // (12, 2287) 111
- (short) 0xc4ef, // (12, 1263) 112
- (short) 0xccef, // (12, 3311) 113
- (short) 0xc2ef, // (12, 751) 114
- (short) 0xcaef, // (12, 2799) 115
- (short) 0xc6ef, // (12, 1775) 116
- (short) 0xceef, // (12, 3823) 117
- (short) 0xc1ef, // (12, 495) 118
- (short) 0xc9ef, // (12, 2543) 119
- (short) 0xc5ef, // (12, 1519) 120
- (short) 0xcdef, // (12, 3567) 121
- (short) 0xc3ef, // (12, 1007) 122
- (short) 0xcbef, // (12, 3055) 123
- (short) 0xc7ef, // (12, 2031) 124
- (short) 0xcfef, // (12, 4079) 125
- (short) 0xc01f, // (12, 31) 126
- (short) 0xc81f, // (12, 2079) 127
- (short) 0xa30f, // (10, 783) 128
- (short) 0xb18f, // (11, 399) 129
- (short) 0xc41f, // (12, 1055) 130
- (short) 0xcc1f, // (12, 3103) 131
- (short) 0xc21f, // (12, 543) 132
- (short) 0xca1f, // (12, 2591) 133
- (short) 0xc61f, // (12, 1567) 134
- (short) 0xce1f, // (12, 3615) 135
- (short) 0xc11f, // (12, 287) 136
- (short) 0xc91f, // (12, 2335) 137
- (short) 0xc51f, // (12, 1311) 138
- (short) 0xcd1f, // (12, 3359) 139
- (short) 0xc31f, // (12, 799) 140
- (short) 0xcb1f, // (12, 2847) 141
- (short) 0xc71f, // (12, 1823) 142
- (short) 0xcf1f, // (12, 3871) 143
- (short) 0xc09f, // (12, 159) 144
- (short) 0xc89f, // (12, 2207) 145
- (short) 0xc49f, // (12, 1183) 146
- (short) 0xcc9f, // (12, 3231) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 17 of 22) (midrange 1 of 6) (c/k = 0.833333333 = 5.0 / 6.0)
- // entropy: 2.9553294756640680063
- // avg_length: 3.0766035704232641557; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x2000, // ( 2, 0) 0
- (short) 0x2002, // ( 2, 2) 1
- (short) 0x3001, // ( 3, 1) 2
- (short) 0x4005, // ( 4, 5) 3
- (short) 0x400d, // ( 4, 13) 4
- (short) 0x5003, // ( 5, 3) 5
- (short) 0x600b, // ( 6, 11) 6
- (short) 0x602b, // ( 6, 43) 7
- (short) 0x5013, // ( 5, 19) 8
- (short) 0x601b, // ( 6, 27) 9
- (short) 0x7007, // ( 7, 7) 10
- (short) 0x7047, // ( 7, 71) 11
- (short) 0x8017, // ( 8, 23) 12
- (short) 0x90b7, // ( 9, 183) 13
- (short) 0xa1f7, // (10, 503) 14
- (short) 0xa3f7, // (10, 1015) 15
- (short) 0x603b, // ( 6, 59) 16
- (short) 0x7027, // ( 7, 39) 17
- (short) 0x8097, // ( 8, 151) 18
- (short) 0x8057, // ( 8, 87) 19
- (short) 0x91b7, // ( 9, 439) 20
- (short) 0xa00f, // (10, 15) 21
- (short) 0xb18f, // (11, 399) 22
- (short) 0xb58f, // (11, 1423) 23
- (short) 0xa20f, // (10, 527) 24
- (short) 0xb38f, // (11, 911) 25
- (short) 0xc54f, // (12, 1359) 26
- (short) 0xcd4f, // (12, 3407) 27
- (short) 0xc34f, // (12, 847) 28
- (short) 0xcb4f, // (12, 2895) 29
- (short) 0xc74f, // (12, 1871) 30
- (short) 0xcf4f, // (12, 3919) 31
- (short) 0x7067, // ( 7, 103) 32
- (short) 0x80d7, // ( 8, 215) 33
- (short) 0x9077, // ( 9, 119) 34
- (short) 0xa10f, // (10, 271) 35
- (short) 0xa30f, // (10, 783) 36
- (short) 0xb78f, // (11, 1935) 37
- (short) 0xc0cf, // (12, 207) 38
- (short) 0xc8cf, // (12, 2255) 39
- (short) 0xb04f, // (11, 79) 40
- (short) 0xc4cf, // (12, 1231) 41
- (short) 0xcccf, // (12, 3279) 42
- (short) 0xc2cf, // (12, 719) 43
- (short) 0xcacf, // (12, 2767) 44
- (short) 0xc6cf, // (12, 1743) 45
- (short) 0xcecf, // (12, 3791) 46
- (short) 0xc1cf, // (12, 463) 47
- (short) 0xc9cf, // (12, 2511) 48
- (short) 0xc5cf, // (12, 1487) 49
- (short) 0xcdcf, // (12, 3535) 50
- (short) 0xc3cf, // (12, 975) 51
- (short) 0xcbcf, // (12, 3023) 52
- (short) 0xc7cf, // (12, 1999) 53
- (short) 0xcfcf, // (12, 4047) 54
- (short) 0xc02f, // (12, 47) 55
- (short) 0xc82f, // (12, 2095) 56
- (short) 0xc42f, // (12, 1071) 57
- (short) 0xcc2f, // (12, 3119) 58
- (short) 0xc22f, // (12, 559) 59
- (short) 0xca2f, // (12, 2607) 60
- (short) 0xc62f, // (12, 1583) 61
- (short) 0xce2f, // (12, 3631) 62
- (short) 0xc12f, // (12, 303) 63
- (short) 0x8037, // ( 8, 55) 64
- (short) 0x9177, // ( 9, 375) 65
- (short) 0xa08f, // (10, 143) 66
- (short) 0xb44f, // (11, 1103) 67
- (short) 0xb24f, // (11, 591) 68
- (short) 0xc92f, // (12, 2351) 69
- (short) 0xc52f, // (12, 1327) 70
- (short) 0xcd2f, // (12, 3375) 71
- (short) 0xc32f, // (12, 815) 72
- (short) 0xcb2f, // (12, 2863) 73
- (short) 0xc72f, // (12, 1839) 74
- (short) 0xcf2f, // (12, 3887) 75
- (short) 0xc0af, // (12, 175) 76
- (short) 0xc8af, // (12, 2223) 77
- (short) 0xc4af, // (12, 1199) 78
- (short) 0xccaf, // (12, 3247) 79
- (short) 0xc2af, // (12, 687) 80
- (short) 0xcaaf, // (12, 2735) 81
- (short) 0xc6af, // (12, 1711) 82
- (short) 0xceaf, // (12, 3759) 83
- (short) 0xc1af, // (12, 431) 84
- (short) 0xc9af, // (12, 2479) 85
- (short) 0xc5af, // (12, 1455) 86
- (short) 0xcdaf, // (12, 3503) 87
- (short) 0xc3af, // (12, 943) 88
- (short) 0xcbaf, // (12, 2991) 89
- (short) 0xc7af, // (12, 1967) 90
- (short) 0xcfaf, // (12, 4015) 91
- (short) 0xc06f, // (12, 111) 92
- (short) 0xc86f, // (12, 2159) 93
- (short) 0xc46f, // (12, 1135) 94
- (short) 0xcc6f, // (12, 3183) 95
- (short) 0xc26f, // (12, 623) 96
- (short) 0xca6f, // (12, 2671) 97
- (short) 0xc66f, // (12, 1647) 98
- (short) 0xce6f, // (12, 3695) 99
- (short) 0xc16f, // (12, 367) 100
- (short) 0xc96f, // (12, 2415) 101
- (short) 0xc56f, // (12, 1391) 102
- (short) 0xcd6f, // (12, 3439) 103
- (short) 0xc36f, // (12, 879) 104
- (short) 0xcb6f, // (12, 2927) 105
- (short) 0xc76f, // (12, 1903) 106
- (short) 0xcf6f, // (12, 3951) 107
- (short) 0xc0ef, // (12, 239) 108
- (short) 0xc8ef, // (12, 2287) 109
- (short) 0xc4ef, // (12, 1263) 110
- (short) 0xccef, // (12, 3311) 111
- (short) 0xc2ef, // (12, 751) 112
- (short) 0xcaef, // (12, 2799) 113
- (short) 0xc6ef, // (12, 1775) 114
- (short) 0xceef, // (12, 3823) 115
- (short) 0xc1ef, // (12, 495) 116
- (short) 0xc9ef, // (12, 2543) 117
- (short) 0xc5ef, // (12, 1519) 118
- (short) 0xcdef, // (12, 3567) 119
- (short) 0xc3ef, // (12, 1007) 120
- (short) 0xcbef, // (12, 3055) 121
- (short) 0xc7ef, // (12, 2031) 122
- (short) 0xcfef, // (12, 4079) 123
- (short) 0xc01f, // (12, 31) 124
- (short) 0xc81f, // (12, 2079) 125
- (short) 0xc41f, // (12, 1055) 126
- (short) 0xcc1f, // (12, 3103) 127
- (short) 0x90f7, // ( 9, 247) 128
- (short) 0xa28f, // (10, 655) 129
- (short) 0xb64f, // (11, 1615) 130
- (short) 0xb14f, // (11, 335) 131
- (short) 0xc21f, // (12, 543) 132
- (short) 0xca1f, // (12, 2591) 133
- (short) 0xc61f, // (12, 1567) 134
- (short) 0xce1f, // (12, 3615) 135
- (short) 0xc11f, // (12, 287) 136
- (short) 0xc91f, // (12, 2335) 137
- (short) 0xc51f, // (12, 1311) 138
- (short) 0xcd1f, // (12, 3359) 139
- (short) 0xc31f, // (12, 799) 140
- (short) 0xcb1f, // (12, 2847) 141
- (short) 0xc71f, // (12, 1823) 142
- (short) 0xcf1f, // (12, 3871) 143
- (short) 0xc09f, // (12, 159) 144
- (short) 0xc89f, // (12, 2207) 145
- (short) 0xc49f, // (12, 1183) 146
- (short) 0xcc9f, // (12, 3231) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 18 of 22) (midrange 2 of 6) (c/k = 1.166666667 = 7.0 / 6.0)
- // entropy: 3.5218672531711128215
- // avg_length: 3.6153551492375441967; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x2000, // ( 2, 0) 0
- (short) 0x2002, // ( 2, 2) 1
- (short) 0x4005, // ( 4, 5) 2
- (short) 0x3001, // ( 3, 1) 3
- (short) 0x5003, // ( 5, 3) 4
- (short) 0x400d, // ( 4, 13) 5
- (short) 0x600b, // ( 6, 11) 6
- (short) 0x602b, // ( 6, 43) 7
- (short) 0x601b, // ( 6, 27) 8
- (short) 0x5013, // ( 5, 19) 9
- (short) 0x703b, // ( 7, 59) 10
- (short) 0x707b, // ( 7, 123) 11
- (short) 0x8067, // ( 8, 103) 12
- (short) 0x80e7, // ( 8, 231) 13
- (short) 0x90d7, // ( 9, 215) 14
- (short) 0x91d7, // ( 9, 471) 15
- (short) 0x7007, // ( 7, 7) 16
- (short) 0x7047, // ( 7, 71) 17
- (short) 0x8017, // ( 8, 23) 18
- (short) 0x8097, // ( 8, 151) 19
- (short) 0x9037, // ( 9, 55) 20
- (short) 0x9137, // ( 9, 311) 21
- (short) 0xa1f7, // (10, 503) 22
- (short) 0xa3f7, // (10, 1015) 23
- (short) 0xa00f, // (10, 15) 24
- (short) 0xa20f, // (10, 527) 25
- (short) 0xb38f, // (11, 911) 26
- (short) 0xb78f, // (11, 1935) 27
- (short) 0xc0cf, // (12, 207) 28
- (short) 0xc8cf, // (12, 2255) 29
- (short) 0xc4cf, // (12, 1231) 30
- (short) 0xcccf, // (12, 3279) 31
- (short) 0x8057, // ( 8, 87) 32
- (short) 0x7027, // ( 7, 39) 33
- (short) 0x90b7, // ( 9, 183) 34
- (short) 0x91b7, // ( 9, 439) 35
- (short) 0xa10f, // (10, 271) 36
- (short) 0xa30f, // (10, 783) 37
- (short) 0xb04f, // (11, 79) 38
- (short) 0xb44f, // (11, 1103) 39
- (short) 0xb24f, // (11, 591) 40
- (short) 0xb64f, // (11, 1615) 41
- (short) 0xc2cf, // (12, 719) 42
- (short) 0xcacf, // (12, 2767) 43
- (short) 0xc6cf, // (12, 1743) 44
- (short) 0xcecf, // (12, 3791) 45
- (short) 0xc1cf, // (12, 463) 46
- (short) 0xc9cf, // (12, 2511) 47
- (short) 0xc5cf, // (12, 1487) 48
- (short) 0xcdcf, // (12, 3535) 49
- (short) 0xc3cf, // (12, 975) 50
- (short) 0xcbcf, // (12, 3023) 51
- (short) 0xc7cf, // (12, 1999) 52
- (short) 0xcfcf, // (12, 4047) 53
- (short) 0xc02f, // (12, 47) 54
- (short) 0xc82f, // (12, 2095) 55
- (short) 0xc42f, // (12, 1071) 56
- (short) 0xcc2f, // (12, 3119) 57
- (short) 0xc22f, // (12, 559) 58
- (short) 0xca2f, // (12, 2607) 59
- (short) 0xc62f, // (12, 1583) 60
- (short) 0xce2f, // (12, 3631) 61
- (short) 0xc12f, // (12, 303) 62
- (short) 0xc92f, // (12, 2351) 63
- (short) 0x9077, // ( 9, 119) 64
- (short) 0x9177, // ( 9, 375) 65
- (short) 0xa08f, // (10, 143) 66
- (short) 0xa28f, // (10, 655) 67
- (short) 0xb14f, // (11, 335) 68
- (short) 0xb54f, // (11, 1359) 69
- (short) 0xc52f, // (12, 1327) 70
- (short) 0xcd2f, // (12, 3375) 71
- (short) 0xc32f, // (12, 815) 72
- (short) 0xcb2f, // (12, 2863) 73
- (short) 0xc72f, // (12, 1839) 74
- (short) 0xcf2f, // (12, 3887) 75
- (short) 0xc0af, // (12, 175) 76
- (short) 0xc8af, // (12, 2223) 77
- (short) 0xc4af, // (12, 1199) 78
- (short) 0xccaf, // (12, 3247) 79
- (short) 0xc2af, // (12, 687) 80
- (short) 0xcaaf, // (12, 2735) 81
- (short) 0xc6af, // (12, 1711) 82
- (short) 0xceaf, // (12, 3759) 83
- (short) 0xc1af, // (12, 431) 84
- (short) 0xc9af, // (12, 2479) 85
- (short) 0xc5af, // (12, 1455) 86
- (short) 0xcdaf, // (12, 3503) 87
- (short) 0xc3af, // (12, 943) 88
- (short) 0xcbaf, // (12, 2991) 89
- (short) 0xc7af, // (12, 1967) 90
- (short) 0xcfaf, // (12, 4015) 91
- (short) 0xc06f, // (12, 111) 92
- (short) 0xc86f, // (12, 2159) 93
- (short) 0xc46f, // (12, 1135) 94
- (short) 0xcc6f, // (12, 3183) 95
- (short) 0xc26f, // (12, 623) 96
- (short) 0xca6f, // (12, 2671) 97
- (short) 0xc66f, // (12, 1647) 98
- (short) 0xce6f, // (12, 3695) 99
- (short) 0xc16f, // (12, 367) 100
- (short) 0xc96f, // (12, 2415) 101
- (short) 0xc56f, // (12, 1391) 102
- (short) 0xcd6f, // (12, 3439) 103
- (short) 0xc36f, // (12, 879) 104
- (short) 0xcb6f, // (12, 2927) 105
- (short) 0xc76f, // (12, 1903) 106
- (short) 0xcf6f, // (12, 3951) 107
- (short) 0xc0ef, // (12, 239) 108
- (short) 0xc8ef, // (12, 2287) 109
- (short) 0xc4ef, // (12, 1263) 110
- (short) 0xccef, // (12, 3311) 111
- (short) 0xc2ef, // (12, 751) 112
- (short) 0xcaef, // (12, 2799) 113
- (short) 0xc6ef, // (12, 1775) 114
- (short) 0xceef, // (12, 3823) 115
- (short) 0xc1ef, // (12, 495) 116
- (short) 0xc9ef, // (12, 2543) 117
- (short) 0xc5ef, // (12, 1519) 118
- (short) 0xcdef, // (12, 3567) 119
- (short) 0xc3ef, // (12, 1007) 120
- (short) 0xcbef, // (12, 3055) 121
- (short) 0xc7ef, // (12, 2031) 122
- (short) 0xcfef, // (12, 4079) 123
- (short) 0xc01f, // (12, 31) 124
- (short) 0xc81f, // (12, 2079) 125
- (short) 0xc41f, // (12, 1055) 126
- (short) 0xcc1f, // (12, 3103) 127
- (short) 0xa18f, // (10, 399) 128
- (short) 0x90f7, // ( 9, 247) 129
- (short) 0xb34f, // (11, 847) 130
- (short) 0xb74f, // (11, 1871) 131
- (short) 0xc21f, // (12, 543) 132
- (short) 0xca1f, // (12, 2591) 133
- (short) 0xc61f, // (12, 1567) 134
- (short) 0xce1f, // (12, 3615) 135
- (short) 0xc11f, // (12, 287) 136
- (short) 0xc91f, // (12, 2335) 137
- (short) 0xc51f, // (12, 1311) 138
- (short) 0xcd1f, // (12, 3359) 139
- (short) 0xc31f, // (12, 799) 140
- (short) 0xcb1f, // (12, 2847) 141
- (short) 0xc71f, // (12, 1823) 142
- (short) 0xcf1f, // (12, 3871) 143
- (short) 0xc09f, // (12, 159) 144
- (short) 0xc89f, // (12, 2207) 145
- (short) 0xc49f, // (12, 1183) 146
- (short) 0xcc9f, // (12, 3231) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 19 of 22) (midrange 3 of 6) (c/k = 1.500000000 = 9.0 / 6.0)
- // entropy: 3.9228873257934386842
- // avg_length: 3.9989687586992346269; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x3002, // ( 3, 2) 0
- (short) 0x2000, // ( 2, 0) 1
- (short) 0x4001, // ( 4, 1) 2
- (short) 0x3006, // ( 3, 6) 3
- (short) 0x500d, // ( 5, 13) 4
- (short) 0x4009, // ( 4, 9) 5
- (short) 0x501d, // ( 5, 29) 6
- (short) 0x4005, // ( 4, 5) 7
- (short) 0x6013, // ( 6, 19) 8
- (short) 0x5003, // ( 5, 3) 9
- (short) 0x6033, // ( 6, 51) 10
- (short) 0x600b, // ( 6, 11) 11
- (short) 0x8027, // ( 8, 39) 12
- (short) 0x701b, // ( 7, 27) 13
- (short) 0x80a7, // ( 8, 167) 14
- (short) 0x705b, // ( 7, 91) 15
- (short) 0x703b, // ( 7, 59) 16
- (short) 0x602b, // ( 6, 43) 17
- (short) 0x707b, // ( 7, 123) 18
- (short) 0x7007, // ( 7, 7) 19
- (short) 0x90d7, // ( 9, 215) 20
- (short) 0x8067, // ( 8, 103) 21
- (short) 0x91d7, // ( 9, 471) 22
- (short) 0x80e7, // ( 8, 231) 23
- (short) 0xa1f7, // (10, 503) 24
- (short) 0x9037, // ( 9, 55) 25
- (short) 0xa3f7, // (10, 1015) 26
- (short) 0xa00f, // (10, 15) 27
- (short) 0xc5cf, // (12, 1487) 28
- (short) 0xb04f, // (11, 79) 29
- (short) 0xcdcf, // (12, 3535) 30
- (short) 0xb44f, // (11, 1103) 31
- (short) 0x8017, // ( 8, 23) 32
- (short) 0x7047, // ( 7, 71) 33
- (short) 0x9137, // ( 9, 311) 34
- (short) 0x8097, // ( 8, 151) 35
- (short) 0xa20f, // (10, 527) 36
- (short) 0x90b7, // ( 9, 183) 37
- (short) 0xa10f, // (10, 271) 38
- (short) 0x91b7, // ( 9, 439) 39
- (short) 0xb24f, // (11, 591) 40
- (short) 0xa30f, // (10, 783) 41
- (short) 0xb64f, // (11, 1615) 42
- (short) 0xb14f, // (11, 335) 43
- (short) 0xc3cf, // (12, 975) 44
- (short) 0xcbcf, // (12, 3023) 45
- (short) 0xc7cf, // (12, 1999) 46
- (short) 0xcfcf, // (12, 4047) 47
- (short) 0xc02f, // (12, 47) 48
- (short) 0xb54f, // (11, 1359) 49
- (short) 0xc82f, // (12, 2095) 50
- (short) 0xc42f, // (12, 1071) 51
- (short) 0xcc2f, // (12, 3119) 52
- (short) 0xc22f, // (12, 559) 53
- (short) 0xca2f, // (12, 2607) 54
- (short) 0xc62f, // (12, 1583) 55
- (short) 0xce2f, // (12, 3631) 56
- (short) 0xc12f, // (12, 303) 57
- (short) 0xc92f, // (12, 2351) 58
- (short) 0xc52f, // (12, 1327) 59
- (short) 0xcd2f, // (12, 3375) 60
- (short) 0xc32f, // (12, 815) 61
- (short) 0xcb2f, // (12, 2863) 62
- (short) 0xc72f, // (12, 1839) 63
- (short) 0x9077, // ( 9, 119) 64
- (short) 0x8057, // ( 8, 87) 65
- (short) 0xa08f, // (10, 143) 66
- (short) 0x9177, // ( 9, 375) 67
- (short) 0xb34f, // (11, 847) 68
- (short) 0xa28f, // (10, 655) 69
- (short) 0xb74f, // (11, 1871) 70
- (short) 0xb0cf, // (11, 207) 71
- (short) 0xcf2f, // (12, 3887) 72
- (short) 0xb4cf, // (11, 1231) 73
- (short) 0xc0af, // (12, 175) 74
- (short) 0xc8af, // (12, 2223) 75
- (short) 0xc4af, // (12, 1199) 76
- (short) 0xccaf, // (12, 3247) 77
- (short) 0xc2af, // (12, 687) 78
- (short) 0xcaaf, // (12, 2735) 79
- (short) 0xc6af, // (12, 1711) 80
- (short) 0xceaf, // (12, 3759) 81
- (short) 0xc1af, // (12, 431) 82
- (short) 0xc9af, // (12, 2479) 83
- (short) 0xc5af, // (12, 1455) 84
- (short) 0xcdaf, // (12, 3503) 85
- (short) 0xc3af, // (12, 943) 86
- (short) 0xcbaf, // (12, 2991) 87
- (short) 0xc7af, // (12, 1967) 88
- (short) 0xcfaf, // (12, 4015) 89
- (short) 0xc06f, // (12, 111) 90
- (short) 0xc86f, // (12, 2159) 91
- (short) 0xc46f, // (12, 1135) 92
- (short) 0xcc6f, // (12, 3183) 93
- (short) 0xc26f, // (12, 623) 94
- (short) 0xca6f, // (12, 2671) 95
- (short) 0xc66f, // (12, 1647) 96
- (short) 0xce6f, // (12, 3695) 97
- (short) 0xc16f, // (12, 367) 98
- (short) 0xc96f, // (12, 2415) 99
- (short) 0xc56f, // (12, 1391) 100
- (short) 0xcd6f, // (12, 3439) 101
- (short) 0xc36f, // (12, 879) 102
- (short) 0xcb6f, // (12, 2927) 103
- (short) 0xc76f, // (12, 1903) 104
- (short) 0xcf6f, // (12, 3951) 105
- (short) 0xc0ef, // (12, 239) 106
- (short) 0xc8ef, // (12, 2287) 107
- (short) 0xc4ef, // (12, 1263) 108
- (short) 0xccef, // (12, 3311) 109
- (short) 0xc2ef, // (12, 751) 110
- (short) 0xcaef, // (12, 2799) 111
- (short) 0xc6ef, // (12, 1775) 112
- (short) 0xceef, // (12, 3823) 113
- (short) 0xc1ef, // (12, 495) 114
- (short) 0xc9ef, // (12, 2543) 115
- (short) 0xc5ef, // (12, 1519) 116
- (short) 0xcdef, // (12, 3567) 117
- (short) 0xc3ef, // (12, 1007) 118
- (short) 0xcbef, // (12, 3055) 119
- (short) 0xc7ef, // (12, 2031) 120
- (short) 0xcfef, // (12, 4079) 121
- (short) 0xc01f, // (12, 31) 122
- (short) 0xc81f, // (12, 2079) 123
- (short) 0xc41f, // (12, 1055) 124
- (short) 0xcc1f, // (12, 3103) 125
- (short) 0xc21f, // (12, 543) 126
- (short) 0xca1f, // (12, 2591) 127
- (short) 0xa18f, // (10, 399) 128
- (short) 0x90f7, // ( 9, 247) 129
- (short) 0xb2cf, // (11, 719) 130
- (short) 0xa38f, // (10, 911) 131
- (short) 0xc61f, // (12, 1567) 132
- (short) 0xb6cf, // (11, 1743) 133
- (short) 0xce1f, // (12, 3615) 134
- (short) 0xb1cf, // (11, 463) 135
- (short) 0xc11f, // (12, 287) 136
- (short) 0xc91f, // (12, 2335) 137
- (short) 0xc51f, // (12, 1311) 138
- (short) 0xcd1f, // (12, 3359) 139
- (short) 0xc31f, // (12, 799) 140
- (short) 0xcb1f, // (12, 2847) 141
- (short) 0xc71f, // (12, 1823) 142
- (short) 0xcf1f, // (12, 3871) 143
- (short) 0xc09f, // (12, 159) 144
- (short) 0xc89f, // (12, 2207) 145
- (short) 0xc49f, // (12, 1183) 146
- (short) 0xcc9f, // (12, 3231) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 20 of 22) (midrange 4 of 6) (c/k = 1.833333333 = 11.0 / 6.0)
- // entropy: 4.1937026483207340277
- // avg_length: 4.2809622975207295426; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x4006, // ( 4, 6) 0
- (short) 0x2000, // ( 2, 0) 1
- (short) 0x400e, // ( 4, 14) 2
- (short) 0x3002, // ( 3, 2) 3
- (short) 0x5005, // ( 5, 5) 4
- (short) 0x4001, // ( 4, 1) 5
- (short) 0x5015, // ( 5, 21) 6
- (short) 0x4009, // ( 4, 9) 7
- (short) 0x6003, // ( 6, 3) 8
- (short) 0x500d, // ( 5, 13) 9
- (short) 0x6023, // ( 6, 35) 10
- (short) 0x501d, // ( 5, 29) 11
- (short) 0x8047, // ( 8, 71) 12
- (short) 0x6013, // ( 6, 19) 13
- (short) 0x80c7, // ( 8, 199) 14
- (short) 0x6033, // ( 6, 51) 15
- (short) 0x701b, // ( 7, 27) 16
- (short) 0x600b, // ( 6, 11) 17
- (short) 0x8027, // ( 8, 39) 18
- (short) 0x602b, // ( 6, 43) 19
- (short) 0x90d7, // ( 9, 215) 20
- (short) 0x705b, // ( 7, 91) 21
- (short) 0x91d7, // ( 9, 471) 22
- (short) 0x703b, // ( 7, 59) 23
- (short) 0xa1f7, // (10, 503) 24
- (short) 0x80a7, // ( 8, 167) 25
- (short) 0xa3f7, // (10, 1015) 26
- (short) 0x8067, // ( 8, 103) 27
- (short) 0xb24f, // (11, 591) 28
- (short) 0xa00f, // (10, 15) 29
- (short) 0xb64f, // (11, 1615) 30
- (short) 0xa20f, // (10, 527) 31
- (short) 0x9037, // ( 9, 55) 32
- (short) 0x707b, // ( 7, 123) 33
- (short) 0x9137, // ( 9, 311) 34
- (short) 0x7007, // ( 7, 7) 35
- (short) 0xa10f, // (10, 271) 36
- (short) 0x80e7, // ( 8, 231) 37
- (short) 0xa30f, // (10, 783) 38
- (short) 0x8017, // ( 8, 23) 39
- (short) 0xb14f, // (11, 335) 40
- (short) 0x90b7, // ( 9, 183) 41
- (short) 0xb54f, // (11, 1359) 42
- (short) 0xa08f, // (10, 143) 43
- (short) 0xc02f, // (12, 47) 44
- (short) 0xb34f, // (11, 847) 45
- (short) 0xc82f, // (12, 2095) 46
- (short) 0xb74f, // (11, 1871) 47
- (short) 0xc42f, // (12, 1071) 48
- (short) 0xb0cf, // (11, 207) 49
- (short) 0xcc2f, // (12, 3119) 50
- (short) 0xb4cf, // (11, 1231) 51
- (short) 0xc22f, // (12, 559) 52
- (short) 0xca2f, // (12, 2607) 53
- (short) 0xc62f, // (12, 1583) 54
- (short) 0xce2f, // (12, 3631) 55
- (short) 0xc12f, // (12, 303) 56
- (short) 0xc92f, // (12, 2351) 57
- (short) 0xc52f, // (12, 1327) 58
- (short) 0xcd2f, // (12, 3375) 59
- (short) 0xc32f, // (12, 815) 60
- (short) 0xcb2f, // (12, 2863) 61
- (short) 0xc72f, // (12, 1839) 62
- (short) 0xcf2f, // (12, 3887) 63
- (short) 0xa28f, // (10, 655) 64
- (short) 0x8097, // ( 8, 151) 65
- (short) 0xa18f, // (10, 399) 66
- (short) 0x8057, // ( 8, 87) 67
- (short) 0xb2cf, // (11, 719) 68
- (short) 0x91b7, // ( 9, 439) 69
- (short) 0xb6cf, // (11, 1743) 70
- (short) 0x9077, // ( 9, 119) 71
- (short) 0xc0af, // (12, 175) 72
- (short) 0xb1cf, // (11, 463) 73
- (short) 0xc8af, // (12, 2223) 74
- (short) 0xb5cf, // (11, 1487) 75
- (short) 0xc4af, // (12, 1199) 76
- (short) 0xccaf, // (12, 3247) 77
- (short) 0xc2af, // (12, 687) 78
- (short) 0xcaaf, // (12, 2735) 79
- (short) 0xc6af, // (12, 1711) 80
- (short) 0xceaf, // (12, 3759) 81
- (short) 0xc1af, // (12, 431) 82
- (short) 0xc9af, // (12, 2479) 83
- (short) 0xc5af, // (12, 1455) 84
- (short) 0xcdaf, // (12, 3503) 85
- (short) 0xc3af, // (12, 943) 86
- (short) 0xcbaf, // (12, 2991) 87
- (short) 0xc7af, // (12, 1967) 88
- (short) 0xcfaf, // (12, 4015) 89
- (short) 0xc06f, // (12, 111) 90
- (short) 0xc86f, // (12, 2159) 91
- (short) 0xc46f, // (12, 1135) 92
- (short) 0xcc6f, // (12, 3183) 93
- (short) 0xc26f, // (12, 623) 94
- (short) 0xca6f, // (12, 2671) 95
- (short) 0xc66f, // (12, 1647) 96
- (short) 0xce6f, // (12, 3695) 97
- (short) 0xc16f, // (12, 367) 98
- (short) 0xc96f, // (12, 2415) 99
- (short) 0xc56f, // (12, 1391) 100
- (short) 0xcd6f, // (12, 3439) 101
- (short) 0xc36f, // (12, 879) 102
- (short) 0xcb6f, // (12, 2927) 103
- (short) 0xc76f, // (12, 1903) 104
- (short) 0xcf6f, // (12, 3951) 105
- (short) 0xc0ef, // (12, 239) 106
- (short) 0xc8ef, // (12, 2287) 107
- (short) 0xc4ef, // (12, 1263) 108
- (short) 0xccef, // (12, 3311) 109
- (short) 0xc2ef, // (12, 751) 110
- (short) 0xcaef, // (12, 2799) 111
- (short) 0xc6ef, // (12, 1775) 112
- (short) 0xceef, // (12, 3823) 113
- (short) 0xc1ef, // (12, 495) 114
- (short) 0xc9ef, // (12, 2543) 115
- (short) 0xc5ef, // (12, 1519) 116
- (short) 0xcdef, // (12, 3567) 117
- (short) 0xc3ef, // (12, 1007) 118
- (short) 0xcbef, // (12, 3055) 119
- (short) 0xc7ef, // (12, 2031) 120
- (short) 0xcfef, // (12, 4079) 121
- (short) 0xc01f, // (12, 31) 122
- (short) 0xc81f, // (12, 2079) 123
- (short) 0xc41f, // (12, 1055) 124
- (short) 0xcc1f, // (12, 3103) 125
- (short) 0xc21f, // (12, 543) 126
- (short) 0xca1f, // (12, 2591) 127
- (short) 0xb3cf, // (11, 975) 128
- (short) 0x9177, // ( 9, 375) 129
- (short) 0xb7cf, // (11, 1999) 130
- (short) 0x90f7, // ( 9, 247) 131
- (short) 0xc61f, // (12, 1567) 132
- (short) 0xa38f, // (10, 911) 133
- (short) 0xce1f, // (12, 3615) 134
- (short) 0xa04f, // (10, 79) 135
- (short) 0xc11f, // (12, 287) 136
- (short) 0xc91f, // (12, 2335) 137
- (short) 0xc51f, // (12, 1311) 138
- (short) 0xcd1f, // (12, 3359) 139
- (short) 0xc31f, // (12, 799) 140
- (short) 0xcb1f, // (12, 2847) 141
- (short) 0xc71f, // (12, 1823) 142
- (short) 0xcf1f, // (12, 3871) 143
- (short) 0xc09f, // (12, 159) 144
- (short) 0xc89f, // (12, 2207) 145
- (short) 0xc49f, // (12, 1183) 146
- (short) 0xcc9f, // (12, 3231) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- },
-
- // (table 21 of 22) (midrange 5 of 6) (c/k = 2.166666667 = 13.0 / 6.0)
- // entropy: 4.3601926041863263706
- // avg_length: 4.4384101723259572481; max_length = 12; num_symbols = 256
- {
- //table, // (4 bits, 12 bits) symbol
- //entry, // (length, codeword) [byte]
- (short) 0x5009, // ( 5, 9) 0
- (short) 0x3002, // ( 3, 2) 1
- (short) 0x5019, // ( 5, 25) 2
- (short) 0x2000, // ( 2, 0) 3
- (short) 0x6003, // ( 6, 3) 4
- (short) 0x4001, // ( 4, 1) 5
- (short) 0x5005, // ( 5, 5) 6
- (short) 0x3006, // ( 3, 6) 7
- (short) 0x702b, // ( 7, 43) 8
- (short) 0x5015, // ( 5, 21) 9
- (short) 0x706b, // ( 7, 107) 10
- (short) 0x500d, // ( 5, 13) 11
- (short) 0x8007, // ( 8, 7) 12
- (short) 0x6023, // ( 6, 35) 13
- (short) 0x8087, // ( 8, 135) 14
- (short) 0x501d, // ( 5, 29) 15
- (short) 0x8047, // ( 8, 71) 16
- (short) 0x6013, // ( 6, 19) 17
- (short) 0x80c7, // ( 8, 199) 18
- (short) 0x6033, // ( 6, 51) 19
- (short) 0x9097, // ( 9, 151) 20
- (short) 0x701b, // ( 7, 27) 21
- (short) 0x9197, // ( 9, 407) 22
- (short) 0x600b, // ( 6, 11) 23
- (short) 0xa0f7, // (10, 247) 24
- (short) 0x8027, // ( 8, 39) 25
- (short) 0xa2f7, // (10, 759) 26
- (short) 0x80a7, // ( 8, 167) 27
- (short) 0xb14f, // (11, 335) 28
- (short) 0x9057, // ( 9, 87) 29
- (short) 0xb54f, // (11, 1359) 30
- (short) 0x9157, // ( 9, 343) 31
- (short) 0x90d7, // ( 9, 215) 32
- (short) 0x705b, // ( 7, 91) 33
- (short) 0x91d7, // ( 9, 471) 34
- (short) 0x703b, // ( 7, 59) 35
- (short) 0xa1f7, // (10, 503) 36
- (short) 0x8067, // ( 8, 103) 37
- (short) 0xa3f7, // (10, 1015) 38
- (short) 0x707b, // ( 7, 123) 39
- (short) 0xb34f, // (11, 847) 40
- (short) 0x9037, // ( 9, 55) 41
- (short) 0xb74f, // (11, 1871) 42
- (short) 0x9137, // ( 9, 311) 43
- (short) 0xc12f, // (12, 303) 44
- (short) 0xa00f, // (10, 15) 45
- (short) 0xc92f, // (12, 2351) 46
- (short) 0xa20f, // (10, 527) 47
- (short) 0xc52f, // (12, 1327) 48
- (short) 0xa10f, // (10, 271) 49
- (short) 0xcd2f, // (12, 3375) 50
- (short) 0xa30f, // (10, 783) 51
- (short) 0xc32f, // (12, 815) 52
- (short) 0xb0cf, // (11, 207) 53
- (short) 0xcb2f, // (12, 2863) 54
- (short) 0xb4cf, // (11, 1231) 55
- (short) 0xc72f, // (12, 1839) 56
- (short) 0xcf2f, // (12, 3887) 57
- (short) 0xc0af, // (12, 175) 58
- (short) 0xc8af, // (12, 2223) 59
- (short) 0xc4af, // (12, 1199) 60
- (short) 0xccaf, // (12, 3247) 61
- (short) 0xc2af, // (12, 687) 62
- (short) 0xcaaf, // (12, 2735) 63
- (short) 0xa08f, // (10, 143) 64
- (short) 0x80e7, // ( 8, 231) 65
- (short) 0xa28f, // (10, 655) 66
- (short) 0x8017, // ( 8, 23) 67
- (short) 0xb2cf, // (11, 719) 68
- (short) 0x90b7, // ( 9, 183) 69
- (short) 0xb6cf, // (11, 1743) 70
- (short) 0x91b7, // ( 9, 439) 71
- (short) 0xc6af, // (12, 1711) 72
- (short) 0xa18f, // (10, 399) 73
- (short) 0xceaf, // (12, 3759) 74
- (short) 0xa38f, // (10, 911) 75
- (short) 0xc1af, // (12, 431) 76
- (short) 0xb1cf, // (11, 463) 77
- (short) 0xc9af, // (12, 2479) 78
- (short) 0xb5cf, // (11, 1487) 79
- (short) 0xc5af, // (12, 1455) 80
- (short) 0xb3cf, // (11, 975) 81
- (short) 0xcdaf, // (12, 3503) 82
- (short) 0xb7cf, // (11, 1999) 83
- (short) 0xc3af, // (12, 943) 84
- (short) 0xcbaf, // (12, 2991) 85
- (short) 0xc7af, // (12, 1967) 86
- (short) 0xcfaf, // (12, 4015) 87
- (short) 0xc06f, // (12, 111) 88
- (short) 0xc86f, // (12, 2159) 89
- (short) 0xc46f, // (12, 1135) 90
- (short) 0xcc6f, // (12, 3183) 91
- (short) 0xc26f, // (12, 623) 92
- (short) 0xca6f, // (12, 2671) 93
- (short) 0xc66f, // (12, 1647) 94
- (short) 0xce6f, // (12, 3695) 95
- (short) 0xc16f, // (12, 367) 96
- (short) 0xc96f, // (12, 2415) 97
- (short) 0xc56f, // (12, 1391) 98
- (short) 0xcd6f, // (12, 3439) 99
- (short) 0xc36f, // (12, 879) 100
- (short) 0xcb6f, // (12, 2927) 101
- (short) 0xc76f, // (12, 1903) 102
- (short) 0xcf6f, // (12, 3951) 103
- (short) 0xc0ef, // (12, 239) 104
- (short) 0xc8ef, // (12, 2287) 105
- (short) 0xc4ef, // (12, 1263) 106
- (short) 0xccef, // (12, 3311) 107
- (short) 0xc2ef, // (12, 751) 108
- (short) 0xcaef, // (12, 2799) 109
- (short) 0xc6ef, // (12, 1775) 110
- (short) 0xceef, // (12, 3823) 111
- (short) 0xc1ef, // (12, 495) 112
- (short) 0xc9ef, // (12, 2543) 113
- (short) 0xc5ef, // (12, 1519) 114
- (short) 0xcdef, // (12, 3567) 115
- (short) 0xc3ef, // (12, 1007) 116
- (short) 0xcbef, // (12, 3055) 117
- (short) 0xc7ef, // (12, 2031) 118
- (short) 0xcfef, // (12, 4079) 119
- (short) 0xc01f, // (12, 31) 120
- (short) 0xc81f, // (12, 2079) 121
- (short) 0xc41f, // (12, 1055) 122
- (short) 0xcc1f, // (12, 3103) 123
- (short) 0xc21f, // (12, 543) 124
- (short) 0xca1f, // (12, 2591) 125
- (short) 0xc61f, // (12, 1567) 126
- (short) 0xce1f, // (12, 3615) 127
- (short) 0xb02f, // (11, 47) 128
- (short) 0x9077, // ( 9, 119) 129
- (short) 0xb42f, // (11, 1071) 130
- (short) 0x9177, // ( 9, 375) 131
- (short) 0xc11f, // (12, 287) 132
- (short) 0xa04f, // (10, 79) 133
- (short) 0xc91f, // (12, 2335) 134
- (short) 0xa24f, // (10, 591) 135
- (short) 0xc51f, // (12, 1311) 136
- (short) 0xb22f, // (11, 559) 137
- (short) 0xcd1f, // (12, 3359) 138
- (short) 0xb62f, // (11, 1583) 139
- (short) 0xc31f, // (12, 799) 140
- (short) 0xcb1f, // (12, 2847) 141
- (short) 0xc71f, // (12, 1823) 142
- (short) 0xcf1f, // (12, 3871) 143
- (short) 0xc09f, // (12, 159) 144
- (short) 0xc89f, // (12, 2207) 145
- (short) 0xc49f, // (12, 1183) 146
- (short) 0xcc9f, // (12, 3231) 147
- (short) 0xc29f, // (12, 671) 148
- (short) 0xca9f, // (12, 2719) 149
- (short) 0xc69f, // (12, 1695) 150
- (short) 0xce9f, // (12, 3743) 151
- (short) 0xc19f, // (12, 415) 152
- (short) 0xc99f, // (12, 2463) 153
- (short) 0xc59f, // (12, 1439) 154
- (short) 0xcd9f, // (12, 3487) 155
- (short) 0xc39f, // (12, 927) 156
- (short) 0xcb9f, // (12, 2975) 157
- (short) 0xc79f, // (12, 1951) 158
- (short) 0xcf9f, // (12, 3999) 159
- (short) 0xc05f, // (12, 95) 160
- (short) 0xc85f, // (12, 2143) 161
- (short) 0xc45f, // (12, 1119) 162
- (short) 0xcc5f, // (12, 3167) 163
- (short) 0xc25f, // (12, 607) 164
- (short) 0xca5f, // (12, 2655) 165
- (short) 0xc65f, // (12, 1631) 166
- (short) 0xce5f, // (12, 3679) 167
- (short) 0xc15f, // (12, 351) 168
- (short) 0xc95f, // (12, 2399) 169
- (short) 0xc55f, // (12, 1375) 170
- (short) 0xcd5f, // (12, 3423) 171
- (short) 0xc35f, // (12, 863) 172
- (short) 0xcb5f, // (12, 2911) 173
- (short) 0xc75f, // (12, 1887) 174
- (short) 0xcf5f, // (12, 3935) 175
- (short) 0xc0df, // (12, 223) 176
- (short) 0xc8df, // (12, 2271) 177
- (short) 0xc4df, // (12, 1247) 178
- (short) 0xccdf, // (12, 3295) 179
- (short) 0xc2df, // (12, 735) 180
- (short) 0xcadf, // (12, 2783) 181
- (short) 0xc6df, // (12, 1759) 182
- (short) 0xcedf, // (12, 3807) 183
- (short) 0xc1df, // (12, 479) 184
- (short) 0xc9df, // (12, 2527) 185
- (short) 0xc5df, // (12, 1503) 186
- (short) 0xcddf, // (12, 3551) 187
- (short) 0xc3df, // (12, 991) 188
- (short) 0xcbdf, // (12, 3039) 189
- (short) 0xc7df, // (12, 2015) 190
- (short) 0xcfdf, // (12, 4063) 191
- (short) 0xc03f, // (12, 63) 192
- (short) 0xc83f, // (12, 2111) 193
- (short) 0xc43f, // (12, 1087) 194
- (short) 0xcc3f, // (12, 3135) 195
- (short) 0xc23f, // (12, 575) 196
- (short) 0xca3f, // (12, 2623) 197
- (short) 0xc63f, // (12, 1599) 198
- (short) 0xce3f, // (12, 3647) 199
- (short) 0xc13f, // (12, 319) 200
- (short) 0xc93f, // (12, 2367) 201
- (short) 0xc53f, // (12, 1343) 202
- (short) 0xcd3f, // (12, 3391) 203
- (short) 0xc33f, // (12, 831) 204
- (short) 0xcb3f, // (12, 2879) 205
- (short) 0xc73f, // (12, 1855) 206
- (short) 0xcf3f, // (12, 3903) 207
- (short) 0xc0bf, // (12, 191) 208
- (short) 0xc8bf, // (12, 2239) 209
- (short) 0xc4bf, // (12, 1215) 210
- (short) 0xccbf, // (12, 3263) 211
- (short) 0xc2bf, // (12, 703) 212
- (short) 0xcabf, // (12, 2751) 213
- (short) 0xc6bf, // (12, 1727) 214
- (short) 0xcebf, // (12, 3775) 215
- (short) 0xc1bf, // (12, 447) 216
- (short) 0xc9bf, // (12, 2495) 217
- (short) 0xc5bf, // (12, 1471) 218
- (short) 0xcdbf, // (12, 3519) 219
- (short) 0xc3bf, // (12, 959) 220
- (short) 0xcbbf, // (12, 3007) 221
- (short) 0xc7bf, // (12, 1983) 222
- (short) 0xcfbf, // (12, 4031) 223
- (short) 0xc07f, // (12, 127) 224
- (short) 0xc87f, // (12, 2175) 225
- (short) 0xc47f, // (12, 1151) 226
- (short) 0xcc7f, // (12, 3199) 227
- (short) 0xc27f, // (12, 639) 228
- (short) 0xca7f, // (12, 2687) 229
- (short) 0xc67f, // (12, 1663) 230
- (short) 0xce7f, // (12, 3711) 231
- (short) 0xc17f, // (12, 383) 232
- (short) 0xc97f, // (12, 2431) 233
- (short) 0xc57f, // (12, 1407) 234
- (short) 0xcd7f, // (12, 3455) 235
- (short) 0xc37f, // (12, 895) 236
- (short) 0xcb7f, // (12, 2943) 237
- (short) 0xc77f, // (12, 1919) 238
- (short) 0xcf7f, // (12, 3967) 239
- (short) 0xc0ff, // (12, 255) 240
- (short) 0xc8ff, // (12, 2303) 241
- (short) 0xc4ff, // (12, 1279) 242
- (short) 0xccff, // (12, 3327) 243
- (short) 0xc2ff, // (12, 767) 244
- (short) 0xcaff, // (12, 2815) 245
- (short) 0xc6ff, // (12, 1791) 246
- (short) 0xceff, // (12, 3839) 247
- (short) 0xc1ff, // (12, 511) 248
- (short) 0xc9ff, // (12, 2559) 249
- (short) 0xc5ff, // (12, 1535) 250
- (short) 0xcdff, // (12, 3583) 251
- (short) 0xc3ff, // (12, 1023) 252
- (short) 0xcbff, // (12, 3071) 253
- (short) 0xc7ff, // (12, 2047) 254
- (short) 0xcfff // (12, 4095) 255
- }
- };
-
- /**
- * Notice that there are only 65 symbols here, which is different from our
- * usual 8 to 12 coding scheme which handles 256 symbols.
- */
- static short[] lengthLimitedUnaryDecodingTable65 = null;
-
- static short[] lengthLimitedUnaryEncodingTable65 = new short[] //[65]
- {
- // Length-limited "unary" code with 65 symbols.
- // entropy: 2.0
- // avg_length: 2.0249023437500000000; max_length = 12; num_symbols = 65
-
- //table, (4 bits, 12 bits) symbol
- //entry, (length, codeword) [byte]
- (short) 0x1000, // ( 1, 0) 0
- (short) 0x2001, // ( 2, 1) 1
- (short) 0x3003, // ( 3, 3) 2
- (short) 0x4007, // ( 4, 7) 3
- (short) 0x500f, // ( 5, 15) 4
- (short) 0x701f, // ( 7, 31) 5
- (short) 0x805f, // ( 8, 95) 6
- (short) 0x80df, // ( 8, 223) 7
- (short) 0xa03f, // (10, 63) 8
- (short) 0xa23f, // (10, 575) 9
- (short) 0xb13f, // (11, 319) 10
- (short) 0xc53f, // (12, 1343) 11
- (short) 0xcd3f, // (12, 3391) 12
- (short) 0xc33f, // (12, 831) 13
- (short) 0xcb3f, // (12, 2879) 14
- (short) 0xc73f, // (12, 1855) 15
- (short) 0xcf3f, // (12, 3903) 16
- (short) 0xc0bf, // (12, 191) 17
- (short) 0xc8bf, // (12, 2239) 18
- (short) 0xc4bf, // (12, 1215) 19
- (short) 0xccbf, // (12, 3263) 20
- (short) 0xc2bf, // (12, 703) 21
- (short) 0xcabf, // (12, 2751) 22
- (short) 0xc6bf, // (12, 1727) 23
- (short) 0xcebf, // (12, 3775) 24
- (short) 0xc1bf, // (12, 447) 25
- (short) 0xc9bf, // (12, 2495) 26
- (short) 0xc5bf, // (12, 1471) 27
- (short) 0xcdbf, // (12, 3519) 28
- (short) 0xc3bf, // (12, 959) 29
- (short) 0xcbbf, // (12, 3007) 30
- (short) 0xc7bf, // (12, 1983) 31
- (short) 0xcfbf, // (12, 4031) 32
- (short) 0xc07f, // (12, 127) 33
- (short) 0xc87f, // (12, 2175) 34
- (short) 0xc47f, // (12, 1151) 35
- (short) 0xcc7f, // (12, 3199) 36
- (short) 0xc27f, // (12, 639) 37
- (short) 0xca7f, // (12, 2687) 38
- (short) 0xc67f, // (12, 1663) 39
- (short) 0xce7f, // (12, 3711) 40
- (short) 0xc17f, // (12, 383) 41
- (short) 0xc97f, // (12, 2431) 42
- (short) 0xc57f, // (12, 1407) 43
- (short) 0xcd7f, // (12, 3455) 44
- (short) 0xc37f, // (12, 895) 45
- (short) 0xcb7f, // (12, 2943) 46
- (short) 0xc77f, // (12, 1919) 47
- (short) 0xcf7f, // (12, 3967) 48
- (short) 0xc0ff, // (12, 255) 49
- (short) 0xc8ff, // (12, 2303) 50
- (short) 0xc4ff, // (12, 1279) 51
- (short) 0xccff, // (12, 3327) 52
- (short) 0xc2ff, // (12, 767) 53
- (short) 0xcaff, // (12, 2815) 54
- (short) 0xc6ff, // (12, 1791) 55
- (short) 0xceff, // (12, 3839) 56
- (short) 0xc1ff, // (12, 511) 57
- (short) 0xc9ff, // (12, 2559) 58
- (short) 0xc5ff, // (12, 1535) 59
- (short) 0xcdff, // (12, 3583) 60
- (short) 0xc3ff, // (12, 1023) 61
- (short) 0xcbff, // (12, 3071) 62
- (short) 0xc7ff, // (12, 2047) 63
- (short) 0xcfff // (12, 4095) 64
- };
-
- /**
- * Note: these column permutations are part of the encoding scheme for sketches where
- * C ≥ 3.375 * K.
- * In each row, we identify the (0-based) column indices of all surprising bits
- * outside of the high-entropy byte.
- *
- * These indices are "rotated right" via the formula
- * new = (old - (8+shift_by) + 64) mod 64 = (old + 56 - shift_by) mod 64.
- * resulting in canonicalized indices between 0 and 55 inclusive.
- *
- * These are then mapped through the forwards permutation specified below (and selected
- * by the phase of C / K). Finally, the remapped indices are encoding with a unary code
- * (with delta encoding for rows containing more than one surprising bit).
- */
- static byte[][] columnPermutationsForDecoding = new byte[16][]; //[16][56]
-
- /**
- * These permutations were created by
- * the ocaml program "generatePermutationsForSLIDING.ml".
- */
- static final byte[][] columnPermutationsForEncoding = new byte[][] //[16] [56]
- {
- // for phase = 1 / 32
- {0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 34, 14, 4},
- // for phase = 3 / 32
- {0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 35, 15, 4},
- // for phase = 5 / 32
- {0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 37, 16, 5},
- // for phase = 7 / 32
- {0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 39, 17, 5},
- // for phase = 9 / 32
- {0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
- 40, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 41, 18, 6},
- // for phase = 11 / 32
- {0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
- 40, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 43, 19, 6},
- // for phase = 13 / 32
- {1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22,
- 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 45, 20, 7, 0},
- // for phase = 15 / 32
- {1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 22,
- 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 48, 49, 50, 51, 52, 53, 54, 55, 47, 21, 7, 0},
- // for phase = 17 / 32
- {1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
- 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 51, 52, 53, 54, 55, 50, 22, 8, 0},
- // for phase = 19 / 32
- {0, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
- 22, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 55, 52, 23, 9, 1},
- // for phase = 21 / 32
- {0, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 25, 9, 1},
- // for phase = 23 / 32
- {0, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 26, 10, 1},
- // for phase = 25 / 32
- {0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 27, 11, 2},
- // for phase = 27 / 32
- {0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 29, 11, 2},
- // for phase = 29 / 32
- {0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 30, 12, 3},
- // for phase = 31 / 32
- {0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15, 16, 17, 18, 19, 20, 21,
- 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 33, 34, 35, 36, 37, 38, 39, 40,
- 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 32, 13, 3}
- };
-
- //Initialize this class
- static {
- makeTheDecodingTables();
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/CpcCompression.java b/src/main/java/org/apache/datasketches/cpc/CpcCompression.java
deleted file mode 100644
index aa73b94e7..000000000
--- a/src/main/java/org/apache/datasketches/cpc/CpcCompression.java
+++ /dev/null
@@ -1,840 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static org.apache.datasketches.cpc.CompressionData.columnPermutationsForDecoding;
-import static org.apache.datasketches.cpc.CompressionData.columnPermutationsForEncoding;
-import static org.apache.datasketches.cpc.CompressionData.decodingTablesForHighEntropyByte;
-import static org.apache.datasketches.cpc.CompressionData.encodingTablesForHighEntropyByte;
-import static org.apache.datasketches.cpc.CompressionData.lengthLimitedUnaryDecodingTable65;
-import static org.apache.datasketches.cpc.CompressionData.lengthLimitedUnaryEncodingTable65;
-import static org.apache.datasketches.cpc.PairTable.introspectiveInsertionSort;
-//import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssertEquals;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class CpcCompression {
- //visible for test
- static final int NEXT_WORD_IDX = 0; //ptrArr[NEXT_WORD_IDX]
- static final int BIT_BUF = 1; //ptrArr[BIT_BUF]
- static final int BUF_BITS = 2; //ptrArr[BUF_BITS]
-
- //visible for test
- static void writeUnary(
- final int[] compressedWords,
- final long[] ptrArr,
- final int theValue) {
-
- int nextWordIndex = (int) ptrArr[NEXT_WORD_IDX]; //must be int
- assert (nextWordIndex == ptrArr[NEXT_WORD_IDX]); //catch truncation error
- long bitBuf = ptrArr[BIT_BUF]; //must be long
- int bufBits = (int) ptrArr[BUF_BITS]; //could be byte
-
- assert (compressedWords != null);
- assert (nextWordIndex >= 0);
- assert (bitBuf >= 0);
- assert ((bufBits >= 0) && (bufBits <= 31));
-
- int remaining = theValue;
-
- while (remaining >= 16) {
- remaining -= 16;
- // Here we output 16 zeros, but we don't need to physically write them into bitbuf
- // because it already contains zeros in that region.
- bufBits += 16; // Record the fact that 16 bits of output have occurred.
- //MAYBE_FLUSH_BITBUF(compressedWords, nextWordIndex);
- if (bufBits >= 32) {
- compressedWords[nextWordIndex++] = (int) bitBuf;
- bitBuf >>>= 32;
- bufBits -= 32;
- }
- }
-
- assert (remaining >= 0) && (remaining <= 15);
-
- final long theUnaryCode = 1L << remaining; //must be a long
- bitBuf |= theUnaryCode << bufBits;
- bufBits += (1 + remaining);
- //MAYBE_FLUSH_BITBUF(compressedWords, nextWordIndex);
- if (bufBits >= 32) {
- compressedWords[nextWordIndex++] = (int) bitBuf;
- bitBuf >>>= 32;
- bufBits -= 32;
- }
-
- ptrArr[NEXT_WORD_IDX] = nextWordIndex;
- assert nextWordIndex == ptrArr[NEXT_WORD_IDX]; //catch sign extension error
- ptrArr[BIT_BUF] = bitBuf;
- ptrArr[BUF_BITS] = bufBits;
- }
-
- //visible for test
- static long readUnary(
- final int[] compressedWords,
- final long[] ptrArr) {
-
- int nextWordIndex = (int) ptrArr[NEXT_WORD_IDX]; //must be int
- assert nextWordIndex == ptrArr[NEXT_WORD_IDX]; //catch truncation error
- long bitBuf = ptrArr[BIT_BUF];
- int bufBits = (int) ptrArr[BUF_BITS];
-
- assert compressedWords != null;
- assert nextWordIndex >= 0;
- assert bitBuf >= 0;
- assert bufBits >= 0;
-
- long subTotal = 0;
- int trailingZeros;
-
- //readUnaryLoop:
- while (true) {
- //MAYBE_FILL_BITBUF(compressedWords,nextWordIndex,8); // ensure 8 bits in bit buffer
- if (bufBits < 8) { // Prepare for an 8-bit peek into the bitstream.
- bitBuf |= ((compressedWords[nextWordIndex++] & 0XFFFF_FFFFL) << bufBits);
- bufBits += 32;
- }
-
- // These 8 bits include either all or part of the Unary codeword.
- final int peek8 = (int) (bitBuf & 0XFFL);
- trailingZeros = Math.min(8, Integer.numberOfTrailingZeros(peek8));
-
- assert ((trailingZeros >= 0) && (trailingZeros <= 8)) : "TZ+ " + trailingZeros;
-
- if (trailingZeros == 8) { // The codeword was partial, so read some more.
- subTotal += 8;
- bufBits -= 8;
- bitBuf >>>= 8;
- continue;
- }
- break;
- }
-
- bufBits -= (1 + trailingZeros);
- bitBuf >>>= (1 + trailingZeros);
-
- ptrArr[NEXT_WORD_IDX] = nextWordIndex;
- assert nextWordIndex == ptrArr[NEXT_WORD_IDX]; //catch sign extension error
- ptrArr[BIT_BUF] = bitBuf;
- ptrArr[BUF_BITS] = bufBits;
- return subTotal + trailingZeros;
- }
-
- /**
- * This returns the number of compressedWords that were actually used.
- * @param byteArray input
- * @param numBytesToEncode input
- * @param encodingTable input
- * @param compressedWords output
- * @return the number of compressedWords that were actually used.
- */
- //visible for test
- //It is the caller's responsibility to ensure that the compressedWords array is long enough.
- static int lowLevelCompressBytes(
- final byte[] byteArray, // input
- final int numBytesToEncode, // input, must be an int
- final short[] encodingTable, // input
- final int[] compressedWords) { // output
-
- int nextWordIndex = 0;
- long bitBuf = 0; // bits are packed into this first, then are flushed to compressedWords
- int bufBits = 0; // number of bits currently in bitbuf; must be between 0 and 31
-
- for (int byteIndex = 0; byteIndex < numBytesToEncode; byteIndex++) {
- final int theByte = byteArray[byteIndex] & 0XFF;
- final long codeInfo = (encodingTable[theByte] & 0XFFFFL);
- final long codeVal = codeInfo & 0XFFFL;
- final int codeWordLength = (int) (codeInfo >>> 12);
- bitBuf |= (codeVal << bufBits);
- bufBits += codeWordLength;
- //MAYBE_FLUSH_BITBUF(compressedWords, nextWordIndex);
- if (bufBits >= 32) {
- compressedWords[nextWordIndex++] = (int) bitBuf;
- bitBuf >>>= 32;
- bufBits -= 32;
- }
- }
-
- //Pad the bitstream with 11 zero-bits so that the decompressor's 12-bit peek
- // can't overrun its input.
- bufBits += 11;
- //MAYBE_FLUSH_BITBUF(compressedWords, nextWordIndex);
- if (bufBits >= 32) {
- compressedWords[nextWordIndex++] = (int) bitBuf;
- bitBuf >>>= 32;
- bufBits -= 32;
- }
-
- if (bufBits > 0) { // We are done encoding now, so we flush the bit buffer.
- assert (bufBits < 32);
- compressedWords[nextWordIndex++] = (int) bitBuf;
- }
- return nextWordIndex;
- }
-
- //visible for test
- static void lowLevelUncompressBytes(
- final byte[] byteArray, // output
- final int numBytesToDecode, // input (but refers to the output)
- final short[] decodingTable, // input
- final int[] compressedWords, // input
- final long numCompressedWords) { // input
-
- int byteIndex = 0;
- int nextWordIndex = 0;
- long bitBuf = 0;
- int bufBits = 0;
-
- assert (byteArray != null);
- assert (decodingTable != null);
- assert (compressedWords != null);
-
- for (byteIndex = 0; byteIndex < numBytesToDecode; byteIndex++) {
- //MAYBE_FILL_BITBUF(compressedWords,wordIndex,12); // ensure 12 bits in bit buffer
- if (bufBits < 12) { // Prepare for a 12-bit peek into the bitstream.
- bitBuf |= ((compressedWords[nextWordIndex++] & 0XFFFF_FFFFL) << bufBits);
- bufBits += 32;
- }
-
- // These 12 bits will include an entire Huffman codeword.
- final int peek12 = (int) (bitBuf & 0XFFFL);
- final int lookup = decodingTable[peek12] & 0XFFFF;
- final int codeWordLength = lookup >>> 8;
- final byte decodedByte = (byte) (lookup & 0XFF);
- byteArray[byteIndex] = decodedByte;
- bitBuf >>>= codeWordLength;
- bufBits -= codeWordLength;
- }
-
- // Buffer over-run should be impossible unless there is a bug.
- // However, we might as well check here.
- assert (nextWordIndex <= numCompressedWords);
- }
-
- /**
- * Here "pairs" refers to row/column pairs that specify the positions of surprising values in
- * the bit matrix.
- * @param pairArray input
- * @param numPairsToEncode input
- * @param numBaseBits input
- * @param compressedWords output
- * @return the number of compressedWords actually used
- */
- //visible for test
- static long lowLevelCompressPairs(
- final int[] pairArray, // input
- final int numPairsToEncode, // input
- final int numBaseBits, // input //cannot exceed 63 or 6 bits, could be byte
- final int[] compressedWords) { // output
-
- int pairIndex = 0;
-
- final long[] ptrArr = new long[3];
- int nextWordIndex = 0; //must be int
- long bitBuf = 0; //must be long
- int bufBits = 0; //could be byte
-
- final long golombLoMask = (1L << numBaseBits) - 1L;
-
- int predictedRowIndex = 0;
- int predictedColIndex = 0;
-
- for (pairIndex = 0; pairIndex < numPairsToEncode; pairIndex++) {
- final int rowCol = pairArray[pairIndex];
- final int rowIndex = rowCol >>> 6;
- final int colIndex = rowCol & 0X3F; //63
-
- if (rowIndex != predictedRowIndex) { predictedColIndex = 0; }
-
- assert (rowIndex >= predictedRowIndex);
- assert (colIndex >= predictedColIndex);
-
- final long yDelta = rowIndex - predictedRowIndex; //cannot exceed 2^26
- final int xDelta = colIndex - predictedColIndex; //cannot exceed 65
-
- predictedRowIndex = rowIndex;
- predictedColIndex = colIndex + 1;
-
- final long codeInfo = lengthLimitedUnaryEncodingTable65[xDelta] & 0XFFFFL;
- final long codeVal = codeInfo & 0XFFFL;
- final int codeLen = (int) (codeInfo >>> 12);
- bitBuf |= (codeVal << bufBits);
- bufBits += codeLen;
- //MAYBE_FLUSH_BITBUF(compressedWords, nextWordIndex);
- if (bufBits >= 32) {
- compressedWords[nextWordIndex++] = (int) bitBuf;
- bitBuf >>>= 32;
- bufBits -= 32;
- }
-
- final long golombLo = yDelta & golombLoMask; //long for bitBuf
- final long golombHi = yDelta >>> numBaseBits; //cannot exceed 2^26
-
- // Inline WriteUnary
- ptrArr[NEXT_WORD_IDX] = nextWordIndex;
- ptrArr[BIT_BUF] = bitBuf;
- ptrArr[BUF_BITS] = bufBits;
- assert nextWordIndex == ptrArr[NEXT_WORD_IDX]; //catch sign extension error
-
- writeUnary(compressedWords, ptrArr, (int) golombHi);
-
- nextWordIndex = (int) ptrArr[NEXT_WORD_IDX];
- bitBuf = ptrArr[BIT_BUF];
- bufBits = (int) ptrArr[BUF_BITS];
- assert nextWordIndex == ptrArr[NEXT_WORD_IDX]; //catch truncation error
- //END Inline WriteUnary
-
- bitBuf |= golombLo << bufBits;
- bufBits += numBaseBits;
- //MAYBE_FLUSH_BITBUF(compressedWords, nextWordIndex);
- if (bufBits >= 32) {
- compressedWords[nextWordIndex++] = (int) bitBuf;
- bitBuf >>>= 32;
- bufBits -= 32;
- }
- }
-
- // Pad the bitstream so that the decompressor's 12-bit peek can't overrun its input.
- int padding = 10 - numBaseBits;
- if (padding < 0) { padding = 0; }
- bufBits += padding;
- //MAYBE_FLUSH_BITBUF(compressedWords, nextWordIndex);
- if (bufBits >= 32) {
- compressedWords[nextWordIndex++] = (int) bitBuf;
- bitBuf >>>= 32;
- bufBits -= 32;
- }
-
- if (bufBits > 0) { // We are done encoding now, so we flush the bit buffer.
- assert (bufBits < 32);
- compressedWords[nextWordIndex++] = (int) bitBuf;
- //bitBuf = 0;
- //bufBits = 0; // not really necessary
- }
- return nextWordIndex;
- }
-
- //visible for test
- static void lowLevelUncompressPairs(
- final int[] pairArray, // output
- final int numPairsToDecode, // input, size of output, must be int
- final int numBaseBits, // input, cannot exceed 6 bits
- final int[] compressedWords, // input
- final long numCompressedWords) { // input
-
- int pairIndex = 0;
-
- final long[] ptrArr = new long[3];
- int nextWordIndex = 0;
- long bitBuf = 0;
- int bufBits = 0;
-
- final long golombLoMask = (1L << numBaseBits) - 1L;
-
- int predictedRowIndex = 0;
- int predictedColIndex = 0;
-
- // for each pair we need to read:
- // xDelta (12-bit length-limited unary)
- // yDeltaHi (unary)
- // yDeltaLo (basebits)
-
- for (pairIndex = 0; pairIndex < numPairsToDecode; pairIndex++) {
-
- //MAYBE_FILL_BITBUF(compressedWords,wordIndex,12); // ensure 12 bits in bit buffer
- if (bufBits < 12) { // Prepare for a 12-bit peek into the bitstream.
- bitBuf |= ((compressedWords[nextWordIndex++] & 0XFFFF_FFFFL) << bufBits);
- bufBits += 32;
- }
-
- final int peek12 = (int) (bitBuf & 0XFFFL);
- final int lookup = lengthLimitedUnaryDecodingTable65[peek12] & 0XFFFF;
- final int codeWordLength = lookup >>> 8;
- final int xDelta = lookup & 0XFF;
- bitBuf >>>= codeWordLength;
- bufBits -= codeWordLength;
-
- // Inline ReadUnary
- ptrArr[NEXT_WORD_IDX] = nextWordIndex;
- ptrArr[BIT_BUF] = bitBuf;
- ptrArr[BUF_BITS] = bufBits;
- assert nextWordIndex == ptrArr[NEXT_WORD_IDX]; //catch sign extension error
-
- final long golombHi = readUnary(compressedWords, ptrArr);
-
- nextWordIndex = (int) ptrArr[NEXT_WORD_IDX];
- bitBuf = ptrArr[BIT_BUF];
- bufBits = (int) ptrArr[BUF_BITS];
- assert nextWordIndex == ptrArr[NEXT_WORD_IDX]; //catch truncation error
- //END Inline ReadUnary
-
- //MAYBE_FILL_BITBUF(compressedWords,wordIndex,numBaseBits); // ensure numBaseBits in bit buffer
- if (bufBits < numBaseBits) { // Prepare for a numBaseBits peek into the bitstream.
- bitBuf |= ((compressedWords[nextWordIndex++] & 0XFFFF_FFFFL) << bufBits);
- bufBits += 32;
- }
-
- final long golombLo = bitBuf & golombLoMask;
- bitBuf >>>= numBaseBits;
- bufBits -= numBaseBits;
- final long yDelta = (golombHi << numBaseBits) | golombLo;
-
- // Now that we have yDelta and xDelta, we can compute the pair's row and column.
- if (yDelta > 0) { predictedColIndex = 0; }
- final int rowIndex = predictedRowIndex + (int) yDelta;
- final int colIndex = predictedColIndex + xDelta;
- final int rowCol = (rowIndex << 6) | colIndex;
- pairArray[pairIndex] = rowCol;
- predictedRowIndex = rowIndex;
- predictedColIndex = colIndex + 1;
- }
- // check for buffer over-run
- assert (nextWordIndex <= numCompressedWords)
- : "nextWdIdx: " + nextWordIndex + ", #CompWds: " + numCompressedWords;
- }
-
- private static int safeLengthForCompressedPairBuf(
- final long k, final long numPairs, final long numBaseBits) {
- assert (numPairs > 0);
- // long ybits = k + numPairs; // simpler and safer UB
- // The following tighter UB on ybits is based on page 198
- // of the textbook "Managing Gigabytes" by Witten, Moffat, and Bell.
- // Notice that if numBaseBits == 0 it coincides with (k + numPairs).
- final long ybits = (numPairs * (1L + numBaseBits)) + (k >>> numBaseBits);
- final long xbits = 12 * numPairs;
- long padding = 10L - numBaseBits;
- if (padding < 0) { padding = 0; }
- final long bits = xbits + ybits + padding;
- //final long words = divideLongsRoundingUp(bits, 32);
- final long words = CpcCompression.divideBy32RoundingUp(bits);
- assert words < (1L << 31);
- return (int) words;
- }
-
- // Explanation of padding: we write
- // 1) xdelta (huffman, provides at least 1 bit, requires 12-bit lookahead)
- // 2) ydeltaGolombHi (unary, provides at least 1 bit, requires 8-bit lookahead)
- // 3) ydeltaGolombLo (straight B bits).
- // So the 12-bit lookahead is the tight constraint, but there are at least (2 + B) bits emitted,
- // so we would be safe with max (0, 10 - B) bits of padding at the end of the bitstream.
- private static int safeLengthForCompressedWindowBuf(final long k) { // measured in 32-bit words
- // 11 bits of padding, due to 12-bit lookahead, with 1 bit certainly present.
- final long bits = (12 * k) + 11;
- //cannot exceed Integer.MAX_VALUE
- //return (int) (divideLongsRoundingUp(bits, 32));
- return (int) CpcCompression.divideBy32RoundingUp(bits);
- }
-
- private static int determinePseudoPhase(final int lgK, final long numCoupons) {
- final long k = 1L << lgK;
- final long c = numCoupons;
- // This midrange logic produces pseudo-phases. They are used to select encoding tables.
- // The thresholds were chosen by hand after looking at plots of measured compression.
- if ((1000 * c) < (2375 * k)) {
- if ( (4 * c) < (3 * k)) { return ( 16 + 0 ); } // midrange table
- else if ( (10 * c) < (11 * k)) { return ( 16 + 1 ); } // midrange table
- else if ( (100 * c) < (132 * k)) { return ( 16 + 2 ); } // midrange table
- else if ( (3 * c) < (5 * k)) { return ( 16 + 3 ); } // midrange table
- else if ((1000 * c) < (1965 * k)) { return ( 16 + 4 ); } // midrange table
- else if ((1000 * c) < (2275 * k)) { return ( 16 + 5 ); } // midrange table
- else { return 6; } // steady-state table employed before its actual phase
- }
- else {
- // This steady-state logic produces true phases. They are used to select
- // encoding tables, and also column permutations for the "Sliding" flavor.
- assert lgK >= 4;
- final long tmp = c >>> (lgK - 4);
- final int phase = (int) (tmp & 15L);
- assert (phase >= 0) && (phase < 16);
- return phase;
- }
- }
-
- private static void compressTheWindow(final CompressedState target, final CpcSketch source) {
- final int srcLgK = source.lgK;
- final int srcK = 1 << srcLgK;
- final int windowBufLen = safeLengthForCompressedWindowBuf(srcK);
- final int[] windowBuf = new int[windowBufLen];
- final int pseudoPhase = determinePseudoPhase(srcLgK, source.numCoupons);
- target.cwLengthInts = lowLevelCompressBytes(
- source.slidingWindow,
- srcK,
- encodingTablesForHighEntropyByte[pseudoPhase],
- windowBuf);
-
- // At this point we free the unused portion of the compression output buffer.
- // final int[] shorterBuf = Arrays.copyOf(windowBuf, target.cwLength);
- // target.compressedWindow = shorterBuf;
- target.cwStream = windowBuf; //avoid extra copy
- }
-
- private static void uncompressTheWindow(final CpcSketch target, final CompressedState source) {
- final int srcLgK = source.lgK;
- final int srcK = 1 << srcLgK;
- final byte[] window = new byte[srcK];
- // bzero ((void *) window, (size_t) k); // zeroing not needed here (unlike the Hybrid Flavor)
- assert (target.slidingWindow == null);
- target.slidingWindow = window;
- final int pseudoPhase = determinePseudoPhase(srcLgK, source.numCoupons);
- assert (source.cwStream != null);
- lowLevelUncompressBytes(target.slidingWindow, srcK,
- decodingTablesForHighEntropyByte[pseudoPhase],
- source.cwStream,
- source.cwLengthInts);
- }
-
- private static void compressTheSurprisingValues(final CompressedState target, final CpcSketch source,
- final int[] pairs, final int numPairs) {
- assert (numPairs > 0);
- target.numCsv = numPairs;
- final int srcK = 1 << source.lgK;
- final int numBaseBits = CpcCompression.golombChooseNumberOfBaseBits(srcK + numPairs, numPairs);
- final int pairBufLen = safeLengthForCompressedPairBuf(srcK, numPairs, numBaseBits);
- final int[] pairBuf = new int[pairBufLen];
-
- target.csvLengthInts = (int) lowLevelCompressPairs(pairs, numPairs, numBaseBits, pairBuf);
-
- // At this point we free the unused portion of the compression output buffer.
- // final int[] shorterBuf = Arrays.copyOf(pairBuf, target.csvLength);
- // target.compressedWindow = shorterBuf;
-
- target.csvStream = pairBuf; //avoid extra copy
- }
-
- //allocates and returns an array of uncompressed pairs.
- //the length of this array is known to the source sketch.
- private static int[] uncompressTheSurprisingValues(final CompressedState source) {
- final int srcK = 1 << source.lgK;
- final int numPairs = source.numCsv;
- assert numPairs > 0;
- final int[] pairs = new int[numPairs];
- final int numBaseBits = CpcCompression.golombChooseNumberOfBaseBits(srcK + numPairs, numPairs);
- lowLevelUncompressPairs(pairs, numPairs, numBaseBits, source.csvStream, source.csvLengthInts);
- return pairs;
- }
-
- private static void compressSparseFlavor(final CompressedState target, final CpcSketch source) {
- assert (source.slidingWindow == null); //there is no window to compress
- final PairTable srcPairTable = source.pairTable;
- final int srcNumPairs = srcPairTable.getNumPairs();
- final int[] srcPairArr = PairTable.unwrappingGetItems(srcPairTable, srcNumPairs);
- introspectiveInsertionSort(srcPairArr, 0, srcNumPairs - 1);
- compressTheSurprisingValues(target, source, srcPairArr, srcNumPairs);
- }
-
- private static void uncompressSparseFlavor(final CpcSketch target, final CompressedState source) {
- assert (source.cwStream == null);
- assert (source.csvStream != null);
- final int[] srcPairArr = uncompressTheSurprisingValues(source);
- final int numPairs = source.numCsv;
- final PairTable table = PairTable.newInstanceFromPairsArray(srcPairArr, numPairs, source.lgK);
- target.pairTable = table;
- }
-
- //The empty space that this leaves at the beginning of the output array
- // will be filled in later by the caller.
- private static int[] trickyGetPairsFromWindow(final byte[] window, final int k, final int numPairsToGet,
- final int emptySpace) {
- final int outputLength = emptySpace + numPairsToGet;
- final int[] pairs = new int[outputLength];
- int rowIndex = 0;
- int pairIndex = emptySpace;
- for (rowIndex = 0; rowIndex < k; rowIndex++) {
- int wByte = window[rowIndex] & 0XFF;
- while (wByte != 0) {
- final int colIndex = Integer.numberOfTrailingZeros(wByte);
- // assert (colIndex < 8);
- wByte ^= (1 << colIndex); // erase the 1
- pairs[pairIndex++] = (rowIndex << 6) | colIndex;
- }
- }
- assert (pairIndex == outputLength);
- return (pairs);
- }
-
- //This is complicated because it effectively builds a Sparse version
- //of a Pinned sketch before compressing it. Hence the name Hybrid.
- private static void compressHybridFlavor(final CompressedState target, final CpcSketch source) {
- final int srcK = 1 << source.lgK;
- final PairTable srcPairTable = source.pairTable;
- final int srcNumPairs = srcPairTable.getNumPairs();
- final int[] srcPairArr = PairTable.unwrappingGetItems(srcPairTable, srcNumPairs);
- introspectiveInsertionSort(srcPairArr, 0, srcNumPairs - 1);
- final byte[] srcSlidingWindow = source.slidingWindow;
- final int srcWindowOffset = source.windowOffset;
- final long srcNumCoupons = source.numCoupons;
- assert (srcSlidingWindow != null);
- assert (srcWindowOffset == 0);
- final long numPairs = srcNumCoupons - srcNumPairs; // because the window offset is zero
- assert numPairs < Integer.MAX_VALUE;
- final int numPairsFromArray = (int) numPairs;
-
- assert (numPairsFromArray + srcNumPairs) == srcNumCoupons; //for test
- final int[] allPairs
- = trickyGetPairsFromWindow(srcSlidingWindow, srcK, numPairsFromArray, srcNumPairs);
-
- PairTable.merge(srcPairArr, 0, srcNumPairs,
- allPairs, srcNumPairs, numPairsFromArray,
- allPairs, 0); // note the overlapping subarray trick
-
- //FOR TESTING If needed
- // for (int i = 0; i < (source.numCoupons - 1); i++) {
- // assert (Integer.compareUnsigned(allPairs[i], allPairs[i + 1]) < 0); }
-
- compressTheSurprisingValues(target, source, allPairs, (int) srcNumCoupons);
- }
-
- private static void uncompressHybridFlavor(final CpcSketch target, final CompressedState source) {
- assert (source.cwStream == null);
- assert (source.csvStream != null);
- final int[] pairs = uncompressTheSurprisingValues(source); //fail path 3
- final int numPairs = source.numCsv;
- // In the hybrid flavor, some of these pairs actually
- // belong in the window, so we will separate them out,
- // moving the "true" pairs to the bottom of the array.
- final int srcLgK = source.lgK;
- final int k = 1 << srcLgK;
-
- final byte[] window = new byte[k];
-
- int nextTruePair = 0;
-
- for (int i = 0; i < numPairs; i++) {
- final int rowCol = pairs[i];
- assert (rowCol != -1);
- final int col = rowCol & 63;
- if (col < 8) {
- final int row = rowCol >>> 6;
- window[row] |= (1 << col); // set the window bit
- }
- else {
- pairs[nextTruePair++] = rowCol; // move true pair down
- }
- }
-
- assert (source.getWindowOffset() == 0);
- target.windowOffset = 0;
-
- final PairTable table = PairTable.newInstanceFromPairsArray(pairs, nextTruePair, srcLgK);
- target.pairTable = table;
- target.slidingWindow = window;
- }
-
- private static void compressPinnedFlavor(final CompressedState target, final CpcSketch source) {
- compressTheWindow(target, source);
- final PairTable srcPairTable = source.pairTable;
- final int numPairs = srcPairTable.getNumPairs();
-
- if (numPairs > 0) {
- final int[] pairs = PairTable.unwrappingGetItems(srcPairTable, numPairs);
-
- // Here we subtract 8 from the column indices. Because they are stored in the low 6 bits
- // of each rowCol pair, and because no column index is less than 8 for a "Pinned" sketch,
- // I believe we can simply subtract 8 from the pairs themselves.
-
- // shift the columns over by 8 positions before compressing (because of the window)
- for (int i = 0; i < numPairs; i++) {
- assert (pairs[i] & 63) >= 8;
- pairs[i] -= 8;
- }
-
- introspectiveInsertionSort(pairs, 0, numPairs - 1);
- compressTheSurprisingValues(target, source, pairs, numPairs);
- }
- }
-
- private static void uncompressPinnedFlavor(final CpcSketch target, final CompressedState source) {
- assert (source.cwStream != null);
- uncompressTheWindow(target, source);
- final int srcLgK = source.lgK;
- final int numPairs = source.numCsv;
- if (numPairs == 0) {
- target.pairTable = new PairTable(2, 6 + srcLgK);
- }
- else {
- assert numPairs > 0;
- assert source.csvStream != null;
- final int[] pairs = uncompressTheSurprisingValues(source);
- // undo the compressor's 8-column shift
- for (int i = 0; i < numPairs; i++) {
- assert (pairs[i] & 63) < 56;
- pairs[i] += 8;
- }
- final PairTable table = PairTable.newInstanceFromPairsArray(pairs, numPairs, srcLgK);
- target.pairTable = table;
- }
- }
-
- //Complicated by the existence of both a left fringe and a right fringe.
- private static void compressSlidingFlavor(final CompressedState target, final CpcSketch source) {
-
- compressTheWindow(target, source);
- final PairTable srcPairTable = source.pairTable;
-
- final int numPairs = srcPairTable.getNumPairs();
-
- if (numPairs > 0) {
- final int[] pairs = PairTable.unwrappingGetItems(srcPairTable, numPairs);
-
- // Here we apply a complicated transformation to the column indices, which
- // changes the implied ordering of the pairs, so we must do it before sorting.
-
- final int pseudoPhase = determinePseudoPhase(source.lgK, source.numCoupons); // NB
- assert (pseudoPhase < 16);
- final byte[] permutation = columnPermutationsForEncoding[pseudoPhase];
-
- final int offset = source.windowOffset;
- assert ((offset > 0) && (offset <= 56));
-
- for (int i = 0; i < numPairs; i++) {
- final int rowCol = pairs[i];
- final int row = rowCol >>> 6;
- int col = (rowCol & 63);
- // first rotate the columns into a canonical configuration:
- // new = ((old - (offset+8)) + 64) mod 64
- col = ((col + 56) - offset) & 63;
- assert (col >= 0) && (col < 56);
- // then apply the permutation
- col = permutation[col];
- pairs[i] = (row << 6) | col;
- }
-
- introspectiveInsertionSort(pairs, 0, numPairs - 1);
- compressTheSurprisingValues(target, source, pairs, numPairs);
- }
- }
-
- private static void uncompressSlidingFlavor(final CpcSketch target, final CompressedState source) {
- assert (source.cwStream != null);
- uncompressTheWindow(target, source);
- final int srcLgK = source.lgK;
- final int numPairs = source.numCsv;
- if (numPairs == 0) {
- target.pairTable = new PairTable(2, 6 + srcLgK);
- }
- else {
- assert (numPairs > 0);
- assert (source.csvStream != null);
- final int[] pairs = uncompressTheSurprisingValues(source);
- final int pseudoPhase = determinePseudoPhase(srcLgK, source.numCoupons); // NB
- assert (pseudoPhase < 16);
- final byte[] permutation = columnPermutationsForDecoding[pseudoPhase];
-
- final int offset = source.getWindowOffset();
- assert (offset > 0) && (offset <= 56);
-
- for (int i = 0; i < numPairs; i++) {
- final int rowCol = pairs[i];
- final int row = rowCol >>> 6;
- int col = rowCol & 63;
- // first undo the permutation
- col = permutation[col];
- // then undo the rotation: old = (new + (offset+8)) mod 64
- col = (col + (offset + 8)) & 63;
- pairs[i] = (row << 6) | col;
- }
-
- final PairTable table = PairTable.newInstanceFromPairsArray(pairs, numPairs, srcLgK);
- target.pairTable = table;
- }
- }
-
- static CompressedState compress(final CpcSketch source, final CompressedState target) {
-
- final Flavor srcFlavor = source.getFlavor();
-
- switch (srcFlavor) {
- case EMPTY: break;
- case SPARSE:
- compressSparseFlavor(target, source);
- assert (target.cwStream == null);
- assert (target.csvStream != null);
- break;
- case HYBRID:
- compressHybridFlavor(target, source);
- assert (target.cwStream == null);
- assert (target.csvStream != null);
- break;
- case PINNED:
- compressPinnedFlavor(target, source);
- assert (target.cwStream != null);
- break;
- case SLIDING:
- compressSlidingFlavor(target, source);
- assert (target.cwStream != null);
- break;
- //default: not possible
- }
-
- return target;
- }
-
- static CpcSketch uncompress(final CompressedState source, final CpcSketch target) {
- assert (target != null);
-
- final Flavor srcFlavor = source.getFlavor();
- switch (srcFlavor) {
- case EMPTY: break;
- case SPARSE:
- assert (source.cwStream == null);
- uncompressSparseFlavor(target, source);
- break;
- case HYBRID:
- uncompressHybridFlavor(target, source);
- break;
- case PINNED:
- assert (source.cwStream != null);
- uncompressPinnedFlavor(target, source);
- break;
- case SLIDING:
- uncompressSlidingFlavor(target, source);
- break;
- //default: not possible
- }
- return target;
- }
-
- private static int golombChooseNumberOfBaseBits(final int k, final long count) {
- assert k >= 1L;
- assert count >= 1L;
- final long quotient = (k - count) / count; // integer division
- if (quotient == 0) { return 0; }
- return (int) floorLog2ofLong(quotient);
- }
-
- private static long floorLog2ofLong(final long x) { //not a good name
- assert (x >= 1L);
- long p = 0;
- long y = 1;
- while (true) {
- if (y == x) { return p; }
- if (y > x) { return p - 1; }
- p += 1;
- y <<= 1;
- }
- }
-
- private static long divideBy32RoundingUp(final long x) {
- final long tmp = x >>> 5;
- return ((tmp << 5) == x) ? tmp : tmp + 1;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/CpcConfidence.java b/src/main/java/org/apache/datasketches/cpc/CpcConfidence.java
deleted file mode 100644
index d948678b6..000000000
--- a/src/main/java/org/apache/datasketches/cpc/CpcConfidence.java
+++ /dev/null
@@ -1,160 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static java.lang.Math.ceil;
-import static java.lang.Math.log;
-import static java.lang.Math.sqrt;
-import static org.apache.datasketches.cpc.IconEstimator.getIconEstimate;
-
-/**
- * Tables and methods for estimating upper and lower bounds.
- *
- * Tables were generated from empirical measurements at N = 1000 * K using millions of trials.
- *
- * @author Lee Rhodes
- */
-final class CpcConfidence {
- private static final double iconErrorConstant = log(2.0); //0.693147180559945286
- private static final double hipErrorConstant = sqrt(log(2.0) / 2.0); //0.588705011257737332
-
- static short[] iconLowSideData = {
- //1, 2, 3, kappa
- // lgK numtrials
- 6037, 5720, 5328, // 4 1000000
- 6411, 6262, 5682, // 5 1000000
- 6724, 6403, 6127, // 6 1000000
- 6665, 6411, 6208, // 7 1000000
- 6959, 6525, 6427, // 8 1000000
- 6892, 6665, 6619, // 9 1000000
- 6792, 6752, 6690, // 10 1000000
- 6899, 6818, 6708, // 11 1000000
- 6871, 6845, 6812, // 12 1046369
- 6909, 6861, 6828, // 13 1043411
- 6919, 6897, 6842, // 14 1000297
- };
-
- static short[] iconHighSideData = {
- //1, 2, 3, kappa
- // lgK numtrials
- 8031, 8559, 9309, // 4 1000000
- 7084, 7959, 8660, // 5 1000000
- 7141, 7514, 7876, // 6 1000000
- 7458, 7430, 7572, // 7 1000000
- 6892, 7141, 7497, // 8 1000000
- 6889, 7132, 7290, // 9 1000000
- 7075, 7118, 7185, // 10 1000000
- 7040, 7047, 7085, // 11 1000000
- 6993, 7019, 7053, // 12 1046369
- 6953, 7001, 6983, // 13 1043411
- 6944, 6966, 7004, // 14 1000297
- };
-
- static short[] hipLowSideData = {
- //1, 2, 3, kappa
- // lgK numtrials
- 5871, 5247, 4826, // 4 1000000
- 5877, 5403, 5070, // 5 1000000
- 5873, 5533, 5304, // 6 1000000
- 5878, 5632, 5464, // 7 1000000
- 5874, 5690, 5564, // 8 1000000
- 5880, 5745, 5619, // 9 1000000
- 5875, 5784, 5701, // 10 1000000
- 5866, 5789, 5742, // 11 1000000
- 5869, 5827, 5784, // 12 1046369
- 5876, 5860, 5827, // 13 1043411
- 5881, 5853, 5842, // 14 1000297
- };
-
- static short[] hipHighSideData = {
- //1, 2, 3, kappa
- // lgK numtrials
- 5855, 6688, 7391, // 4 1000000
- 5886, 6444, 6923, // 5 1000000
- 5885, 6254, 6594, // 6 1000000
- 5889, 6134, 6326, // 7 1000000
- 5900, 6072, 6203, // 8 1000000
- 5875, 6005, 6089, // 9 1000000
- 5871, 5980, 6040, // 10 1000000
- 5889, 5941, 6015, // 11 1000000
- 5871, 5926, 5973, // 12 1046369
- 5866, 5901, 5915, // 13 1043411
- 5880, 5914, 5953, // 14 1000297
- };
-
- static double getIconConfidenceLB(final int lgK, final long numCoupons, final int kappa) {
- if (numCoupons == 0) { return 0.0; }
- assert lgK >= 4;
- assert (kappa >= 1) && (kappa <= 3);
- double x = iconErrorConstant;
- if (lgK <= 14) { x = (iconHighSideData[(3 * (lgK - 4)) + (kappa - 1)]) / 10000.0; }
- final double rel = x / sqrt(1 << lgK);
- final double eps = kappa * rel;
- final double est = getIconEstimate(lgK, numCoupons);
- double result = est / (1.0 + eps);
- if (result < numCoupons) { result = numCoupons; }
- return result;
- }
-
- static double getIconConfidenceUB(final int lgK, final long numCoupons, final int kappa) {
- if (numCoupons == 0) { return 0.0; }
- assert lgK >= 4;
- assert (kappa >= 1) && (kappa <= 3);
- double x = iconErrorConstant;
- if (lgK <= 14) { x = (iconLowSideData[(3 * (lgK - 4)) + (kappa - 1)]) / 10000.0; }
- final double rel = x / sqrt(1 << lgK);
- final double eps = kappa * rel;
- final double est = getIconEstimate(lgK, numCoupons);
- final double result = est / (1.0 - eps);
- return ceil(result); // slight widening of interval to be conservative
- }
-
- //mergeFlag must already be checked as false
- static double getHipConfidenceLB(final int lgK, final long numCoupons, final double hipEstAccum,
- final int kappa) {
- if (numCoupons == 0) { return 0.0; }
- assert lgK >= 4;
- assert (kappa >= 1) && (kappa <= 3);
- double x = hipErrorConstant;
- if (lgK <= 14) { x = (hipHighSideData[(3 * (lgK - 4)) + (kappa - 1)]) / 10000.0; }
- final double rel = x / sqrt(1 << lgK);
- final double eps = kappa * rel;
- final double est = hipEstAccum;
- double result = est / (1.0 + eps);
- if (result < numCoupons) { result = numCoupons; }
- return result;
- }
-
- //mergeFlag must already be checked as false
- static double getHipConfidenceUB(final int lgK, final long numCoupons, final double hipEstAccum,
- final int kappa) {
- if (numCoupons == 0) { return 0.0; }
- assert lgK >= 4;
- assert (kappa >= 1) && (kappa <= 3);
- double x = hipErrorConstant;
- if (lgK <= 14) { x = (hipLowSideData[(3 * (lgK - 4)) + (kappa - 1)]) / 10000.0; }
- final double rel = x / sqrt(1 << lgK);
- final double eps = kappa * rel;
- final double est = hipEstAccum;
- final double result = est / (1.0 - eps);
- return ceil(result); // widening for coverage
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/CpcSketch.java b/src/main/java/org/apache/datasketches/cpc/CpcSketch.java
deleted file mode 100644
index d5e17388a..000000000
--- a/src/main/java/org/apache/datasketches/cpc/CpcSketch.java
+++ /dev/null
@@ -1,804 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static java.lang.Math.log;
-import static java.lang.Math.sqrt;
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.common.Util.invPow2;
-import static org.apache.datasketches.common.Util.zeroPad;
-import static org.apache.datasketches.cpc.CpcUtil.bitMatrixOfSketch;
-import static org.apache.datasketches.cpc.CpcUtil.checkLgK;
-import static org.apache.datasketches.cpc.CpcUtil.countBitsSetInMatrix;
-import static org.apache.datasketches.hash.MurmurHash3.hash;
-
-import java.lang.foreign.MemorySegment;
-import java.nio.ByteBuffer;
-import java.util.Arrays;
-
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.Util;
-
-/**
- * This is a unique-counting sketch that implements the
- * Compressed Probabilistic Counting (CPC, a.k.a FM85) algorithms developed by Kevin Lang in
- * his paper
- * Back to the Future: an Even More Nearly
- * Optimal Cardinality Estimation Algorithm.
- *
- * This sketch is extremely space-efficient when serialized. In an apples-to-apples empirical
- * comparison against compressed HyperLogLog sketches, this new algorithm simultaneously wins on
- * the two dimensions of the space/accuracy tradeoff and produces sketches that are
- * smaller than the entropy of HLL, so no possible implementation of compressed HLL can match its
- * space efficiency for a given accuracy. As described in the paper this sketch implements a newly
- * developed ICON estimator algorithm that survives unioning operations, another
- * well-known estimator, the
- * Historical Inverse Probability (HIP) estimator
- * does not.
- * The update speed performance of this sketch is quite fast and is comparable to the speed of HLL.
- * The unioning (merging) capability of this sketch also allows for merging of sketches with
- * different configurations of K.
- *
- * For additional security this sketch can be configured with a user-specified hash seed.
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-public final class CpcSketch {
- private static final double[] kxpByteLookup = new double[256];
-
- /**
- * The default Log_base2 of K
- */
- public static final int DEFAULT_LG_K = 11;
- final long seed;
- //common variables
- final int lgK;
- long numCoupons; // The number of coupons collected so far.
- boolean mergeFlag; // Is the sketch the result of merging?
- int fiCol; // First Interesting Column. This is part of a speed optimization.
-
- int windowOffset;
- byte[] slidingWindow; //either null or size K bytes
- PairTable pairTable; //for sparse and surprising values, either null or variable size
-
- //The following variables are only valid in HIP variants
- double kxp; //used with HIP
- double hipEstAccum; //used with HIP
-
- /**
- * Constructor with default log_base2 of k
- */
- public CpcSketch() {
- this(DEFAULT_LG_K, Util.DEFAULT_UPDATE_SEED);
- }
-
- /**
- * Constructor with log_base2 of k.
- * @param lgK the given log_base2 of k
- */
- public CpcSketch(final int lgK) {
- this(lgK, Util.DEFAULT_UPDATE_SEED);
- }
-
- /**
- * Constructor with log_base2 of k and seed.
- * @param lgK the given log_base2 of k
- * @param seed the given seed
- */
- public CpcSketch(final int lgK, final long seed) {
- checkLgK(lgK);
- this.lgK = (byte) lgK;
- this.seed = seed;
- kxp = 1 << lgK;
- reset();
- }
-
- /**
- * Returns a copy of this sketch
- * @return a copy of this sketch
- */
- CpcSketch copy() {
- final CpcSketch copy = new CpcSketch(lgK, seed);
- copy.numCoupons = numCoupons;
- copy.mergeFlag = mergeFlag;
- copy.fiCol = fiCol;
-
- copy.windowOffset = windowOffset;
- copy.slidingWindow = (slidingWindow == null) ? null : slidingWindow.clone();
- copy.pairTable = (pairTable == null) ? null : pairTable.copy();
-
- copy.kxp = kxp;
- copy.hipEstAccum = hipEstAccum;
- return copy;
- }
-
- /**
- * Returns the best estimate of the cardinality of the sketch.
- * @return the best estimate of the cardinality of the sketch.
- */
- public double getEstimate() {
- if (mergeFlag) { return IconEstimator.getIconEstimate(lgK, numCoupons); }
- return hipEstAccum;
- }
-
- /**
- * Return the DataSketches identifier for this CPC family of sketches.
- * @return the DataSketches identifier for this CPC family of sketches.
- */
- public static Family getFamily() {
- return Family.CPC;
- }
-
- /**
- * Return the parameter LgK.
- * @return the parameter LgK.
- */
- public int getLgK() {
- return lgK;
- }
-
- /**
- * Returns the best estimate of the lower bound of the confidence interval given kappa,
- * the number of standard deviations from the mean.
- * @param kappa the given number of standard deviations from the mean: 1, 2 or 3.
- * @return the best estimate of the lower bound of the confidence interval given kappa.
- */
- public double getLowerBound(final int kappa) {
- if (mergeFlag) {
- return CpcConfidence.getIconConfidenceLB(lgK, numCoupons, kappa);
- }
- return CpcConfidence.getHipConfidenceLB(lgK, numCoupons, hipEstAccum, kappa);
- }
-
- /*
- * These empirical values for the 99.9th percentile of size in bytes were measured using 100,000
- * trials. The value for each trial is the maximum of 5*16=80 measurements that were equally
- * spaced over values of the quantity C/K between 3.0 and 8.0. This table does not include the
- * worst-case space for the preamble, which is added by the function.
- */
- private static final int empiricalSizeMaxLgK = 19;
- private static final int[] empiricalMaxBytes = {
- 24, // lgK = 4
- 36, // lgK = 5
- 56, // lgK = 6
- 100, // lgK = 7
- 180, // lgK = 8
- 344, // lgK = 9
- 660, // lgK = 10
- 1292, // lgK = 11
- 2540, // lgK = 12
- 5020, // lgK = 13
- 9968, // lgK = 14
- 19836, // lgK = 15
- 39532, // lgK = 16
- 78880, // lgK = 17
- 157516, // lgK = 18
- 314656 // lgK = 19
- };
- private static final double empiricalMaxSizeFactor = 0.6; // 0.6 = 4.8 / 8.0
- private static final int maxPreambleSizeBytes = 40;
-
- /**
- * The actual size of a compressed CPC sketch has a small random variance, but the following
- * empirically measured size should be large enough for at least 99.9 percent of sketches.
- *
- * For small values of n the size can be much smaller.
- *
- * @param lgK the given value of lgK.
- * @return the estimated maximum compressed serialized size of a sketch.
- */
- public static int getMaxSerializedBytes(final int lgK) {
- checkLgK(lgK);
- if (lgK <= empiricalSizeMaxLgK) { return empiricalMaxBytes[lgK - CpcUtil.minLgK] + maxPreambleSizeBytes; }
- final int k = 1 << lgK;
- return (int) (empiricalMaxSizeFactor * k) + maxPreambleSizeBytes;
- }
-
- /**
- * Returns the best estimate of the upper bound of the confidence interval given kappa,
- * the number of standard deviations from the mean.
- * @param kappa the given number of standard deviations from the mean: 1, 2 or 3.
- * @return the best estimate of the upper bound of the confidence interval given kappa.
- */
- public double getUpperBound(final int kappa) {
- if (mergeFlag) {
- return CpcConfidence.getIconConfidenceUB(lgK, numCoupons, kappa);
- }
- return CpcConfidence.getHipConfidenceUB(lgK, numCoupons, hipEstAccum, kappa);
- }
-
- /**
- * Return the given MemorySegment as a CpcSketch on the Java heap using the DEFAULT_UPDATE_SEED.
- * @param seg the given MemorySegment
- * @return the given MemorySegment as a CpcSketch on the Java heap.
- */
- public static CpcSketch heapify(final MemorySegment seg) {
- return heapify(seg, Util.DEFAULT_UPDATE_SEED);
- }
-
- /**
- * Return the given byte array as a CpcSketch on the Java heap using the DEFAULT_UPDATE_SEED.
- * @param byteArray the given byte array
- * @return the given byte array as a CpcSketch on the Java heap.
- */
- public static CpcSketch heapify(final byte[] byteArray) {
- return heapify(byteArray, Util.DEFAULT_UPDATE_SEED);
- }
-
- /**
- * Return the given MemorySegment as a CpcSketch on the Java heap.
- * @param seg the given MemorySegment
- * @param seed the seed used to create the original sketch from which the MemorySegment was derived.
- * @return the given MemorySegment as a CpcSketch on the Java heap.
- */
- public static CpcSketch heapify(final MemorySegment seg, final long seed) {
- final CompressedState state = CompressedState.importFromSegment(seg);
- return uncompress(state, seed);
- }
-
- /**
- * Return the given byte array as a CpcSketch on the Java heap.
- * @param byteArray the given byte array
- * @param seed the seed used to create the original sketch from which the byte array was derived.
- * @return the given byte array as a CpcSketch on the Java heap.
- */
- public static CpcSketch heapify(final byte[] byteArray, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(byteArray);
- return heapify(seg, seed);
- }
-
- /**
- * Return true if this sketch is empty
- * @return true if this sketch is empty
- */
- public boolean isEmpty() {
- return numCoupons == 0;
- }
-
- /**
- * Resets this sketch to empty but retains the original LgK and Seed.
- */
- public void reset() {
- numCoupons = 0;
- mergeFlag = false;
- fiCol = 0;
-
- windowOffset = 0;
- slidingWindow = null;
- pairTable = null;
-
- kxp = 1 << lgK;
- hipEstAccum = 0;
- }
-
- /**
- * Return this sketch as a compressed byte array.
- * @return this sketch as a compressed byte array.
- */
- public byte[] toByteArray() {
- final CompressedState state = CompressedState.compress(this);
- final long cap = state.getRequiredSerializedBytes();
- final MemorySegment wseg = MemorySegment.ofArray(new byte[(int) cap]);
- state.exportToSegment(wseg);
- return wseg.toArray(JAVA_BYTE);
- }
-
- /**
- * Present the given long as a potential unique item.
- *
- * @param datum The given long datum.
- */
- public void update(final long datum) {
- final long[] data = { datum };
- final long[] arr = hash(data, seed);
- hashUpdate(arr[0], arr[1]);
- }
-
- /**
- * Present the given double (or float) datum as a potential unique item.
- * The double will be converted to a long using Double.doubleToLongBits(datum),
- * which normalizes all NaN values to a single NaN representation.
- * Plus and minus zero will be normalized to plus zero.
- * The special floating-point values NaN and +/- Infinity are treated as distinct.
- *
- * @param datum The given double datum.
- */
- public void update(final double datum) {
- final double d = (datum == 0.0) ? 0.0 : datum; // canonicalize -0.0, 0.0
- final long[] data = { Double.doubleToLongBits(d) };// canonicalize all NaN forms
- final long[] arr = hash(data, seed);
- hashUpdate(arr[0], arr[1]);
- }
-
- /**
- * Present the given String as a potential unique item.
- * The string is converted to a byte array using UTF8 encoding.
- * If the string is null or empty no update attempt is made and the method returns.
- *
- * Note: About 2X faster performance can be obtained by first converting the String to a
- * char[] and updating the sketch with that. This bypasses the complexity of the Java UTF_8
- * encoding. This, of course, will not produce the same internal hash values as updating directly
- * with a String. So be consistent! Unioning two sketches, one fed with strings and the other
- * fed with char[] will be meaningless.
- * Note: this will not produce the same output hash values as the {@link #update(String)}
- * method but will be a little faster as it avoids the complexity of the UTF8 encoding. If you are starting with a serialized image as a byte array, first heapify
- * the byte array to a sketch, which performs a number of checks. Then use this
- * function as one additional check on the sketch. First, we compare the K values of the union and the source sketch.
- *
- * If (source.K < union.K), we reduce the union's K to match, which
- * requires downsampling the union's internal sketch.
- *
- * Here is how to perform the downsampling.
- *
- * If the union contains a bitMatrix, downsample it by row-wise OR'ing.
- *
- * If the union contains a sparse sketch, then create a new empty
- * sketch, and walk the old target sketch updating the new one (with modulo).
- * At the end, check whether the new target
- * sketch is still in sparse mode (it might not be, because downsampling
- * densifies the set of collected coupons). If it is NOT in sparse mode,
- * immediately convert it to a bitmatrix.
- *
- * At this point, we have source.K ≥ union.K.
- * [We won't keep mentioning this, but in all of the following the
- * source's row indices are used mod union.K while updating the union's sketch.
- * That takes care of the situation where source.K < union.K.]
- *
- * Case A: union is Sparse and source is Sparse. We walk the source sketch
- * updating the union's sketch. At the end, if the union's sketch
- * is no longer in sparse mode, we convert it to a bitmatrix.
- *
- * Case B: union is bitmatrix and source is Sparse. We walk the source sketch,
- * setting bits in the bitmatrix.
- *
- * In the remaining cases, we have flavor(source) > Sparse, so we immediately convert the
- * union's sketch to a bitmatrix (even if the union contains very few coupons). Then:
- *
- * Case C: union is bitmatrix and source is Hybrid or Pinned. Then we OR the source's
- * sliding window into the bitmatrix, and walk the source's table, setting bits in the bitmatrix.
- *
- * Case D: union is bitmatrix, and source is Sliding. Then we convert the source into
- * a bitmatrix, and OR it into the union's bitmatrix. [Important note; merely walking the source
- * wouldn't work because of the partially inverted Logic in the Sliding flavor, where the presence of
- * coupons is sometimes indicated by the ABSENCE of rowCol pairs in the surprises table.]
- *
- * How does getResult work?
- *
- * If the union is using its accumulator field, make a copy of that sketch.
- *
- * If the union is using its bitMatrix field, then we have to convert the
- * bitMatrix back into a sketch, which requires doing some extra work to
- * figure out the values of numCoupons, offset, fiCol, and KxQ.
- *
- */
-/**
- * The union (merge) operation for the CPC sketches.
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-public class CpcUnion {
- private final long seed;
- private int lgK;
-
- // Note: at most one of bitMatrix and accumulator will be non-null at any given moment.
- // accumulator is a sketch object that is employed until it graduates out of Sparse mode.
- // At that point, it is converted into a full-sized bitMatrix, which is mathematically a sketch,
- // but doesn't maintain any of the "extra" fields of our sketch objects, so some additional work
- // is required when getResult is called at the end.
- private long[] bitMatrix;
- private CpcSketch accumulator; //can only be empty or sparse Flavor
-
- /**
- * Construct this unioning object with the default LgK and the default update seed.
- */
- public CpcUnion() {
- this(CpcSketch.DEFAULT_LG_K, Util.DEFAULT_UPDATE_SEED);
- }
-
- /**
- * Construct this unioning object with LgK and the default update seed.
- * @param lgK The given log2 of K.
- */
- public CpcUnion(final int lgK) {
- this(lgK, Util.DEFAULT_UPDATE_SEED);
- }
-
- /**
- * Construct this unioning object with LgK and a given seed.
- * @param lgK The given log2 of K.
- * @param seed The given seed.
- */
- public CpcUnion(final int lgK, final long seed) {
- this.seed = seed;
- this.lgK = lgK;
- bitMatrix = null;
- // We begin with the accumulator holding an EMPTY_MERGED sketch object.
- // As an optimization the accumulator could start as NULL, but that would require changes elsewhere.
- accumulator = new CpcSketch(lgK, seed);
- }
-
- /**
- * Update this union with a CpcSketch.
- * @param sketch the given CpcSketch.
- */
- public void update(final CpcSketch sketch) {
- mergeInto(this, sketch);
- }
-
- /**
- * Returns the result of union operations as a CPC sketch.
- * @return the result of union operations as a CPC sketch.
- */
- public CpcSketch getResult() {
- return getResult(this);
- }
-
- /**
- * Returns the current value of Log_base2 of K. Note that due to merging with source sketches that
- * may have a lower value of LgK, this value can be less than what the union object was configured
- * with.
- *
- * @return the current value of Log_base2 of K.
- */
- public int getLgK() {
- return lgK;
- }
-
- /**
- * Return the DataSketches identifier for this CPC family of sketches.
- * @return the DataSketches identifier for this CPC family of sketches.
- */
- public static Family getFamily() {
- return Family.CPC;
- }
-
- //used for testing only
- long getNumCoupons() {
- if (bitMatrix != null) {
- return countBitsSetInMatrix(bitMatrix);
- }
- return accumulator.numCoupons;
- }
-
- //used for testing only
- static long[] getBitMatrix(final CpcUnion union) {
- checkUnionState(union);
- return (union.bitMatrix != null)
- ? union.bitMatrix
- : CpcUtil.bitMatrixOfSketch(union.accumulator);
- }
-
- private static void walkTableUpdatingSketch(final CpcSketch dest, final PairTable table) {
- final int[] slots = table.getSlotsArr();
- final int numSlots = (1 << table.getLgSizeInts());
- assert dest.lgK <= 26;
- final int destMask = (((1 << dest.lgK) - 1) << 6) | 63; //downsamples when destlgK < srcLgK
-
- /* Using the inverse golden ratio stride fixes the
- * Snow Plow Effect.
- */
- int stride = (int) (INVERSE_GOLDEN * numSlots);
- assert stride >= 2;
- if (stride == ((stride >>> 1) << 1)) { stride += 1; } //force the stride to be odd
- assert (stride >= 3) && (stride < numSlots);
-
- for (int i = 0, j = 0; i < numSlots; i++, j += stride) {
- j &= (numSlots - 1);
- final int rowCol = slots[j];
- if (rowCol != -1) {
- dest.rowColUpdate(rowCol & destMask);
- }
- }
- }
-
- private static void orTableIntoMatrix(final long[] bitMatrix, final int destLgK, final PairTable table) {
- final int[] slots = table.getSlotsArr();
- final int numSlots = 1 << table.getLgSizeInts();
- final int destMask = (1 << destLgK) - 1; // downsamples when destlgK < srcLgK
- for (int i = 0; i < numSlots; i++) {
- final int rowCol = slots[i];
- if (rowCol != -1) {
- final int col = rowCol & 63;
- final int row = rowCol >>> 6;
- bitMatrix[row & destMask] |= (1L << col); // Set the bit.
- }
- }
- }
-
- private static void orWindowIntoMatrix(final long[] destMatrix, final int destLgK,
- final byte[] srcWindow, final int srcOffset, final int srcLgK) {
- assert (destLgK <= srcLgK);
- final int destMask = (1 << destLgK) - 1; // downsamples when destlgK < srcLgK
- final int srcK = 1 << srcLgK;
- for (int srcRow = 0; srcRow < srcK; srcRow++) {
- destMatrix[srcRow & destMask] |= ((srcWindow[srcRow] & 0XFFL) << srcOffset);
- }
- }
-
- private static void orMatrixIntoMatrix(final long[] destMatrix, final int destLgK,
- final long[] srcMatrix, final int srcLgK) {
- assert (destLgK <= srcLgK);
- final int destMask = (1 << destLgK) - 1; // downsamples when destlgK < srcLgK
- final int srcK = 1 << srcLgK;
- for (int srcRow = 0; srcRow < srcK; srcRow++) {
- destMatrix[srcRow & destMask] |= srcMatrix[srcRow];
- }
- }
-
- private static void reduceUnionK(final CpcUnion union, final int newLgK) {
- assert (newLgK < union.lgK);
-
- if (union.bitMatrix != null) { // downsample the union's bit matrix
- final int newK = 1 << newLgK;
- final long[] newMatrix = new long[newK];
-
- orMatrixIntoMatrix(newMatrix, newLgK, union.bitMatrix, union.lgK);
- union.bitMatrix = newMatrix;
- union.lgK = newLgK;
- }
-
- else { // downsample the union's accumulator
- final CpcSketch oldSketch = union.accumulator;
-
- if (oldSketch.numCoupons == 0) {
- union.accumulator = new CpcSketch(newLgK, oldSketch.seed);
- union.lgK = newLgK;
- return;
- }
-
- final CpcSketch newSketch = new CpcSketch(newLgK, oldSketch.seed);
- walkTableUpdatingSketch(newSketch, oldSketch.pairTable);
-
- final Flavor finalNewFlavor = newSketch.getFlavor();
- assert (finalNewFlavor != EMPTY); //SV table had to have something in it
-
- if (finalNewFlavor == SPARSE) {
- union.accumulator = newSketch;
- union.lgK = newLgK;
- return;
- }
-
- // the new sketch has graduated beyond sparse, so convert to bitMatrix
- union.accumulator = null;
- union.bitMatrix = CpcUtil.bitMatrixOfSketch(newSketch);
- union.lgK = newLgK;
- }
- }
-
- private static void mergeInto(final CpcUnion union, final CpcSketch source) {
- if (source == null) { return; }
- checkSeeds(union.seed, source.seed);
-
- final int sourceFlavorOrd = source.getFlavor().ordinal();
- if (sourceFlavorOrd == 0) { return; } //EMPTY
-
- //Accumulator and bitMatrix must be mutually exclusive,
- //so bitMatrix != null => accumulator == null and visa versa
- //if (Accumulator != null) union must be EMPTY or SPARSE,
- checkUnionState(union);
-
- if (source.lgK < union.lgK) { reduceUnionK(union, source.lgK); }
-
- // if source is past SPARSE mode, make sure that union is a bitMatrix.
- if ((sourceFlavorOrd > 1) && (union.accumulator != null)) {
- union.bitMatrix = CpcUtil.bitMatrixOfSketch(union.accumulator);
- union.accumulator = null;
- }
-
- final int state = ((sourceFlavorOrd - 1) << 1) | ((union.bitMatrix != null) ? 1 : 0);
- switch (state) {
- case 0 : { //A: Sparse, bitMatrix == null, accumulator valid
- if (union.accumulator == null) {
- //CodeQL could not figure this out so I have to insert this.
- throw new SketchesStateException("union.accumulator can never be null here.");
- }
- if ((union.accumulator.getFlavor() == EMPTY)
- && (union.lgK == source.lgK)) {
- union.accumulator = source.copy();
- break;
- }
- walkTableUpdatingSketch(union.accumulator, source.pairTable);
- // if the accumulator has graduated beyond sparse, switch union to a bitMatrix
- if (union.accumulator.getFlavor().ordinal() > 1) {
- union.bitMatrix = CpcUtil.bitMatrixOfSketch(union.accumulator);
- union.accumulator = null;
- }
- break;
- }
- case 1 : { //B: Sparse, bitMatrix valid, accumulator == null
- orTableIntoMatrix(union.bitMatrix, union.lgK, source.pairTable);
- break;
- }
- case 3 : //C: Hybrid, bitMatrix valid, accumulator == null
- case 5 : { //C: Pinned, bitMatrix valid, accumulator == null
- orWindowIntoMatrix(union.bitMatrix, union.lgK, source.slidingWindow,
- source.windowOffset, source.lgK);
- orTableIntoMatrix(union.bitMatrix, union.lgK, source.pairTable);
- break;
- }
- case 7 : { //D: Sliding, bitMatrix valid, accumulator == null
- // SLIDING mode involves inverted logic, so we can't just walk the source sketch.
- // Instead, we convert it to a bitMatrix that can be OR'ed into the destination.
- final long[] sourceMatrix = CpcUtil.bitMatrixOfSketch(source);
- orMatrixIntoMatrix(union.bitMatrix, union.lgK, sourceMatrix, source.lgK);
- break;
- }
- default: throw new SketchesStateException("Illegal Union state: " + state);
- }
- }
-
- private static CpcSketch getResult(final CpcUnion union) {
- checkUnionState(union);
-
- if (union.accumulator != null) { // start of case where union contains a sketch
- if (union.accumulator.numCoupons == 0) {
- final CpcSketch result = new CpcSketch(union.lgK, union.accumulator.seed);
- result.mergeFlag = true;
- return (result);
- }
- assert (SPARSE == union.accumulator.getFlavor());
- final CpcSketch result = union.accumulator.copy();
- result.mergeFlag = true;
- return (result);
- } // end of case where union contains a sketch
-
- // start of case where union contains a bitMatrix
- final long[] matrix = union.bitMatrix;
- final int lgK = union.lgK;
- final CpcSketch result = new CpcSketch(union.lgK, union.seed);
-
- final long numCoupons = countBitsSetInMatrix(matrix);
- result.numCoupons = numCoupons;
-
- final Flavor flavor = CpcUtil.determineFlavor(lgK, numCoupons);
- assert ((flavor.ordinal() > SPARSE.ordinal()) );
-
- final int offset = CpcUtil.determineCorrectOffset(lgK, numCoupons);
- result.windowOffset = offset;
-
- //Build the window and pair table
-
- final int k = 1 << lgK;
- final byte[] window = new byte[k];
- result.slidingWindow = window;
-
- // LgSize = K/16; in some cases this will end up being oversized
- final int newTableLgSize = Math.max(lgK - 4, 2);
- final PairTable table = new PairTable(newTableLgSize, 6 + lgK);
- result.pairTable = table;
-
- // The following works even when the offset is zero.
- final long maskForClearingWindow = (0XFFL << offset) ^ -1L;
- final long maskForFlippingEarlyZone = (1L << offset) - 1L;
- long allSurprisesORed = 0;
-
- /* using a sufficiently large hash table avoids the
- * Snow Plow Effect
- */
- for (int i = 0; i < k; i++) {
- long pattern = matrix[i];
- window[i] = (byte) ((pattern >>> offset) & 0XFFL);
- pattern &= maskForClearingWindow;
- pattern ^= maskForFlippingEarlyZone; // This flipping converts surprising 0's to 1's.
- allSurprisesORed |= pattern;
- while (pattern != 0) {
- final int col = Long.numberOfTrailingZeros(pattern);
- pattern = pattern ^ (1L << col); // erase the 1.
- final int rowCol = (i << 6) | col;
- final boolean isNovel = PairTable.maybeInsert(table, rowCol);
- assert isNovel;
- }
- }
-
- // At this point we could shrink an oversize hash table, but the relative waste isn't very big.
- result.fiCol = Long.numberOfTrailingZeros(allSurprisesORed);
- if (result.fiCol > offset) {
- result.fiCol = offset;
- } // corner case
-
- // NB: the HIP-related fields will contain bogus values, but that is okay.
-
- result.mergeFlag = true;
- return result;
- // end of case where union contains a bitMatrix
- }
-
- private static void checkSeeds(final long seedA, final long seedB) {
- if (seedA != seedB) {
- throw new SketchesArgumentException("Hash Seeds do not match.");
- }
- }
-
- private static void checkUnionState(final CpcUnion union) {
- if (union == null) {
- throw new SketchesStateException("union cannot be null");
- }
- final CpcSketch accumulator = union.accumulator;
- if ( !((accumulator != null) ^ (union.bitMatrix != null)) ) {
- throw new SketchesStateException(
- "accumulator and bitMatrix cannot be both valid or both null: "
- + "accumValid = " + (accumulator != null)
- + ", bitMatrixValid = " + (union.bitMatrix != null));
- }
- if (accumulator != null) { //must be SPARSE or EMPTY
- if (accumulator.numCoupons > 0) { //SPARSE
- if ( !((accumulator.slidingWindow == null) && (accumulator.pairTable != null)) ) {
- throw new SketchesStateException(
- "Non-empty union accumulator must be SPARSE: " + accumulator.getFlavor());
- }
- } //else EMPTY
- if (union.lgK != accumulator.lgK) {
- throw new SketchesStateException("union LgK must equal accumulator LgK");
- }
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/CpcUtil.java b/src/main/java/org/apache/datasketches/cpc/CpcUtil.java
deleted file mode 100644
index da7cba086..000000000
--- a/src/main/java/org/apache/datasketches/cpc/CpcUtil.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import java.util.Arrays;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class CpcUtil {
- static final int minLgK = 4;
- static final int maxLgK = 26;
-
- static void checkLgK(final int lgK) {
- if ((lgK < minLgK) || (lgK > maxLgK)) {
- throw new SketchesArgumentException("LgK must be >= 4 and <= 26: " + lgK);
- }
- }
-
- static Flavor determineFlavor(final int lgK, final long numCoupons) {
- final long c = numCoupons;
- final long k = 1L << lgK;
- final long c2 = c << 1;
- final long c8 = c << 3;
- final long c32 = c << 5;
- if (c == 0) {
- return Flavor.EMPTY; // 0 == C < 1
- }
- if (c32 < (3 * k)) {
- return Flavor.SPARSE; // 1 <= C < 3K/32
- }
- if (c2 < k) {
- return Flavor.HYBRID; // 3K/32 <= C < K/2
- }
- if (c8 < (27 * k)) {
- return Flavor.PINNED; // K/2 <= C < 27K/8
- }
- else {
- return Flavor.SLIDING; // 27K/8 <= C
- }
- }
-
- /**
- * Warning: this is called in several places, including during the
- * transitional moments during which sketch invariants involving
- * flavor and offset are out of whack and in fact we are re-imposing
- * them. Therefore it cannot rely on determineFlavor() or
- * determineCorrectOffset(). Instead it interprets the low level data
- * structures "as is".
- *
- * This produces a full-size k-by-64 bit matrix from any Live sketch.
- *
- * @param sketch the given sketch
- * @return the bit matrix as an array of longs.
- */
- static long[] bitMatrixOfSketch(final CpcSketch sketch) {
- final int k = (1 << sketch.lgK);
- final int offset = sketch.windowOffset;
- assert (offset >= 0) && (offset <= 56);
-
- final long[] matrix = new long[k];
-
- if (sketch.numCoupons == 0) {
- return matrix; // Returning a matrix of zeros rather than NULL.
- }
-
- //Fill the matrix with default rows in which the "early zone" is filled with ones.
- //This is essential for the routine's O(k) time cost (as opposed to O(C)).
- final long defaultRow = (1L << offset) - 1L;
- Arrays.fill(matrix, defaultRow);
-
- final byte[] window = sketch.slidingWindow;
- if (window != null) { // In other words, we are in window mode, not sparse mode.
- for (int i = 0; i < k; i++) { // set the window bits, trusting the sketch's current offset.
- matrix[i] |= ((window[i] & 0XFFL) << offset);
- }
- }
- final PairTable table = sketch.pairTable;
- assert (table != null);
- final int[] slots = table.getSlotsArr();
- final int numSlots = 1 << table.getLgSizeInts();
-
- for (int i = 0; i < numSlots; i++) {
- final int rowCol = slots[i];
- if (rowCol != -1) {
- final int col = rowCol & 63;
- final int row = rowCol >>> 6;
- // Flip the specified matrix bit from its default value.
- // In the "early" zone the bit changes from 1 to 0.
- // In the "late" zone the bit changes from 0 to 1.
- matrix[row] ^= (1L << col);
- }
- }
- return matrix;
- }
-
- static long countBitsSetInMatrix(final long[] matrix) {
- long count = 0;
- final int len = matrix.length;
- for (int i = 0; i < len; i++) { count += Long.bitCount(matrix[i]); }
- return count;
- }
-
- static int determineCorrectOffset(final int lgK, final long numCoupons) {
- final long c = numCoupons;
- final long k = (1L << lgK);
- final long tmp = (c << 3) - (19L * k); // 8C - 19K
- if (tmp < 0) { return 0; }
- return (int) (tmp >>> (lgK + 3)); // tmp / 8K
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/CpcWrapper.java b/src/main/java/org/apache/datasketches/cpc/CpcWrapper.java
deleted file mode 100644
index 083eeadf7..000000000
--- a/src/main/java/org/apache/datasketches/cpc/CpcWrapper.java
+++ /dev/null
@@ -1,121 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static org.apache.datasketches.cpc.CpcConfidence.getHipConfidenceLB;
-import static org.apache.datasketches.cpc.CpcConfidence.getHipConfidenceUB;
-import static org.apache.datasketches.cpc.CpcConfidence.getIconConfidenceLB;
-import static org.apache.datasketches.cpc.CpcConfidence.getIconConfidenceUB;
-import static org.apache.datasketches.cpc.IconEstimator.getIconEstimate;
-import static org.apache.datasketches.cpc.PreambleUtil.checkLoPreamble;
-import static org.apache.datasketches.cpc.PreambleUtil.getHipAccum;
-import static org.apache.datasketches.cpc.PreambleUtil.getNumCoupons;
-import static org.apache.datasketches.cpc.PreambleUtil.hasHip;
-import static org.apache.datasketches.cpc.PreambleUtil.isCompressed;
-import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssert;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.SuppressFBWarnings;
-
-/**
- * This provides a read-only view of a serialized image of a CpcSketch, which can be
- * on-heap or off-heap represented as a MemorySegment object, or on-heap represented as a byte array.
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-public final class CpcWrapper {
- MemorySegment seg;
-
- /**
- * Construct a read-only view of the given MemorySegment that contains a CpcSketch
- * @param seg the given MemorySegment
- */
- @SuppressFBWarnings(value = "EI_EXPOSE_REP2", justification = "This is OK here")
- public CpcWrapper(final MemorySegment seg) {
- this.seg = seg;
- checkLoPreamble(seg);
- rtAssert(isCompressed(seg));
-
- }
-
- /**
- * Construct a read-only view of the given byte array that contains a CpcSketch.
- * @param byteArray the given byte array
- */
- public CpcWrapper(final byte[] byteArray) {
- this(MemorySegment.ofArray(byteArray));
- }
-
- /**
- * Returns the best estimate of the cardinality of the sketch.
- * @return the best estimate of the cardinality of the sketch.
- */
- public double getEstimate() {
- if (!hasHip(seg)) {
- return getIconEstimate(PreambleUtil.getLgK(seg), getNumCoupons(seg));
- }
- return getHipAccum(seg);
- }
-
- /**
- * Return the DataSketches identifier for this CPC family of sketches.
- * @return the DataSketches identifier for this CPC family of sketches.
- */
- public static Family getFamily() {
- return Family.CPC;
- }
-
- /**
- * Returns the configured Log_base2 of K of this sketch.
- * @return the configured Log_base2 of K of this sketch.
- */
- public int getLgK() {
- return PreambleUtil.getLgK(seg);
- }
-
- /**
- * Returns the best estimate of the lower bound of the confidence interval given kappa,
- * the number of standard deviations from the mean.
- * @param kappa the given number of standard deviations from the mean: 1, 2 or 3.
- * @return the best estimate of the lower bound of the confidence interval given kappa.
- */
- public double getLowerBound(final int kappa) {
- if (!hasHip(seg)) {
- return getIconConfidenceLB(PreambleUtil.getLgK(seg), getNumCoupons(seg), kappa);
- }
- return getHipConfidenceLB(PreambleUtil.getLgK(seg), getNumCoupons(seg), getHipAccum(seg), kappa);
- }
-
- /**
- * Returns the best estimate of the upper bound of the confidence interval given kappa,
- * the number of standard deviations from the mean.
- * @param kappa the given number of standard deviations from the mean: 1, 2 or 3.
- * @return the best estimate of the upper bound of the confidence interval given kappa.
- */
- public double getUpperBound(final int kappa) {
- if (!hasHip(seg)) {
- return getIconConfidenceUB(PreambleUtil.getLgK(seg), getNumCoupons(seg), kappa);
- }
- return getHipConfidenceUB(PreambleUtil.getLgK(seg), getNumCoupons(seg), getHipAccum(seg), kappa);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/Flavor.java b/src/main/java/org/apache/datasketches/cpc/Flavor.java
deleted file mode 100644
index 52cd66cd6..000000000
--- a/src/main/java/org/apache/datasketches/cpc/Flavor.java
+++ /dev/null
@@ -1,46 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-/**
- * Note: except for brief transitional moments, these sketches always obey the following strict
- * mapping between the flavor of a sketch and the number of coupons that it has collected.
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-enum Flavor {
- EMPTY, // 0 == C < 1
- SPARSE, // 1 <= C < 3K/32
- HYBRID, // 3K/32 <= C < K/2
- PINNED, // K/2 <= C < 27K/8 [NB: 27/8 = 3 + 3/8]
- SLIDING; // 27K/8 <= C
-
- private static Flavor[] fmtArr = Flavor.class.getEnumConstants();
-
- /**
- * Returns the Flavor given its enum ordinal
- * @param ordinal the given enum ordinal
- * @return the Flavor given its enum ordinal
- */
- static Flavor ordinalToFlavor(final int ordinal) {
- return fmtArr[ordinal];
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/Format.java b/src/main/java/org/apache/datasketches/cpc/Format.java
deleted file mode 100644
index 200045170..000000000
--- a/src/main/java/org/apache/datasketches/cpc/Format.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-/**
- * There are seven different preamble formats (8 combinations) that determine the layout of the
- * HiField variables after the first 8 bytes of the preamble.
- * Do not change the order.
- */
-enum Format {
- EMPTY_MERGED,
- EMPTY_HIP,
- SPARSE_HYBRID_MERGED,
- SPARSE_HYBRID_HIP,
- PINNED_SLIDING_MERGED_NOSV,
- PINNED_SLIDING_HIP_NOSV,
- PINNED_SLIDING_MERGED,
- PINNED_SLIDING_HIP;
-
- private static Format[] fmtArr = Format.class.getEnumConstants();
-
- /**
- * Returns the Format given its enum ordinal
- * @param ordinal the given enum ordinal
- * @return the Format given its enum ordinal
- */
- static Format ordinalToFormat(final int ordinal) {
- return fmtArr[ordinal];
- }
-
-} //end enum Format
diff --git a/src/main/java/org/apache/datasketches/cpc/IconEstimator.java b/src/main/java/org/apache/datasketches/cpc/IconEstimator.java
deleted file mode 100644
index cc5382fdc..000000000
--- a/src/main/java/org/apache/datasketches/cpc/IconEstimator.java
+++ /dev/null
@@ -1,93 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static org.apache.datasketches.cpc.CpcUtil.maxLgK;
-import static org.apache.datasketches.cpc.CpcUtil.minLgK;
-import static org.apache.datasketches.cpc.IconPolynomialCoefficients.iconPolynomialCoefficents;
-import static org.apache.datasketches.cpc.IconPolynomialCoefficients.iconPolynomialNumCoefficients;
-
-/**
- * The ICON estimator for CPC sketches is defined by the arXiv paper.
- *
- * The current file provides exact and approximate implementations of this estimator.
- *
- * The exact version works for any value of K, but is quite slow.
- *
- * The much faster approximate version works for K values that are powers of two
- * ranging from 2^4 to 2^32.
- *
- * At a high-level, this approximation can be described as using an
- * exponential approximation when C > K * (5.6 or 5.7), while smaller
- * values of C are handled by a degree-19 polynomial approximation of
- * a pre-conditioned version of the true ICON mapping from C to N_hat.
- *
- * This file also provides a validation procedure that compares its approximate
- * and exact implementations of the CPC ICON estimator.
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class IconEstimator {
-
- static double evaluatePolynomial(final double[] coefficients, final int start, final int num,
- final double x) {
- final int end = (start + num) - 1;
- double total = coefficients[end];
- for (int j = end - 1; j >= start; j--) {
- total *= x;
- total += coefficients[j];
- }
- return total;
- }
-
- static double iconExponentialApproximation(final double k, final double c) {
- return (0.7940236163830469 * k * Math.pow(2.0, c / k));
- }
-
- static double getIconEstimate(final int lgK, final long c) {
- assert lgK >= minLgK;
- assert lgK <= maxLgK;
- if (c < 2L) { return ((c == 0L) ? 0.0 : 1.0); }
- final int k = 1 << lgK;
- final double doubleK = k;
- final double doubleC = c;
- // Differing thresholds ensure that the approximated estimator is monotonically increasing.
- final double thresholdFactor = ((lgK < 14) ? 5.7 : 5.6);
- if (doubleC > (thresholdFactor * doubleK)) {
- return (iconExponentialApproximation(doubleK, doubleC));
- }
- final double factor = evaluatePolynomial(iconPolynomialCoefficents,
- iconPolynomialNumCoefficients * (lgK - minLgK),
- iconPolynomialNumCoefficients,
- // The constant 2.0 is baked into the table iconPolynomialCoefficents[].
- // This factor, although somewhat arbitrary, is based on extensive characterization studies
- // and is considered a safe conservative factor.
- doubleC / (2.0 * doubleK));
- final double ratio = doubleC / doubleK;
- // The constant 66.774757 is baked into the table iconPolynomialCoefficents[].
- // This factor, although somewhat arbitrary, is based on extensive characterization studies
- // and is considered a safe conservative factor.
- final double term = 1.0 + ((ratio * ratio * ratio) / 66.774757);
- final double result = doubleC * factor * term;
- return (result >= doubleC) ? result : doubleC;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/IconPolynomialCoefficients.java b/src/main/java/org/apache/datasketches/cpc/IconPolynomialCoefficients.java
deleted file mode 100644
index 6c08528f9..000000000
--- a/src/main/java/org/apache/datasketches/cpc/IconPolynomialCoefficients.java
+++ /dev/null
@@ -1,175 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static org.apache.datasketches.cpc.CpcUtil.maxLgK;
-import static org.apache.datasketches.cpc.CpcUtil.minLgK;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class IconPolynomialCoefficients {
- static final int iconPolynomialDegree = 19;
- static final int iconPolynomialNumCoefficients = 1 + iconPolynomialDegree;
- static final int iconTableSize = iconPolynomialNumCoefficients * ((1 + maxLgK) - minLgK);
-
- //CHECKSTYLE.OFF: LineLength
- static final double[] iconPolynomialCoefficents = new double[] {
- // log K = 4
- 0.9895027971889700513, 0.3319496644645180128, 0.1242818722715769986, -0.03324149686026930256, -0.2985637298081619817,
- 1.366555923595830002, -4.705499366260569971, 11.61506432505530029, -21.11254986175579873, 28.89421695078809904,
- -30.1383659011730991, 24.11946778830730054, -14.83391445199539938, 6.983088767267210173, -2.48964120264876998,
- 0.6593243603602499947, -0.125493534558034997, 0.01620971672896159843, -0.001271267679036929953, 4.567178653294529745e-05,
-
- // log K = 5
- 0.9947713741300230339, 0.3326559581620939787, 0.1250050661634889981, -0.04130073804472530336, -0.2584095537451129854,
- 1.218050389433120051, -4.319106696095399656, 10.87175052045090062, -20.0184979022142997, 27.63210188163320069,
- -28.97950009664030091, 23.26740804691930009, -14.33375703270860058, 6.751281271241110105, -2.406363094133439962,
- 0.6367414734718820357, -0.1210468076141379967, 0.01561196698118279963, -0.001222335432128580056, 4.383502970318410206e-05,
-
- // log K = 6
- 0.9973904854982870161, 0.3330148852217920119, 0.125251536589509993, -0.04434075124043219962, -0.2436238890691720116,
- 1.163293254754570016, -4.177758779777369647, 10.60301981340099964, -19.6274507428828997, 27.18420839597660077,
- -28.56827214174580121, 22.96268674086600114, -14.15234202220280046, 6.665700662642549901, -2.375043356720739851,
- 0.6280993991240929608, -0.119319019358031006, 0.01537674055733759954, -0.001202881695730769916, 4.309894633186929849e-05,
-
- // log K = 7
- 0.9986963310058679655, 0.3331956705633329907, 0.125337696770523005, -0.04546817338088020299, -0.2386752211125199863,
- 1.145927328111949972, -4.135694445582720036, 10.52805060502839929, -19.52408322548339825, 27.06921653903929936,
- -28.46207532143190022, 22.88083524357429965, -14.10057147392659971, 6.63958754983273991, -2.364865219283200037,
- 0.6251341806425250169, -0.1186991327450530043, 0.0152892726403408008, -0.001195439764873199896, 4.281098416794090072e-05,
-
- // log K = 8
- 0.999348600452531044, 0.3332480372393080148, 0.126666900963325002, -0.06495714694254159371, -0.08376282050638980681,
- 0.3760158094643630267, -1.568204791601850001, 4.483117719555970382, -9.119180124379150598, 13.65799293358900002,
- -15.3100211234349004, 12.97546344654869976, -8.351661538536939489, 4.075022612435580172, -1.49387015887069996,
- 0.4040976870253379927, -0.07813232681879349328, 0.01020545649538820085, -0.0008063279210812720381, 2.909334976414100078e-05,
-
- // log K = 9
- 0.9996743787297059924, 0.3332925779481850093, 0.1267124599259649986, -0.06550452970936600228, -0.08191738117533520214,
- 0.3773034458363569987, -1.604679509609959975, 4.636761898691969641, -9.487348609558699408, 14.25164235443030059,
- -15.99674955529870068, 13.56353219046370029, -8.730194904342459594, 4.259010067932120336, -1.56106689792022002,
- 0.4222540912786589828, -0.08165296504921559784, 0.01066878484925220041, -0.0008433887618256910015, 3.045339724886519912e-05,
-
- // log K = 10
- 0.999837191783945034, 0.3333142252339619804, 0.1267759538087240012, -0.06631005632753710077, -0.07692759158286699428,
- 0.3568943956395980166, -1.546598721379510044, 4.51595019978557044, -9.298431968763770428, 14.02586858080080034,
- -15.78858959520439953, 13.41484931677589998, -8.647958125130809748, 4.22398017468472009, -1.549708891200570093,
- 0.419507410264540026, -0.08117411611046250475, 0.01061202286184199928, -0.000839300527596772007, 3.03185874520205985e-05,
-
- // log K = 11
- 0.9999186020796150265, 0.3333249054574359826, 0.126791713589799987, -0.06662487271699729652, -0.07335552427910230211,
- 0.3316370184815959909, -1.434143797561290068, 4.180260309967409604, -8.593906870708760692, 12.95088874800289958,
- -14.56876092520539956, 12.37074367531410068, -7.969152075707960137, 3.888774396648960074, -1.424923326506990051,
- 0.385084561785229984, -0.07435541911616409816, 0.009695363567476529554, -0.0007644375960047160388, 2.75156194717188011e-05,
-
- // log K = 12
- 0.9999592955649559967, 0.3333310560725140093, 0.1267379744020450116, -0.06524495415766619344, -0.08854031542298740343,
- 0.4244320628874230228, -1.794077789033230008, 5.133875262768450298, -10.40149374917120007, 15.47808115629240078,
- -17.2272296137545986, 14.5002173676463002, -9.274819801602760094, 4.500782540026570189, -1.642359389030050076,
- 0.442596113445525019, -0.0853226219238850947, 0.01111969379054169975, -0.0008771614088006969611, 3.161668519459719752e-05,
-
- // log K = 13
- 0.9999796468102559732, 0.3333336602394039727, 0.126728089053198989, -0.06503798598282370391, -0.09050261023823169548,
- 0.4350609244189960201, -1.831274835815670077, 5.223387516985289913, -10.55574395269979959, 15.67359470222429962,
- -17.41263416341029924, 14.63297400889229927, -9.346752431221359458, 4.530124905188380069, -1.651245566462089975,
- 0.444542549250713015, -0.08561720963336499901, 0.01114805146185449992, -0.0008786251203363140043, 3.16416341644572998e-05,
-
- // log K = 14
- 0.9999898187060970445, 0.3333362579300819806, 0.1266984078369459976, -0.06464561179765909715, -0.09343280886228019777,
- 0.4490702549264070087, -1.878087608052450008, 5.338004322057390283, -10.76690603590630069, 15.97069195083200022,
- -17.73440379943459888, 14.90212518309260048, -9.520506013770420495, 4.616238931978830173, -1.68364817877918993,
- 0.4536194960681350086, -0.087448605434800597, 0.01139929991331390009, -0.0008995891451622229631, 3.244407259782900338e-05,
-
- // log K = 15
- 0.9999949072549390028, 0.3333376334705290267, 0.126665364358402005, -0.06411790034705669439, -0.09776009134670660128,
- 0.4704691112248470253, -1.948021675295769972, 5.497760972696490001, -11.03165645315390009, 16.29703330781000048,
- -18.03851029448010124, 15.11836776139680083, -9.638205179917429533, 4.665122328753120051, -1.698980686525759953,
- 0.4571799506245269873, -0.08804011353783609828, 0.01146553155965330043, -0.0009040455800659569869, 3.257931866957050274e-05,
-
- // log K = 16
- 0.9999974544793589493, 0.3333381337614599871, 0.1266524862971120102, -0.06391676499117690535, -0.09929616211306059592,
- 0.4771390820378790254, -1.965762451227349938, 5.526802350376460282, -11.05703067024660058, 16.29535848023060041,
- -18.00114005075790047, 15.06214012231560062, -9.58874727382628933, 4.63537541652793017, -1.686222848555620102,
- 0.4532602373715179933, -0.08719448925964939923, 0.01134365425717459921, -0.0008934965241274289835, 3.216436244471380105e-05,
-
- // log K = 17
- 0.9999987278278800185, 0.3333383411464330148, 0.126642761751724009, -0.06371042959073920653, -0.1013564516034080043,
- 0.4891311195679299839, -2.010971712051409899, 5.644390807952309963, -11.27697253921500042, 16.59957157207080058,
- -18.31808338317799922, 15.31363518393730061, -9.741451446816620674, 4.706207545519429658, -1.711102469010010063,
- 0.4597587341089349744, -0.08841670767182820134, 0.01149999225097850068, -0.0009056651366963050422, 3.259910736274500059e-05,
-
- // log K = 18
- 0.9999993637727100371, 0.3333385511608860097, 0.1266341580529160016, -0.06353272828164230335, -0.103139962850642003,
- 0.4996216017206500104, -2.05099128585287982, 5.749874086531799655, -11.47727638570349917, 16.88141587810320132,
- -18.61744656177490143, 15.55634230427719977, -9.892350736128680211, 4.778033520984200422, -1.737045483861280104,
- 0.4667410882683730167, -0.08977256212421590165, 0.01167940146667079994, -0.0009201381242396030127, 3.313600701586759867e-05,
-
- // log K = 19
- 0.9999996805376010212, 0.3333372324328989778, 0.1267104737214659882, -0.06504749929326139601, -0.0882341962464350954,
- 0.4131871162041140244, -1.725190703567099915, 4.900817515593920426, -9.883452720776510603, 14.6657081190816001,
- -16.29398295135089825, 13.69805011761319946, -8.753475239465899449, 4.244072374564439976, -1.547202527706629915,
- 0.4164770109614310267, -0.08017596922092029565, 0.01043146101701039954, -0.00082124200571200305, 2.953319493719429935e-05,
-
- // log K = 20
- 0.9999998390037539986, 0.3333365859956040067, 0.1267460211029839967, -0.06569456024647769843, -0.0823070353477164951,
- 0.3810826463303410017, -1.611983580241109992, 4.624520077758210057, -9.397308335633589138, 14.03184981378050011,
- -15.6703191315401007, 13.22992718704790072, -8.484216393184780713, 4.125607133488029987, -1.507690650697159906,
- 0.4066678517577320129, -0.07842110121777939868, 0.01021780862225150042, -0.0008054065857047439754, 2.899431830426989844e-05,
-
- // log K = 21
- 0.9999999207001479817, 0.3333384953015239849, 0.1266331480396669928, -0.06345750166298599892, -0.1042341210992499961,
- 0.5077112908497130039, -2.087398133609810191, 5.858842546192500222, -11.70620319777190055, 17.23103975433669888,
- -19.01462552846669851, 15.89674059836560005, -10.11395134034419918, 4.88760796465891989, -1.777886770904629987,
- 0.4780200178339499839, -0.09200895321782050218, 0.01198029553244219989, -0.0009447283875782100165, 3.405716775824710232e-05,
-
- // log K = 22
- 0.9999999606908690497, 0.3333383929524300071, 0.1266456445096819927, -0.06373504294081690225, -0.1012834291081849969,
- 0.4893810690172959998, -2.01391428223606983, 5.656430437473649597, -11.3067201537791, 16.64980594135310099,
- -18.3792355790383013, 15.36879753115040081, -9.778831246425049528, 4.725308061988969577, -1.718423596500280093,
- 0.4618308177809870019, -0.08883675060799739454, 0.01155766944804260087, -0.0009104695617243750358, 3.278237729674439666e-05,
-
- // log K = 23
- 0.9999999794683379628, 0.3333386441751680085, 0.1266463995182049995, -0.06376031920455070556, -0.1010799540803130059,
- 0.488540137426137, -2.012048323537570127, 5.654949475342659682, -11.31023240892979942, 16.66334675284959843,
- -18.40241452866079896, 15.39443572867130072, -9.798844412838670692, 4.736683907539640082, -1.723168363744929987,
- 0.463270349018644001, -0.08914619066708899531, 0.01160235936257320022, -0.0009143600818183229709, 3.293669304679140117e-05,
-
- // log K = 24
- 0.9999999911469820146, 0.3333376076934529975, 0.1266944349940530012, -0.06470524278387919381, -0.09189342220283110152,
- 0.4359182372694809793, -1.815980282951169977, 5.149474056470340066, -10.37086570678100017, 15.36962686758569951,
- -17.05756384717849983, 14.32755177515199918, -9.149944050025640152, 4.434601894497260055, -1.616478926806520056,
- 0.4351979157055039793, -0.08381768225272340223, 0.01091321820476520016, -0.0008600264403629039739, 3.09667800347144002e-05,
-
- // log K = 25
- 0.9999999968592140354, 0.3333379164881000167, 0.1266782495827009913, -0.06434163088961859789, -0.09575258124988890451,
- 0.4597843575354370049, -1.911374431241559924, 5.411856661251520428, -10.88850084646090011, 16.12298941380269923,
- -17.88172178487259956, 15.01301780636859995, -9.585542896142529301, 4.645811872761620442, -1.693952293156189892,
- 0.4563143308861309921, -0.08795976148455289523, 0.01146560428011200033, -0.0009048442931930629528, 3.26358391497329992e-05,
-
- // log K = 26
- 0.9999999970700530483, 0.333338329556315982, 0.126644753076394001, -0.06372365346512399997, -0.1012760856945769949,
- 0.4886852278576360176, -2.009005418394389952, 5.638119224137019714, -11.26276715335160006, 16.57640024218650154,
- -18.29035093605569884, 15.28892246224570073, -9.724916375991760731, 4.6978877652334603, -1.707974125916829955,
- 0.4588937864564729963, -0.08824617586088029375, 0.01147732114826570046, -0.00090384524860747295, 3.253252703695579795e-05,
- };
- //CHECKSTYLE.ON: LineLength
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/MergingValidation.java b/src/main/java/org/apache/datasketches/cpc/MergingValidation.java
deleted file mode 100644
index 213e65d30..000000000
--- a/src/main/java/org/apache/datasketches/cpc/MergingValidation.java
+++ /dev/null
@@ -1,219 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static org.apache.datasketches.common.Util.INVERSE_GOLDEN_U64;
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.common.Util.powerSeriesNextDouble;
-import static org.apache.datasketches.cpc.IconEstimator.getIconEstimate;
-import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssert;
-import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssertEquals;
-
-import java.io.PrintStream;
-import java.io.PrintWriter;
-
-import org.apache.datasketches.common.SuppressFBWarnings;
-
-/**
- * This code is used both by unit tests, for short running tests,
- * and by the characterization repository for longer running, more exhaustive testing. To be
- * accessible for both, this code is part of the main hierarchy. It is not used during normal
- * production runtime.
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-public class MergingValidation {
- private String hfmt;
- private String dfmt;
- private String[] hStrArr;
- private long vIn = 0;
-
- //inputs
- private int lgMinK;
- private int lgMaxK; //inclusive
- private int lgMulK; //multiplier of K to produce maxNa, maxNb
- private int uPPO;
- private int incLgK; //increment of lgK
- private PrintStream printStream;
- private PrintWriter printWriter;
-
- /**
- * Constructor
- * @param lgMinK lgMinK
- * @param lgMaxK lgMaxK
- * @param lgMulK lgMulK
- * @param uPPO uPPO
- * @param incLgK incLgK
- * @param pS pS
- * @param pW pW
- */
- @SuppressFBWarnings(value = "EI_EXPOSE_REP2", justification = "This is OK here")
- public MergingValidation(final int lgMinK, final int lgMaxK, final int lgMulK, final int uPPO,
- final int incLgK, final PrintStream pS, final PrintWriter pW) {
- this.lgMinK = lgMinK;
- this.lgMaxK = lgMaxK;
- this.lgMulK = lgMulK;
- this.uPPO = Math.max(uPPO, 1);
- this.incLgK = Math.max(incLgK, 1);
- printStream = pS;
- printWriter = pW;
- assembleFormats();
- }
-
- /**
- * Start the merging validation process
- */
- public void start() {
- printf(hfmt, (Object[]) hStrArr); //print header
- doRangeOfLgK();
- }
-
- private void doRangeOfLgK() {
- for (int lgK = lgMinK; lgK <= lgMaxK; lgK += incLgK) {
- multiTestMerging(lgK, lgK - 1, lgK - 1);
- multiTestMerging(lgK, lgK - 1, lgK + 0);
- multiTestMerging(lgK, lgK - 1, lgK + 1);
-
- multiTestMerging(lgK, lgK + 0, lgK - 1);
- multiTestMerging(lgK, lgK + 0, lgK + 0);
- multiTestMerging(lgK, lgK + 0, lgK + 1);
-
- multiTestMerging(lgK, lgK + 1, lgK - 1);
- multiTestMerging(lgK, lgK + 1, lgK + 0);
- multiTestMerging(lgK, lgK + 1, lgK + 1);
- }
- }
-
- private void multiTestMerging(final int lgKm, final int lgKa, final int lgKb) {
- final long limA = 1L << (lgKa + lgMulK);
- final long limB = 1L << (lgKa + lgMulK);
- long nA = 0;
- while (nA <= limA) {
- long nB = 0;
- while (nB <= limB) {
- testMerging(lgKm, lgKa, lgKb, nA, nB);
- nB = Math.round(powerSeriesNextDouble(uPPO, nB, true, 2.0));
- }
- nA = Math.round(powerSeriesNextDouble(uPPO, nA, true, 2.0));
- }
- }
-
- private void testMerging(final int lgKm, final int lgKa, final int lgKb, final long nA,
- final long nB) {
- final CpcUnion ugM = new CpcUnion(lgKm);
-
- // int lgKd = ((nA != 0) && (lgKa < lgKm)) ? lgKa : lgKm;
- // lgKd = ((nB != 0) && (lgKb < lgKd)) ? lgKb : lgKd;
- int lgKd = lgKm;
- if ((lgKa < lgKd) && (nA != 0)) { lgKd = lgKa; } //d = min(a,m) : m
- if ((lgKb < lgKd) && (nB != 0)) { lgKd = lgKb; } //d = min(b,d) : d
-
- final CpcSketch skD = new CpcSketch(lgKd); // direct sketch, updated with both streams
-
- final CpcSketch skA = new CpcSketch(lgKa);
- final CpcSketch skB = new CpcSketch(lgKb);
-
- for (long i = 0; i < nA; i++) {
- final long in = (vIn += INVERSE_GOLDEN_U64);
- skA.update(in);
- skD.update(in);
- }
- for (long i = 0; i < nB; i++) {
- final long in = (vIn += INVERSE_GOLDEN_U64);
- skB.update(in);
- skD.update(in);
- }
-
- ugM.update(skA);
- ugM.update(skB);
-
- final int finalLgKm = ugM.getLgK();
- final long[] matrixM = CpcUnion.getBitMatrix(ugM);
-
- final long cM = ugM.getNumCoupons();//countBitsSetInMatrix(matrixM);
- final long cD = skD.numCoupons;
- final Flavor flavorD = skD.getFlavor();
- final Flavor flavorA = skA.getFlavor();
- final Flavor flavorB = skB.getFlavor();
- final String dOff = Integer.toString(skD.windowOffset);
- final String aOff = Integer.toString(skA.windowOffset);
- final String bOff = Integer.toString(skB.windowOffset);
- final String flavorDoff = flavorD + String.format("%2s",dOff);
- final String flavorAoff = flavorA + String.format("%2s",aOff);
- final String flavorBoff = flavorB + String.format("%2s",bOff);
- final double iconEstD = getIconEstimate(lgKd, cD);
-
- rtAssert(finalLgKm <= lgKm);
- rtAssert(cM <= (skA.numCoupons + skB.numCoupons));
- rtAssertEquals(cM, cD);
-
- rtAssertEquals(finalLgKm, lgKd);
- final long[] matrixD = CpcUtil.bitMatrixOfSketch(skD);
- rtAssertEquals(matrixM, matrixD);
-
- final CpcSketch skR = ugM.getResult();
- final double iconEstR = getIconEstimate(skR.lgK, skR.numCoupons);
- rtAssertEquals(iconEstD, iconEstR, 0.0);
- rtAssert(TestUtil.specialEquals(skD, skR, false, true));
-
- printf(dfmt, lgKm, lgKa, lgKb, lgKd, nA, nB, (nA + nB),
- flavorAoff, flavorBoff, flavorDoff,
- skA.numCoupons, skB.numCoupons, cD, iconEstR);
- }
-
- private void printf(final String format, final Object ... args) {
- if (printStream != null) { printStream.printf(format, args); }
- if (printWriter != null) { printWriter.printf(format, args); }
- }
-
- private void assembleFormats() {
- final String[][] assy = {
- {"lgKm", "%4s", "%4d"},
- {"lgKa", "%4s", "%4d"},
- {"lgKb", "%4s", "%4d"},
- {"lgKfd", "%6s", "%6d"},
- {"nA", "%12s", "%12d"},
- {"nB", "%12s", "%12d"},
- {"nA+nB", "%12s", "%12d"},
- {"Flavor_a", "%11s", "%11s"},
- {"Flavor_b", "%11s", "%11s"},
- {"Flavor_fd", "%11s", "%11s"},
- {"Coupons_a", "%9s", "%9d"},
- {"Coupons_b", "%9s", "%9d"},
- {"Coupons_fd", "%9s", "%9d"},
- {"IconEst_dr", "%12s", "%,12.0f"}
- };
- final int cols = assy.length;
- hStrArr = new String[cols];
- final StringBuilder headerFmt = new StringBuilder();
- final StringBuilder dataFmt = new StringBuilder();
- headerFmt.append(LS + "Merging Validation" + LS);
- for (int i = 0; i < cols; i++) {
- hStrArr[i] = assy[i][0];
- headerFmt.append(assy[i][1]);
- headerFmt.append((i < (cols - 1)) ? "\t" : LS);
- dataFmt.append(assy[i][2]);
- dataFmt.append((i < (cols - 1)) ? "\t" : LS);
- }
- hfmt = headerFmt.toString();
- dfmt = dataFmt.toString();
- }
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/PairTable.java b/src/main/java/org/apache/datasketches/cpc/PairTable.java
deleted file mode 100644
index 87258ec5c..000000000
--- a/src/main/java/org/apache/datasketches/cpc/PairTable.java
+++ /dev/null
@@ -1,393 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssert;
-import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssertEquals;
-
-import java.util.Arrays;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * Note: Definition of
- * Snow Plow Effect.
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class PairTable {
- private static final int upsizeNumer = 3;
- private static final int upsizeDenom = 4;
- private static final int downsizeNumer = 1;
- private static final int downsizeDenom = 4;
-
- private int lgSizeInts;
- private final int validBits;
- private int numPairs;
- private int[] slotsArr;
-
- PairTable(final int lgSizeInts, final int numValidBits) {
- checkLgSizeInts(lgSizeInts);
- this.lgSizeInts = lgSizeInts;
- final int numSlots = 1 << lgSizeInts;
- validBits = numValidBits;
- numPairs = 0;
- slotsArr = new int[numSlots];
- for (int i = 0; i < numSlots; i++) { slotsArr[i] = -1; }
- }
-
- //Factory
- static PairTable newInstanceFromPairsArray(final int[] pairs, final int numPairs, final int lgK) {
- int lgNumSlots = 2;
- while ((upsizeDenom * numPairs) > (upsizeNumer * (1 << lgNumSlots))) {
- lgNumSlots++;
- }
- final PairTable table = new PairTable(lgNumSlots, 6 + lgK);
-
- /* Note: there is a possible
- * Snow Plow Effect here because
- * the caller is passing in a sorted pairs array. However, we are starting out with the correct
- * final table size, so the problem is not likely to occur.
- */
-
- for (int i = 0; i < numPairs; i++) {
- mustInsert(table, pairs[i]);
- }
- table.numPairs = numPairs;
- return table;
- }
-
- PairTable clear() {
- Arrays.fill(slotsArr, -1);
- numPairs = 0;
- return this;
- }
-
- PairTable copy() {
- final PairTable copy = new PairTable(lgSizeInts, validBits);
- copy.numPairs = numPairs;
- copy.slotsArr = slotsArr.clone(); //slotsArr can never be null
- return copy;
- }
-
- int getLgSizeInts() {
- return lgSizeInts;
- }
-
- int getNumPairs() {
- return numPairs;
- }
-
- int[] getSlotsArr() {
- return slotsArr;
- }
-
- int getValidBits() {
- return validBits;
- }
-
- /**
- * Rebuilds to a larger size. NumItems and validBits remain unchanged.
- * @param newLgSizeInts the new size
- * @return a larger PairTable
- */
- PairTable rebuild(final int newLgSizeInts) {
- checkLgSizeInts(newLgSizeInts);
- final int newSize = 1 << newLgSizeInts;
- final int oldSize = 1 << lgSizeInts;
- rtAssert(newSize > numPairs);
- final int[] oldSlotsArr = slotsArr;
- slotsArr = new int[newSize];
- Arrays.fill(slotsArr, -1);
- lgSizeInts = newLgSizeInts;
- for (int i = 0; i < oldSize; i++) {
- final int item = oldSlotsArr[i];
- if (item != -1) { mustInsert(this, item); }
- }
- return this;
- }
-
- @Override
- public String toString() {
- return toString(false);
- }
-
- private static void mustInsert(final PairTable table, final int item) {
- //SHARED CODE (implemented as a macro in C and expanded here)
- final int lgSizeInts = table.lgSizeInts;
- final int sizeInts = 1 << lgSizeInts;
- final int mask = sizeInts - 1;
- final int shift = table.validBits - lgSizeInts;
- rtAssert(shift > 0);
- int probe = item >>> shift; //extract high tablesize bits
- rtAssert((probe >= 0) && (probe <= mask));
- final int[] arr = table.slotsArr;
- int fetched = arr[probe];
- while ((fetched != item) && (fetched != -1)) {
- probe = (probe + 1) & mask;
- fetched = arr[probe];
- }
- //END SHARED CODE
- if (fetched == item) { throw new SketchesStateException("PairTable mustInsert() failed"); }
- else {
- assert (fetched == -1);
- arr[probe] = item;
- // counts and resizing must be handled by the caller.
- }
- }
-
- static boolean maybeInsert(final PairTable table, final int item) {
- //SHARED CODE (implemented as a macro in C and expanded here)
- final int lgSizeInts = table.lgSizeInts;
- final int sizeInts = 1 << lgSizeInts;
- final int mask = sizeInts - 1;
- final int shift = table.validBits - lgSizeInts;
- rtAssert(shift > 0);
- int probe = item >>> shift;
- rtAssert((probe >= 0) && (probe <= mask));
- final int[] arr = table.slotsArr;
- int fetched = arr[probe];
- while ((fetched != item) && (fetched != -1)) {
- probe = (probe + 1) & mask;
- fetched = arr[probe];
- }
- //END SHARED CODE
- if (fetched == item) { return false; }
- else {
- assert (fetched == -1);
- arr[probe] = item;
- table.numPairs += 1;
- while ((upsizeDenom * table.numPairs) > (upsizeNumer * (1 << table.lgSizeInts))) {
- table.rebuild(table.lgSizeInts + 1);
- }
- return true;
- }
- }
-
- static boolean maybeDelete(final PairTable table, final int item) {
- //SHARED CODE (implemented as a macro in C and expanded here)
- final int lgSizeInts = table.lgSizeInts;
- final int sizeInts = 1 << lgSizeInts;
- final int mask = sizeInts - 1;
- final int shift = table.validBits - lgSizeInts;
- rtAssert(shift > 0);
- int probe = item >>> shift;
- rtAssert((probe >= 0) && (probe <= mask));
- final int[] arr = table.slotsArr;
- int fetched = arr[probe];
- while ((fetched != item) && (fetched != -1)) {
- probe = (probe + 1) & mask;
- fetched = arr[probe];
- }
- //END SHARED CODE
- if (fetched == -1) { return false; }
- else {
- assert (fetched == item);
- // delete the item
- arr[probe] = -1;
- table.numPairs -= 1; assert (table.numPairs >= 0);
-
- // re-insert all items between the freed slot and the next empty slot
- probe = (probe + 1) & mask; fetched = arr[probe];
- while (fetched != -1) {
- arr[probe] = -1;
- mustInsert(table, fetched);
- probe = (probe + 1) & mask; fetched = arr[probe];
- }
-
- // shrink if necessary
- while (((downsizeDenom * table.numPairs)
- < (downsizeNumer * (1 << table.lgSizeInts))) && (table.lgSizeInts > 2)) {
- table.rebuild(table.lgSizeInts - 1);
- }
- return true;
- }
- }
-
- /**
- * While extracting the items from a linear probing hashtable,
- * this will usually undo the wrap-around provided that the table
- * isn't too full. Experiments suggest that for sufficiently large tables
- * the load factor would have to be over 90 percent before this would fail frequently,
- * and even then the subsequent sort would fix things up.
- * @param table the given table to unwrap
- * @param numPairs the number of valid pairs in the table
- * @return the unwrapped table
- */
- static int[] unwrappingGetItems(final PairTable table, final int numPairs) {
- if (numPairs < 1) { return null; }
- final int[] slotsArr = table.slotsArr;
- final int tableSize = 1 << table.lgSizeInts;
- final int[] result = new int[numPairs];
- int i = 0;
- int l = 0;
- int r = numPairs - 1;
-
- // Special rules for the region before the first empty slot.
- final int hiBit = 1 << (table.validBits - 1);
- while ((i < tableSize) && (slotsArr[i] != -1)) {
- final int item = slotsArr[i++];
- if ((item & hiBit) != 0) { result[r--] = item; } // This item was probably wrapped, so move to end.
- else { result[l++] = item; }
- }
-
- // The rest of the table is processed normally.
- while (i < tableSize) {
- final int look = slotsArr[i++];
- if (look != -1) { result[l++] = look; }
- }
- assert l == (r + 1);
- return result;
- }
-
- /**
- * In applications where the input array is already nearly sorted,
- * insertion sort runs in linear time with a very small constant.
- * This introspective version of insertion sort protects against
- * the quadratic cost of sorting bad input arrays.
- * It keeps track of how much work has been done, and if that exceeds a
- * constant times the array length, it switches to a different sorting algorithm.
- * @param a the array to sort
- * @param l points AT the leftmost element, i.e., inclusive.
- * @param r points AT the rightmost element, i.e., inclusive.
- */
- static void introspectiveInsertionSort(final int[] a, final int l, final int r) {
- final int length = (r - l) + 1;
- long cost = 0;
- final long costLimit = 8L * length;
- for (int i = l + 1; i <= r; i++) {
- int j = i;
- final long v = a[i] & 0XFFFF_FFFFL; //v must be long
- while ((j >= (l + 1)) && (v < ((a[j - 1]) & 0XFFFF_FFFFL))) {
- a[j] = a[j - 1];
- j -= 1;
- }
- a[j] = (int) v;
- cost += (i - j); // distance moved is a measure of work
-
- if (cost > costLimit) {
- //we need an unsigned sort, but it is linear time and very rarely occurs
- final long[] b = new long[a.length];
- for (int m = 0; m < a.length; m++) { b[m] = a[m] & 0XFFFF_FFFFL; }
- Arrays.sort(b, l, r + 1);
- for (int m = 0; m < a.length; m++) { a[m] = (int) b[m]; }
- // The following sanity check can be used during development
- //int bad = 0;
- //for (int m = l; m < (r - 1); m++) {
- // final long b1 = a[m] & 0XFFFF_FFFFL;
- // final long b2 = a[m + 1] & 0XFFFF_FFFFL;
- // if (b1 > b2) { bad++; }
- //}
- //assert (bad == 0);
- return;
- }
- }
- // The following sanity check can be used during development
- // int bad = 0;
- // for (int m = l; m < (r - 1); m++) {
- // final long b1 = a[m] & 0XFFFF_FFFFL;
- // final long b2 = a[m + 1] & 0XFFFF_FFFFL;
- // if (b1 > b2) { bad++; }
- // }
- // assert (bad == 0);
- }
-
- static void merge(
- final int[] arrA, final int startA, final int lengthA, //input
- final int[] arrB, final int startB, final int lengthB, //input
- final int[] arrC, final int startC) { //output
-
- final int lengthC = lengthA + lengthB;
- final int limA = startA + lengthA;
- final int limB = startB + lengthB;
- final int limC = startC + lengthC;
- int a = startA;
- int b = startB;
- int c = startC;
- for ( ; c < limC ; c++) {
- if (b >= limB) { arrC[c] = arrA[a++]; }
- else if (a >= limA) { arrC[c] = arrB[b++]; }
- else {
- final long aa = arrA[a] & 0XFFFF_FFFFL;
- final long bb = arrB[b] & 0XFFFF_FFFFL;
- if (aa < bb) { arrC[c] = arrA[a++]; }
- else { arrC[c] = arrB[b++]; }
- }
- }
- assert (a == limA);
- assert (b == limB);
- }
-
- static boolean equals(final PairTable a, final PairTable b) {
- if ((a == null) && (b == null)) { return true; }
- if ((a == null) || (b == null)) {
- throw new SketchesArgumentException("PairTable " + ((a == null) ? "a" : "b") + " is null");
- }
- //Note: lgSize array sizes may not be equal, but contents still can be equal
- rtAssertEquals(a.validBits, b.validBits);
- final int numPairsA = a.numPairs;
- final int numPairsB = b.numPairs;
- rtAssertEquals(numPairsA, numPairsB);
- final int[] pairsA = PairTable.unwrappingGetItems(a, numPairsA);
- final int[] pairsB = PairTable.unwrappingGetItems(b, numPairsB);
- PairTable.introspectiveInsertionSort(pairsA, 0, numPairsA - 1);
- PairTable.introspectiveInsertionSort(pairsB, 0, numPairsB - 1);
- rtAssertEquals(pairsA, pairsB);
- return true;
- }
-
- String toString(final boolean detail) {
- final StringBuilder sb = new StringBuilder();
- final int sizeInts = 1 << lgSizeInts;
- sb.append("PairTable").append(LS);
- sb.append(" Lg Size Ints : ").append(lgSizeInts).append(LS);
- sb.append(" Size Ints : ").append(sizeInts).append(LS);
- sb.append(" Valid Bits : ").append(validBits).append(LS);
- sb.append(" Num Pairs : ").append(numPairs).append(LS);
- if (detail) {
- sb.append(" DATA (hex) : ").append(LS);
- final String hdr = String.format("%9s %9s %9s %4s", "Index","Word","Row","Col");
- sb.append(hdr).append(LS);
- for (int i = 0; i < sizeInts; i++) {
- final int word = slotsArr[i];
- if (word == -1) { //empty
- final String h = String.format("%9d %9s", i, "--");
- sb.append(h).append(LS);
- } else { //data
- final int row = word >>> 6;
- final int col = word & 0X3F;
- final String wordStr = Long.toHexString(word & 0XFFFF_FFFFL);
- final String data = String.format("%9d %9s %9d %4d", i, wordStr, row, col);
- sb.append(data).append(LS);
- }
- }
- }
- return sb.toString();
- }
-
- private static void checkLgSizeInts(final int lgSizeInts) {
- if ((lgSizeInts < 2) || (lgSizeInts > 26)) {
- throw new SketchesArgumentException("Illegal LgSizeInts: " + lgSizeInts);
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/cpc/PreambleUtil.java b/src/main/java/org/apache/datasketches/cpc/PreambleUtil.java
deleted file mode 100644
index f24c86dd1..000000000
--- a/src/main/java/org/apache/datasketches/cpc/PreambleUtil.java
+++ /dev/null
@@ -1,819 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.cpc;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_DOUBLE_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_SHORT_UNALIGNED;
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.common.Util.checkBounds;
-import static org.apache.datasketches.common.Util.clear;
-import static org.apache.datasketches.common.Util.zeroPad;
-import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssert;
-import static org.apache.datasketches.cpc.RuntimeAsserts.rtAssertEquals;
-
-import java.lang.foreign.MemorySegment;
-import java.util.Objects;
-
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-
-//@formatter:off
-/**
- * All formats are illustrated as Big-Endian, LSB on the right.
- *
- * Note: NUM_SV has dual meanings: In sparse and hybrid flavors it is equivalent to
- * numCoupons so it isn't stored separately. In pinned and sliding flavors is the
- * numSV of the PairTable, which stores only surprising values. This test of merging is the equal K case and is less exhaustive than TestAlltest
- * but is more practical for large values of K. Suppose our data is a stream of pairs {IP address, User ID} and we want to identify the
- * IP addresses that have the most distinct User IDs. Or conversely, we would like to identify
- * the User IDs that have the most distinct IP addresses. This is a common challenge in the
- * analysis of big data and the FDT sketch helps solve this problem using probabilistic techniques.
- *
- * More generally, given a multiset of tuples with dimensions {d1,d2, d3, ..., dN},
- * and a primary subset of dimensions M < N, our task is to identify the combinations of
- * M subset dimensions that have the most frequent number of distinct combinations of
- * the N-M non-primary dimensions.
- *
- * Please refer to the web page
- *
- * https://datasketches.apache.org/docs/Frequency/FrequentDistinctTuplesSketch.html for a more
- * complete discussion about this sketch.
- *
- * @author Lee Rhodes
- */
-public final class FdtSketch extends ArrayOfStringsTupleSketch {
-
- /**
- * Create new instance of Frequent Distinct Tuples sketch with the given
- * Log-base2 of required nominal entries.
- * @param lgK Log-base2 of required nominal entries.
- */
- public FdtSketch(final int lgK) {
- super(lgK);
- }
-
- /**
- * Used by deserialization.
- * @param seg the image of a FdtSketch
- * @deprecated As of 3.0.0, heapifying an UpdatableSketch is deprecated.
- * This capability will be removed in a future release.
- * Heapifying a CompactSketch is not deprecated.
- */
- @Deprecated
- FdtSketch(final MemorySegment seg) {
- super(seg);
- }
-
- /**
- * Create a new instance of Frequent Distinct Tuples sketch with a size determined by the given
- * threshold and rse.
- * @param threshold : the fraction, between zero and 1.0, of the total distinct stream length
- * that defines a "Frequent" (or heavy) item.
- * @param rse the maximum Relative Standard Error for the estimate of the distinct population of a
- * reported tuple (selected with a primary key) at the threshold.
- */
- public FdtSketch(final double threshold, final double rse) {
- super(computeLgK(threshold, rse));
- }
-
- /**
- * Copy Constructor
- * @param sketch the sketch to copy
- */
- public FdtSketch(final FdtSketch sketch) {
- super(sketch);
- }
-
- /**
- * @return a deep copy of this sketch
- */
- @Override
- public FdtSketch copy() {
- return new FdtSketch(this);
- }
-
- /**
- * Update the sketch with the given string array tuple.
- * @param tuple the given string array tuple.
- */
- public void update(final String[] tuple) {
- super.update(tuple, tuple);
- }
-
- /**
- * Returns an ordered List of Groups of the most frequent distinct population of subset tuples
- * represented by the count of entries of each group.
- * @param priKeyIndices these indices define the dimensions used for the Primary Keys.
- * @param limit the maximum number of groups to return. If this value is ≤ 0, all
- * groups will be returned.
- * @param numStdDev the number of standard deviations for the upper and lower error bounds,
- * this value is an integer and must be one of 1, 2, or 3.
- * See Number of Standard Deviations
- * @param sep the separator character
- * @return an ordered List of Groups of the most frequent distinct population of subset tuples
- * represented by the count of entries of each group.
- */
- public List Rounds the number of bits up to the smallest multiple of 64 (one long)
- * that is not smaller than the specified number.
- */
-abstract class BitArray implements MemorySegmentStatus {
- // MAX_BITS using longs, based on array indices being capped at Integer.MAX_VALUE
- protected static final long MAX_BITS = Integer.MAX_VALUE * (long) Long.SIZE;
-
- protected BitArray() {}
-
- //The position of the PositionalSegment must be the start of the bit array
- static BitArray heapify(final PositionalSegment posSeg, final boolean isEmpty) {
- return HeapBitArray.heapify(posSeg, isEmpty);
- }
-
- static BitArray wrap(final MemorySegment seg, final boolean isEmpty) {
- return DirectBitArrayR.wrap(seg, isEmpty);
- }
-
- static BitArray writableWrap(final MemorySegment wseg, final boolean isEmpty) {
- return DirectBitArray.writableWrap(wseg, isEmpty);
- }
-
- boolean isEmpty() {
- return !isDirty() && (getNumBitsSet() == 0);
- }
-
- @Override
- public abstract boolean hasMemorySegment();
-
- @Override
- public abstract boolean isOffHeap();
-
- abstract boolean isReadOnly();
-
- @Override
- public abstract boolean isSameResource(MemorySegment that);
-
- abstract boolean getBit(final long index);
-
- abstract boolean getAndSetBit(final long index);
-
- abstract void setBit(final long index);
-
- abstract long getNumBitsSet();
-
- abstract void reset();
-
- abstract long getCapacity();
-
- abstract int getArrayLength();
-
- abstract void union(final BitArray other);
-
- abstract void intersect(final BitArray other);
-
- abstract void invert();
-
- // prints the raw BitArray as 0s and 1s, one long per row
- @Override
- public String toString() {
- final StringBuilder sb = new StringBuilder();
- for (int i = 0; i < getArrayLength(); ++i) {
- sb.append(i + ": ")
- .append(printLong(getLong(i)))
- .append(LS);
- }
- return sb.toString();
- }
-
- long getSerializedSizeBytes() {
- // We only really need an int for array length but this will keep everything
- // aligned to 8 bytes.
- // Always write array length, but write numBitsSet only if empty
- return Long.BYTES * (isEmpty() ? 1L : (2L + getArrayLength()));
- }
-
- // returns the number of bytes needed for a non-empty BitArray of the requested size
- static long getSerializedSizeBytes(final long numBits) {
- if (numBits <= 0) {
- throw new SketchesArgumentException("Requested number of bits must be strictly positive");
- }
- if (numBits > MAX_BITS) {
- throw new SketchesArgumentException("Requested number of bits exceeds maximum allowed. "
- + "Requested: " + numBits + ", maximum: " + MAX_BITS);
- }
- final int numLongs = (int) ceilingMultiple2expK(numBits, 6);//Math.ceil(numBits / 64.0)
- return Long.BYTES * (numLongs + 2L);
- }
-
- abstract protected boolean isDirty();
-
- // used to get a long from the array regardless of underlying storage
- // NOT used to query individual bits
- abstract protected long getLong(final int arrayIndex);
-
- // used to set a long in the array regardless of underlying storage
- // NOT used to set individual bits
- abstract protected void setLong(final int arrayIndex, final long value);
-
- // prints a long as a series of 0s and 1s as little endian
- protected static String printLong(final long val) {
- final StringBuilder sb = new StringBuilder();
- for (int j = 0; j < Long.SIZE; ++j) {
- sb.append((val & (1L << j)) != 0 ? "1" : "0");
- if ((j % 8) == 7) { sb.append(" "); }
- }
- return sb.toString();
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/filters/bloomfilter/BloomFilter.java b/src/main/java/org/apache/datasketches/filters/bloomfilter/BloomFilter.java
deleted file mode 100644
index d21335c85..000000000
--- a/src/main/java/org/apache/datasketches/filters/bloomfilter/BloomFilter.java
+++ /dev/null
@@ -1,860 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.filters.bloomfilter;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_LONG_UNALIGNED;
-import static org.apache.datasketches.common.Util.LS;
-
-import java.lang.foreign.MemorySegment;
-import java.nio.charset.StandardCharsets;
-
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.MemorySegmentStatus;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-import org.apache.datasketches.common.positional.PositionalSegment;
-import org.apache.datasketches.hash.XxHash;
-import org.apache.datasketches.hash.XxHash64;
-
-/**
- * A Bloom filter is a data structure that can be used for probabilistic
- * set membership.
- *
- * When querying a Bloom filter, there are no false positives. Specifically:
- * When querying an item that has already been inserted to the filter, the filter will
- * always indicate that the item is present. There is a chance of false positives, where
- * querying an item that has never been presented to the filter will indicate that the
- * item has already been seen. Consequently, any query should be interpreted as
- * "might have seen." A standard Bloom filter is unlike typical sketches in that it is not sub-linear
- * in size and does not resize itself. A Bloom filter will work up to a target number of
- * distinct items, beyond which it will saturate and the false positive rate will start to
- * increase. The size of a Bloom filter will be linear in the expected number of
- * distinct items. See the BloomFilterBuilder class for methods to create a filter, especially
- * one sized correctly for a target number of distinct elements and a target
- * false positive probability. This implementation uses xxHash64 and follows the approach in Kirsch and Mitzenmacher,
- * "Less Hashing, Same Performance: Building a Better Bloom Filter," Wiley Interscience, 2008, pp. 187-218. Note: this will not produce the same output hash values as the {@link #update(char[])}
- * method and will generally be a little slower depending on the complexity of the UTF8 encoding.
- * Note: this will not produce the same output hash values as the {@link #queryAndUpdate(char[])}
- * method and will generally be a little slower depending on the complexity of the UTF8 encoding.
- * Note: this will not produce the same output hash values as the {@link #update(char[])}
- * method and will generally be a little slower depending on the complexity of the UTF8 encoding.
- * Note: Method throws if the serialized size exceeds The underlying math is described in the
- *
- * Wikipedia article on Bloom filters. Rounds the number of bits up to the smallest multiple of 64 (one long)
- * that is not smaller than the specified number.
- */
-final class HeapBitArray extends BitArray {
- private long numBitsSet_; // if -1, need to recompute value
- private boolean isDirty_;
- final private long[] data_;
-
- // creates an array of a given size
- HeapBitArray(final long numBits) {
- if (numBits <= 0) {
- throw new SketchesArgumentException("Number of bits must be strictly positive. Found: " + numBits);
- }
- if (numBits > MAX_BITS) {
- throw new SketchesArgumentException("Number of bits may not exceed " + MAX_BITS + ". Found: " + numBits);
- }
-
- final int numLongs = (int) ceilingMultiple2expK(numBits, 6);//Math.ceil(numBits / 64.0)
- numBitsSet_ = 0;
- isDirty_ = false;
- data_ = new long[numLongs];
- }
-
- // uses the provided array
- HeapBitArray(final long numBitsSet, final long[] data) {
- data_ = data;
- isDirty_ = numBitsSet < 0;
- numBitsSet_ = numBitsSet;
- }
-
- // reads a serialized image, but the BitArray is not fully self-describing so requires
- // a flag to indicate whether the array is empty
- // The PositionslSegment's position must be set to the start of the bit array.
- static HeapBitArray heapify(final PositionalSegment posSeg, final boolean isEmpty) {
- final int numLongs = posSeg.getInt();
- if (numLongs < 0) {
- throw new SketchesArgumentException("Possible corruption: Must have non-negative array size. Found: " + numLongs);
- }
-
- if (isEmpty) {
- return new HeapBitArray((long) numLongs * Long.SIZE);
- }
-
- posSeg.getInt(); // unused
-
- // will be -1 if dirty
- final long numBitsSet = posSeg.getLong();
-
- final long[] data = new long[numLongs];
- posSeg.getLongArray(data, 0, numLongs);
- return new HeapBitArray(numBitsSet, data);
- }
-
- @Override
- protected boolean isDirty() {
- return isDirty_;
- }
-
- @Override
- public boolean hasMemorySegment() {
- return false;
- }
-
- @Override
- public boolean isOffHeap() {
- return false;
- }
-
- @Override
- boolean isReadOnly() { return false; }
-
- @Override
- public boolean isSameResource(final MemorySegment that) { return false; }
-
- // queries a single bit in the array of longs
- @Override
- boolean getBit(final long index) {
- return ((data_[(int) index >>> 6] & (1L << index)) != 0);
- }
-
- // sets a single bit in the array of longs without querying, meaning the method
- // cannot properly track the number of bits set, so set isDirty = true
- @Override
- void setBit(final long index) {
- data_[(int) index >>> 6] |= 1L << index;
- isDirty_ = true;
- }
-
- // returns existing value of bit in an array of longs
- @Override
- boolean getAndSetBit(final long index) {
- final int offset = (int) index >>> 6;
- final long mask = 1L << index;
- if ((data_[offset] & mask) != 0) {
- return true; // already seen
- } else {
- data_[offset] |= mask;
- ++numBitsSet_; // increment regardless of isDirty_
- return false; // new set
- }
- }
-
- // may need to recompute value:
- // O(1) if only getAndSetBit() has been used
- // O(data_.length) if setBit() has ever been used
- @Override
- long getNumBitsSet() {
- if (isDirty_) {
- numBitsSet_ = 0;
- for (final long val : data_) {
- numBitsSet_ += Long.bitCount(val);
- }
- }
- return numBitsSet_;
- }
-
- @Override
- long getCapacity() { return (long) data_.length * Long.SIZE; }
-
- @Override
- int getArrayLength() { return data_.length; }
-
- // applies logical OR
- @Override
- void union(final BitArray other) {
- if (getCapacity() != other.getCapacity()) {
- throw new SketchesArgumentException("Cannot union bit arrays with unequal lengths");
- }
-
- numBitsSet_ = 0;
- for (int i = 0; i < data_.length; ++i) {
- final long val = data_[i] | other.getLong(i);
- numBitsSet_ += Long.bitCount(val);
- data_[i] = val;
- }
- isDirty_ = false;
- }
-
- // applies logical AND
- @Override
- void intersect(final BitArray other) {
- if (getCapacity() != other.getCapacity()) {
- throw new SketchesArgumentException("Cannot intersect bit arrays with unequal lengths");
- }
-
- numBitsSet_ = 0;
- for (int i = 0; i < data_.length; ++i) {
- final long val = data_[i] & other.getLong(i);
- numBitsSet_ += Long.bitCount(val);
- data_[i] = val;
- }
- isDirty_ = false;
- }
-
- // applies bitwise inversion
- @Override
- void invert() {
- if (isDirty_) {
- numBitsSet_ = 0;
- for (int i = 0; i < data_.length; ++i) {
- data_[i] = ~data_[i];
- numBitsSet_ += Long.bitCount(data_[i]);
- }
- isDirty_ = false;
- } else {
- for (int i = 0; i < data_.length; ++i) {
- data_[i] = ~data_[i];
- }
- numBitsSet_ = getCapacity() - numBitsSet_;
- }
- }
-
- void writeToSegmentAsStream(final PositionalSegment posSeg) { //position = 16
- posSeg.setInt(data_.length);
- posSeg.setInt(0); // unused
-
- if (!isEmpty()) {
- posSeg.setLong(isDirty_ ? -1 : numBitsSet_);
- posSeg.setLongArray(data_, 0, data_.length);
- }
- }
-
- @Override
- protected long getLong(final int arrayIndex) {
- return data_[arrayIndex];
- }
-
- @Override
- protected void setLong(final int arrayIndex, final long value) {
- data_[arrayIndex] = value;
- }
-
- // clears the array
- @Override
- void reset() {
- Arrays.fill(data_, 0);
- numBitsSet_ = 0;
- isDirty_ = false;
- }
-}
diff --git a/src/main/java/org/apache/datasketches/filters/bloomfilter/package-info.java b/src/main/java/org/apache/datasketches/filters/bloomfilter/package-info.java
deleted file mode 100644
index 4823fcca7..000000000
--- a/src/main/java/org/apache/datasketches/filters/bloomfilter/package-info.java
+++ /dev/null
@@ -1,23 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * BloomFilter package
- */
-package org.apache.datasketches.filters.bloomfilter;
diff --git a/src/main/java/org/apache/datasketches/filters/package-info.java b/src/main/java/org/apache/datasketches/filters/package-info.java
deleted file mode 100644
index 93410e1c5..000000000
--- a/src/main/java/org/apache/datasketches/filters/package-info.java
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * The filters package contains data structures used to determine
- * approximate set-membership. Classes in this package may not follow
- * the standard sub-linear properties of other offerings in this
- * library, but they fit with the spirit of DataSketches by providing
- * fast and approximate answers to complex problems.
- */
-package org.apache.datasketches.filters;
diff --git a/src/main/java/org/apache/datasketches/frequencies/ErrorType.java b/src/main/java/org/apache/datasketches/frequencies/ErrorType.java
deleted file mode 100644
index 050462acd..000000000
--- a/src/main/java/org/apache/datasketches/frequencies/ErrorType.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.frequencies;
-
-/**
- * Specifies one of two types of error regions of the statistical classification Confusion Matrix
- * that can be excluded from a returned sample of Frequent Items.
- */
-public enum ErrorType {
-
- /**
- * No Type I error samples will be included in the sample set,
- * which means all Truly Negative samples will be excluded from the sample set.
- * However, there may be Type II error samples (False Negatives)
- * that should have been included that were not.
- * This is a subset of the NO_FALSE_NEGATIVES ErrorType.
- */
- NO_FALSE_POSITIVES,
-
- /**
- * No Type II error samples will be excluded from the sample set,
- * which means all Truly Positive samples will be included in the sample set.
- * However, there may be Type I error samples (False Positives)
- * that were included that should not have been.
- */
- NO_FALSE_NEGATIVES
-}
diff --git a/src/main/java/org/apache/datasketches/frequencies/FrequentItemsSketch.java b/src/main/java/org/apache/datasketches/frequencies/FrequentItemsSketch.java
deleted file mode 100644
index 857d8e1f6..000000000
--- a/src/main/java/org/apache/datasketches/frequencies/FrequentItemsSketch.java
+++ /dev/null
@@ -1,751 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.frequencies;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_LONG_UNALIGNED;
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.common.Util.checkBounds;
-import static org.apache.datasketches.common.Util.exactLog2OfInt;
-import static org.apache.datasketches.common.Util.isPowerOf2;
-import static org.apache.datasketches.frequencies.PreambleUtil.EMPTY_FLAG_MASK;
-import static org.apache.datasketches.frequencies.PreambleUtil.SER_VER;
-import static org.apache.datasketches.frequencies.PreambleUtil.extractActiveItems;
-import static org.apache.datasketches.frequencies.PreambleUtil.extractFamilyID;
-import static org.apache.datasketches.frequencies.PreambleUtil.extractFlags;
-import static org.apache.datasketches.frequencies.PreambleUtil.extractLgCurMapSize;
-import static org.apache.datasketches.frequencies.PreambleUtil.extractLgMaxMapSize;
-import static org.apache.datasketches.frequencies.PreambleUtil.extractPreLongs;
-import static org.apache.datasketches.frequencies.PreambleUtil.extractSerVer;
-import static org.apache.datasketches.frequencies.PreambleUtil.insertActiveItems;
-import static org.apache.datasketches.frequencies.PreambleUtil.insertFamilyID;
-import static org.apache.datasketches.frequencies.PreambleUtil.insertFlags;
-import static org.apache.datasketches.frequencies.PreambleUtil.insertLgCurMapSize;
-import static org.apache.datasketches.frequencies.PreambleUtil.insertLgMaxMapSize;
-import static org.apache.datasketches.frequencies.PreambleUtil.insertPreLongs;
-import static org.apache.datasketches.frequencies.PreambleUtil.insertSerVer;
-import static org.apache.datasketches.frequencies.Util.LG_MIN_MAP_SIZE;
-import static org.apache.datasketches.frequencies.Util.SAMPLE_SIZE;
-
-import java.lang.foreign.MemorySegment;
-import java.lang.reflect.Array;
-import java.util.ArrayList;
-import java.util.Comparator;
-import java.util.Objects;
-
-import org.apache.datasketches.common.ArrayOfItemsSerDe;
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * This sketch is based on the paper https://arxiv.org/abs/1705.07001
- * ("A High-Performance Algorithm for Identifying Frequent Items in Data Streams"
- * by Daniel Anderson, Pryce Bevan, Kevin Lang, Edo Liberty, Lee Rhodes, and Justin Thaler)
- * and is useful for tracking approximate frequencies of items of type <T>
- * with optional associated counts (<T> item, long count) that are members of a
- * multiset of such items. The true frequency of an item is defined to be the sum of associated
- * counts.
- *
- * This implementation provides the following capabilities: Space Usage The sketch is initialized with a maxMapSize that specifies the maximum physical
- * length of the internal hash map of the form (<T> item, long count).
- * The maxMapSize must be a power of 2. The hash map starts at a very small size (8 entries), and grows as needed up to the
- * specified maxMapSize. Excluding external space required for the item objects, the internal memory space usage of
- * this sketch is 18 * mapSize bytes (assuming 8 bytes for each Java reference), plus a small
- * constant number of additional bytes. The internal memory space usage of this sketch will never
- * exceed 18 * maxMapSize bytes, plus a small constant number of additional bytes. Maximum Capacity of the Sketch The LOAD_FACTOR for the hash map is internally set at 75%,
- * which means at any time the map capacity of (item, count) pairs is mapCap = 0.75 *
- * mapSize.
- * The maximum capacity of (item, count) pairs of the sketch is maxMapCap = 0.75 *
- * maxMapSize. Updating the sketch with (item, count) pairs If the item is found in the hash map, the mapped count field (the "counter") is
- * incremented by the incoming count, otherwise, a new counter "(item, count) pair" is
- * created. If the number of tracked counters reaches the maximum capacity of the hash map
- * the sketch decrements all of the counters (by an approximately computed median), and
- * removes any non-positive counters. Accuracy If fewer than 0.75 * maxMapSize different items are inserted into the sketch the
- * estimated frequencies returned by the sketch will be exact. The logic of the frequent items sketch is such that the stored counts and true counts are
- * never too different.
- * More specifically, for any item, the sketch can return an estimate of the
- * true frequency of item, along with upper and lower bounds on the frequency
- * (that hold deterministically). For this implementation and for a specific active item, it is guaranteed that
- * the true frequency will be between the Upper Bound (UB) and the Lower Bound (LB) computed for
- * that item. Specifically, (UB- LB) ≤ W * epsilon, where W denotes the
- * sum of all item counts, and epsilon = 3.5/M, where M is the maxMapSize. This is a worst case guarantee that applies to arbitrary inputs.1
- * For inputs typically seen in practice (UB-LB) is usually much smaller.
- * Background This code implements a variant of what is commonly known as the "Misra-Gries
- * algorithm". Variants of it were discovered and rediscovered and redesigned several times
- * over the years: The method first examines all active items in the sketch (items that have a counter).
- *
- * If ErrorType = NO_FALSE_NEGATIVES, this will include an item in the result
- * list if getUpperBound(item) > threshold.
- * There will be no false negatives, i.e., no Type II error.
- * There may be items in the set with true frequencies less than the threshold
- * (false positives). If ErrorType = NO_FALSE_POSITIVES, this will include an item in the result
- * list if getLowerBound(item) > threshold.
- * There will be no false positives, i.e., no Type I error.
- * There may be items omitted from the set with true frequencies greater than the
- * threshold (false negatives). This implementation provides the following capabilities: Space Usage The sketch is initialized with a maxMapSize that specifies the maximum physical
- * length of the internal hash map of the form (long item, long count).
- * The maxMapSize must be a power of 2. The hash map starts at a very small size (8 entries), and grows as needed up to the
- * specified maxMapSize. At any moment the internal memory space usage of this sketch is 18 * mapSize bytes,
- * plus a small constant number of additional bytes. The maximum internal memory space usage of
- * this sketch will never exceed 18 * maxMapSize bytes, plus a small constant number of
- * additional bytes. Maximum Capacity of the Sketch The LOAD_FACTOR for the hash map is internally set at 75%,
- * which means at any time the map capacity of (item, count) pairs is mapCap =
- * 0.75 * mapSize.
- * The maximum capacity of (item, count) pairs of the sketch is maxMapCap =
- * 0.75 * maxMapSize. Updating the sketch with (item, count) pairs If the item is found in the hash map, the mapped count field (the "counter") is
- * incremented by the incoming count, otherwise, a new counter "(item, count) pair" is
- * created. If the number of tracked counters reaches the maximum capacity of the hash map
- * the sketch decrements all of the counters (by an approximately computed median), and
- * removes any non-positive counters. Accuracy If fewer than 0.75 * maxMapSize different items are inserted into the sketch the
- * estimated frequencies returned by the sketch will be exact. The logic of the frequent items sketch is such that the stored counts and true counts are
- * never too different.
- * More specifically, for any item, the sketch can return an estimate of the
- * true frequency of item, along with upper and lower bounds on the frequency
- * (that hold deterministically). For this implementation and for a specific active item, it is guaranteed that
- * the true frequency will be between the Upper Bound (UB) and the Lower Bound (LB) computed for
- * that item. Specifically, (UB- LB) ≤ W * epsilon, where W denotes the
- * sum of all item counts, and epsilon = 3.5/M, where M is the maxMapSize. This is a worst case guarantee that applies to arbitrary inputs.1
- * For inputs typically seen in practice (UB-LB) is usually much smaller.
- * Background This code implements a variant of what is commonly known as the "Misra-Gries
- * algorithm". Variants of it were discovered and rediscovered and redesigned several times
- * over the years: The method first examines all active items in the sketch (items that have a counter).
- *
- * If ErrorType = NO_FALSE_NEGATIVES, this will include an item in the result
- * list if getUpperBound(item) > threshold.
- * There will be no false negatives, i.e., no Type II error.
- * There may be items in the set with true frequencies less than the threshold
- * (false positives). If ErrorType = NO_FALSE_POSITIVES, this will include an item in the result
- * list if getLowerBound(item) > threshold.
- * There will be no false positives, i.e., no Type I error.
- * There may be items omitted from the set with true frequencies greater than the
- * threshold (false negatives). This is a subset of the NO_FALSE_NEGATIVES case. The intent of the design of this class was to isolate the detailed knowledge of the bit and byte
- * layout of the serialized form of the sketches derived from the Sketch class into one place. This
- * allows the possibility of the introduction of different serialization schemes with minimal impact
- * on the rest of the library. MAP: Low significance bytes of this long data structure are on the right. However, the
- * multi-byte integers (int and long) are stored in native byte order. The byte
- * values are treated as unsigned. An empty FrequentItems only requires 8 bytes. All others require 32 bytes of preamble.
- * Austin Appleby's C++
- *
- * MurmurHash3_x64_128(...), final revision 150,
- * which is in the Public Domain, was the inspiration for this implementation in Java.
- *
- * This java implementation pays close attention to the C++ algorithms in order to
- * maintain bit-wise compatibility, but the design is quite different. This implementation has also
- * been extended to include processing of arrays of longs, char or ints, which was not part of the
- * original C++ implementation. This implementation produces the same exact output hash bits as
- * the above C++ method given the same input. In addition, with this implementation, the hash of byte[], char[], int[], or long[] will
- * produce the same hash result if, and only if, all the arrays have the same exact length in
- * bytes, and if the contents of the values in the arrays have the same byte endianness and
- * overall order. There is a unit test for this class that demonstrates this.
- * The structure of this implementation also reflects a separation of code that is dependent on the
- * input structure (in this case byte[], int[] or long[]) from code that is independent of the input
- * structure. This also makes the code more readable and suitable for future extensions.
- * Note that even though this hash function produces 128 bits, the entropy of the resulting hash cannot
- * be greater than the entropy of the input. For example, if the input is only a single long of 64 bits,
- * the entropy of the resulting 128 bit hash is no greater than 64 bits.
- *
- * @author Lee Rhodes
- */
-public final class MurmurHash3 implements Serializable {
- private static final long serialVersionUID = 0L;
-
- private MurmurHash3() {}
-
- //--Hash of long---------------------------------------------------------
- /**
- * Hash the given long.
- *
- * @param key The input long.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final long key, final long seed) {
- final HashState hashState = new HashState(seed, seed);
- return hashState.finalMix128(key, 0, Long.BYTES);
- }
-
- //--Hash of long[]-------------------------------------------------------
- /**
- * Hash the given long[] array.
- *
- * @param key The input long[] array. It must be non-null and non-empty.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final long[] key, final long seed) {
- return hash(key, 0, key.length, seed);
- }
-
- /**
- * Hash a portion of the given long[] array.
- *
- * @param key The input long[] array. It must be non-null and non-empty.
- * @param offsetLongs the starting offset in longs.
- * @param lengthLongs the length in longs of the portion of the array to be hashed.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2
- */
- public static long[] hash(final long[] key, final int offsetLongs, final int lengthLongs, final long seed) {
- Objects.requireNonNull(key);
- final int arrLen = key.length;
- checkPositive(arrLen);
- Util.checkBounds(offsetLongs, lengthLongs, arrLen);
- final HashState hashState = new HashState(seed, seed);
-
- // Number of full 128-bit blocks of 2 longs (the body).
- // Possible exclusion of a remainder of 1 long.
- final int nblocks = lengthLongs >>> 1; //longs / 2
-
- // Process the 128-bit blocks (the body) into the hash
- for (int i = 0; i < nblocks; i++ ) {
- final long k1 = key[offsetLongs + (i << 1)]; //offsetLongs + 0, 2, 4, ...
- final long k2 = key[offsetLongs + (i << 1) + 1]; //offsetLongs + 1, 3, 5, ...
- hashState.blockMix128(k1, k2);
- }
-
- // Get the tail index wrt hashed portion, remainder length
- final int tail = nblocks << 1; // 2 longs / block
- final int rem = lengthLongs - tail; // remainder longs: 0,1
-
- // Get the tail
- final long k1 = rem == 0 ? 0 : key[offsetLongs + tail]; //k2 -> 0
- // Mix the tail into the hash and return
- return hashState.finalMix128(k1, 0, lengthLongs << 3); //convert to bytes
- }
-
- //--Hash of int[]--------------------------------------------------------
- /**
- * Hash the given int[] array.
- *
- * @param key The input int[] array. It must be non-null and non-empty.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final int[] key, final long seed) {
- return hash(key, 0, key.length, seed);
- }
-
- /**
- * Hash a portion of the given int[] array.
- *
- * @param key The input int[] array. It must be non-null and non-empty.
- * @param offsetInts the starting offset in ints.
- * @param lengthInts the length in ints of the portion of the array to be hashed.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final int[] key, final int offsetInts, final int lengthInts, final long seed) {
- Objects.requireNonNull(key);
- final int arrLen = key.length;
- checkPositive(arrLen);
- Util.checkBounds(offsetInts, lengthInts, arrLen);
- final HashState hashState = new HashState(seed, seed);
-
- // Number of full 128-bit blocks of 4 ints.
- // Possible exclusion of a remainder of up to 3 ints.
- final int nblocks = lengthInts >>> 2; //ints / 4
-
- // Process the 128-bit blocks (the body) into the hash
- for (int i = 0; i < nblocks; i++ ) { //4 ints per block
- final long k1 = getLong(key, offsetInts + (i << 2), 2); //offsetInts + 0, 4, 8, ...
- final long k2 = getLong(key, offsetInts + (i << 2) + 2, 2); //offsetInts + 2, 6, 10, ...
- hashState.blockMix128(k1, k2);
- }
-
- // Get the tail index wrt hashed portion, remainder length
- final int tail = nblocks << 2; // 4 ints per block
- final int rem = lengthInts - tail; // remainder ints: 0,1,2,3
-
- // Get the tail
- final long k1;
- final long k2;
- if (rem > 2) { //k1 -> whole; k2 -> partial
- k1 = getLong(key, offsetInts + tail, 2);
- k2 = getLong(key, offsetInts + tail + 2, rem - 2);
- }
- else { //k1 -> whole(2), partial(1) or 0; k2 == 0
- k1 = rem == 0 ? 0 : getLong(key, offsetInts + tail, rem);
- k2 = 0;
- }
- // Mix the tail into the hash and return
- return hashState.finalMix128(k1, k2, lengthInts << 2); //convert to bytes
- }
-
- //--Hash of char[]-------------------------------------------------------
- /**
- * Hash the given char[] array.
- *
- * @param key The input char[] array. It must be non-null and non-empty.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2
- */
- public static long[] hash(final char[] key, final long seed) {
- return hash(key, 0, key.length, seed);
- }
-
- /**
- * Hash a portion of the given char[] array.
- *
- * @param key The input char[] array. It must be non-null and non-empty.
- * @param offsetChars the starting offset in chars.
- * @param lengthChars the length in chars of the portion of the array to be hashed.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2
- */
- public static long[] hash(final char[] key, final int offsetChars, final int lengthChars, final long seed) {
- Objects.requireNonNull(key);
- final int arrLen = key.length;
- checkPositive(arrLen);
- Util.checkBounds(offsetChars, lengthChars, arrLen);
- final HashState hashState = new HashState(seed, seed);
-
- // Number of full 128-bit blocks of 8 chars.
- // Possible exclusion of a remainder of up to 7 chars.
- final int nblocks = lengthChars >>> 3; //chars / 8
-
- // Process the 128-bit blocks (the body) into the hash
- for (int i = 0; i < nblocks; i++ ) { //8 chars per block
- final long k1 = getLong(key, offsetChars + (i << 3), 4); //offsetChars + 0, 8, 16, ...
- final long k2 = getLong(key, offsetChars + (i << 3) + 4, 4); //offsetChars + 4, 12, 20, ...
- hashState.blockMix128(k1, k2);
- }
-
- // Get the tail index wrt hashed portion, remainder length
- final int tail = nblocks << 3; // 8 chars per block
- final int rem = lengthChars - tail; // remainder chars: 0,1,2,3,4,5,6,7
-
- // Get the tail
- final long k1;
- final long k2;
- if (rem > 4) { //k1 -> whole; k2 -> partial
- k1 = getLong(key, offsetChars + tail, 4);
- k2 = getLong(key, offsetChars + tail + 4, rem - 4);
- }
- else { //k1 -> whole, partial or 0; k2 == 0
- k1 = rem == 0 ? 0 : getLong(key, offsetChars + tail, rem);
- k2 = 0;
- }
- // Mix the tail into the hash and return
- return hashState.finalMix128(k1, k2, lengthChars << 1); //convert to bytes
- }
-
- //--Hash of byte[]-------------------------------------------------------
- /**
- * Hash the given byte[] array.
- *
- * @param key The input byte[] array. It must be non-null and non-empty.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final byte[] key, final long seed) {
- return hash(key, 0, key.length, seed);
- }
-
- /**
- * Hash a portion of the given byte[] array.
- *
- * @param key The input byte[] array. It must be non-null and non-empty.
- * @param offsetBytes the starting offset in bytes.
- * @param lengthBytes the length in bytes of the portion of the array to be hashed.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final byte[] key, final int offsetBytes, final int lengthBytes, final long seed) {
- Objects.requireNonNull(key);
- final int arrLen = key.length;
- checkPositive(arrLen);
- Util.checkBounds(offsetBytes, lengthBytes, arrLen);
- final HashState hashState = new HashState(seed, seed);
-
- // Number of full 128-bit blocks of 16 bytes.
- // Possible exclusion of a remainder of up to 15 bytes.
- final int nblocks = lengthBytes >>> 4; //bytes / 16
-
- // Process the 128-bit blocks (the body) into the hash
- for (int i = 0; i < nblocks; i++ ) { //16 bytes per block
- final long k1 = getLong(key, offsetBytes + (i << 4), 8); //offsetBytes + 0, 16, 32, ...
- final long k2 = getLong(key, offsetBytes + (i << 4) + 8, 8); //offsetBytes + 8, 24, 40, ...
- hashState.blockMix128(k1, k2);
- }
-
- // Get the tail index wrt hashed portion, remainder length
- final int tail = nblocks << 4; //16 bytes per block
- final int rem = lengthBytes - tail; // remainder bytes: 0,1,...,15
-
- // Get the tail
- final long k1;
- final long k2;
- if (rem > 8) { //k1 -> whole; k2 -> partial
- k1 = getLong(key, offsetBytes + tail, 8);
- k2 = getLong(key, offsetBytes + tail + 8, rem - 8);
- }
- else { //k1 -> whole, partial or 0; k2 == 0
- k1 = rem == 0 ? 0 : getLong(key, offsetBytes + tail, rem);
- k2 = 0;
- }
- // Mix the tail into the hash and return
- return hashState.finalMix128(k1, k2, lengthBytes);
- }
-
- //--Hash of ByteBuffer---------------------------------------------------
- /**
- * Hash the remaining bytes of the given ByteBuffer starting at position() ending at limit (exclusive).
- *
- * @param buf The input ByteBuffer. It must be non-null and non-empty.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final ByteBuffer buf, final long seed) {
- Objects.requireNonNull(buf);
- final MemorySegment bbSeg = MemorySegment.ofBuffer(buf);
- return hash(bbSeg, seed);
- }
-
- //--Hash of MemorySegment-------------------------------------------------------
- /**
- * Hash the given MemorySegment.
- *
- * Note: if you want to hash only a portion of MemorySegment, convert it to the
- * appropriate Slice first.
- *
- * @param seg The input MemorySegment. It must be non-null and non-empty.
- * @param seed A long valued seed.
- * @return a 128-bit hash of the input as a long array of size 2.
- */
- public static long[] hash(final MemorySegment seg, final long seed) {
- Objects.requireNonNull(seg);
- final long lengthBytes = seg.byteSize();
- checkPositive(lengthBytes);
-
- final HashState hashState = new HashState(seed, seed);
-
- // Number of full 128-bit blocks of 16 bytes.
- // Possible exclusion of a remainder of up to 15 bytes.
- final long nblocks = lengthBytes >>> 4; //bytes / 16
-
- // Process the 128-bit blocks (the body) into the hash
- for (long i = 0; i < nblocks; i++ ) { //16 bytes per block
- final long k1 = seg.get(JAVA_LONG_UNALIGNED, i << 4); //0, 16, 32, ...
- final long k2 = seg.get(JAVA_LONG_UNALIGNED, (i << 4) + 8); //8, 24, 40, ...
- hashState.blockMix128(k1, k2);
- }
-
- // Get the tail index wrt hashed portion, remainder length
- final long tail = nblocks << 4; //16 bytes per block
- final int rem = (int)(lengthBytes - tail); // remainder bytes: 0,1,...,15
-
- // Get the tail
- final long k1;
- final long k2;
- if (rem > 8) { //k1 -> whole; k2 -> partial
- k1 = seg.get(JAVA_LONG_UNALIGNED, tail);
- k2 = getLong(seg, tail + 8, rem - 8);
- }
- else { //k1 -> whole, partial or 0; k2 == 0
- k1 = rem == 0 ? 0 : getLong(seg, tail, rem);
- k2 = 0;
- }
- // Mix the tail into the hash and return
- return hashState.finalMix128(k1, k2, lengthBytes);
- }
-
- //--HashState class------------------------------------------------------
- /**
- * Common processing of the 128-bit hash state independent of input type.
- */
- private static final class HashState {
- private static final long C1 = 0x87c37b91114253d5L;
- private static final long C2 = 0x4cf5ad432745937fL;
- private long h1;
- private long h2;
-
- HashState(final long h1, final long h2) {
- this.h1 = h1;
- this.h2 = h2;
- }
-
- /**
- * Block mix (128-bit block) of input key to internal hash state.
- *
- * @param k1 intermediate mix value
- * @param k2 intermediate mix value
- */
- void blockMix128(final long k1, final long k2) {
- h1 ^= mixK1(k1);
- h1 = Long.rotateLeft(h1, 27);
- h1 += h2;
- h1 = (h1 * 5) + 0x52dce729;
-
- h2 ^= mixK2(k2);
- h2 = Long.rotateLeft(h2, 31);
- h2 += h1;
- h2 = (h2 * 5) + 0x38495ab5;
- }
-
- long[] finalMix128(final long k1, final long k2, final long inputLengthBytes) {
- h1 ^= mixK1(k1);
- h2 ^= mixK2(k2);
- h1 ^= inputLengthBytes;
- h2 ^= inputLengthBytes;
- h1 += h2;
- h2 += h1;
- h1 = finalMix64(h1);
- h2 = finalMix64(h2);
- h1 += h2;
- h2 += h1;
- return new long[] { h1, h2 };
- }
-
- /**
- * Final self mix of h*.
- *
- * @param h input to final mix
- * @return mix
- */
- private static long finalMix64(long h) {
- h ^= h >>> 33;
- h *= 0xff51afd7ed558ccdL;
- h ^= h >>> 33;
- h *= 0xc4ceb9fe1a85ec53L;
- h ^= h >>> 33;
- return h;
- }
-
- /**
- * Self mix of k1
- *
- * @param k1 input argument
- * @return mix
- */
- private static long mixK1(long k1) {
- k1 *= C1;
- k1 = Long.rotateLeft(k1, 31);
- k1 *= C2;
- return k1;
- }
-
- /**
- * Self mix of k2
- *
- * @param k2 input argument
- * @return mix
- */
- private static long mixK2(long k2) {
- k2 *= C2;
- k2 = Long.rotateLeft(k2, 33);
- k2 *= C1;
- return k2;
- }
- }
-
- //--Helper methods-------------------------------------------------------
-
- /**
- * Gets a long from the given int array starting at the given int array index and continuing for
- * remainder (rem) integers. The integers are extracted in little-endian order. There is no limit
- * checking.
- *
- * @param intArr The given input int array.
- * @param index Zero-based index from the start of the int array.
- * @param rem Remainder integers. An integer in the range [1,2].
- * @return long
- */
- private static long getLong(final int[] intArr, final int index, final int rem) {
- long out = 0L;
- for (int i = rem; i-- > 0;) { //i= 1,0
- final int v = intArr[index + i];
- out ^= (v & 0xFFFFFFFFL) << (i * 32); //equivalent to |=
- }
- return out;
- }
-
- /**
- * Gets a long from the given char array starting at the given char array index and continuing for
- * remainder (rem) chars. The chars are extracted in little-endian order. There is no limit
- * checking.
- *
- * @param charArr The given input char array.
- * @param index Zero-based index from the start of the char array.
- * @param rem Remainder chars. An integer in the range [1,4].
- * @return a long
- */
- private static long getLong(final char[] charArr, final int index, final int rem) {
- long out = 0L;
- for (int i = rem; i-- > 0;) { //i= 3,2,1,0
- final char c = charArr[index + i];
- out ^= (c & 0xFFFFL) << (i * 16); //equivalent to |=
- }
- return out;
- }
-
- /**
- * Gets a long from the given byte array starting at the given byte array index and continuing for
- * remainder (rem) bytes. The bytes are extracted in little-endian order. There is no limit
- * checking.
- *
- * @param bArr The given input byte array.
- * @param index Zero-based index from the start of the byte array.
- * @param rem Remainder bytes. An integer in the range [1,8].
- * @return a long
- */
- private static long getLong(final byte[] bArr, final int index, final int rem) {
- long out = 0L;
- for (int i = rem; i-- > 0;) { //i= 7,6,5,4,3,2,1,0
- final byte b = bArr[index + i];
- out ^= (b & 0xFFL) << (i * 8); //equivalent to |=
- }
- return out;
- }
-
- /**
- * Gets a long from the given MemorySegment starting at the given offsetBytes and continuing for
- * remainder (rem) bytes. The bytes are extracted in little-endian order. There is no limit
- * checking.
- *
- * @param seg The given input MemorySegment.
- * @param offsetBytes Zero-based offset in bytes from the start of the MemorySegment.
- * @param rem Remainder bytes. An integer in the range [1,8].
- * @return a long
- */
- private static long getLong(final MemorySegment seg, final long offsetBytes, final int rem) {
- long out = 0L;
- if (rem == 8) {
- return seg.get(JAVA_LONG_UNALIGNED, offsetBytes);
- }
- for (int i = rem; i-- > 0; ) { //i= 7,6,5,4,3,2,1,0
- final byte b = seg.get(JAVA_BYTE, offsetBytes + i);
- out ^= (b & 0xFFL) << (i << 3); //equivalent to |=
- }
- return out;
- }
-
- private static void checkPositive(final long size) {
- if (size <= 0) {
- throw new SketchesArgumentException("Array size must not be negative or zero: " + size);
- }
- }
-}
diff --git a/src/main/java/org/apache/datasketches/hash/MurmurHash3FFM.java b/src/main/java/org/apache/datasketches/hash/MurmurHash3FFM.java
deleted file mode 100644
index 722aba8bb..000000000
--- a/src/main/java/org/apache/datasketches/hash/MurmurHash3FFM.java
+++ /dev/null
@@ -1,373 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hash;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_LONG_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_SHORT_UNALIGNED;
-import static java.nio.charset.StandardCharsets.UTF_8;
-
-import java.lang.foreign.MemorySegment;
-import java.util.Objects;
-
-/**
- * The MurmurHash3 is a fast, non-cryptographic, 128-bit hash function that has
- * excellent avalanche and 2-way bit independence properties.
- *
- * Austin Appleby's C++
- *
- * MurmurHash3_x64_128(...), final revision 150,
- * which is in the Public Domain, was the inspiration for this implementation in Java. This implementation of the MurmurHash3 allows hashing of a block of on-heap MemorySegment defined by an offset
- * and length. The calling API also allows the user to supply the small output array of two longs,
- * so that the entire hash function is static and free of object allocations. This implementation produces exactly the same hash result as the
- * MurmurHash3 function in datasketches-java given compatible inputs. This FFM version of the implementation leverages the java.lang.foreign package (FFM) of JDK-25 in place of
- * the Unsafe class.
- *
- * @author Lee Rhodes
- */
-public final class MurmurHash3FFM {
- private static final long C1 = 0x87c37b91114253d5L;
- private static final long C2 = 0x4cf5ad432745937fL;
-
- private MurmurHash3FFM() { }
-
- /**
- * Returns a 128-bit hash of the input.
- * Provided for compatibility with older version of MurmurHash3,
- * but empty or null input now throws IllegalArgumentException.
- * @param in long array
- * @param seed A long valued seed.
- * @return the hash
- * @throws IllegalArgumentException if input is empty or null
- */
- public static long[] hash(final long[] in, final long seed) {
- if ((in == null) || (in.length == 0)) {
- throw new IllegalArgumentException("Input in is empty or null.");
- }
- return hash(MemorySegment.ofArray(in), 0L, in.length << 3, seed, new long[2]);
- }
-
- /**
- * Returns a 128-bit hash of the input.
- * Provided for compatibility with older version of MurmurHash3,
- * but empty or null input now throws IllegalArgumentException.
- * @param in int array
- * @param seed A long valued seed.
- * @return the hash
- * @throws IllegalArgumentException if input is empty or null
- */
- public static long[] hash(final int[] in, final long seed) {
- if ((in == null) || (in.length == 0)) {
- throw new IllegalArgumentException("Input in is empty or null.");
- }
- return hash(MemorySegment.ofArray(in), 0L, in.length << 2, seed, new long[2]);
- }
-
- /**
- * Returns a 128-bit hash of the input.
- * Provided for compatibility with older version of MurmurHash3,
- * but empty or null input now throws IllegalArgumentException.
- * @param in char array
- * @param seed A long valued seed.
- * @return the hash
- * @throws IllegalArgumentException if input is empty or null
- */
- public static long[] hash(final char[] in, final long seed) {
- if ((in == null) || (in.length == 0)) {
- throw new IllegalArgumentException("Input in is empty or null.");
- }
- return hash(MemorySegment.ofArray(in), 0L, in.length << 1, seed, new long[2]);
- }
-
- /**
- * Returns a 128-bit hash of the input.
- * Provided for compatibility with older version of MurmurHash3,
- * but empty or null input now throws IllegalArgumentException.
- * @param in byte array
- * @param seed A long valued seed.
- * @return the hash
- * @throws IllegalArgumentException if input is empty or null
- */
- public static long[] hash(final byte[] in, final long seed) {
- if ((in == null) || (in.length == 0)) {
- throw new IllegalArgumentException("Input in is empty or null.");
- }
- return hash(MemorySegment.ofArray(in), 0L, in.length, seed, new long[2]);
- }
-
- //Single primitive inputs
-
- /**
- * Returns a 128-bit hash of the input.
- * Note the entropy of the resulting hash cannot be more than 64 bits.
- * @param in a long
- * @param seed A long valued seed.
- * @param hashOut A long array of size 2
- * @return the hash
- */
- public static long[] hash(final long in, final long seed, final long[] hashOut) {
- final long h1 = seed ^ mixK1(in);
- final long h2 = seed;
- return finalMix128(h1, h2, 8, hashOut);
- }
-
- /**
- * Returns a 128-bit hash of the input.
- * Note the entropy of the resulting hash cannot be more than 64 bits.
- * @param in a double
- * @param seed A long valued seed.
- * @param hashOut A long array of size 2
- * @return the hash
- */
- public static long[] hash(final double in, final long seed, final long[] hashOut) {
- final double d = (in == 0.0) ? 0.0 : in; // canonicalize -0.0, 0.0
- final long k1 = Double.doubleToLongBits(d); // canonicalize all NaN forms
- final long h1 = seed ^ mixK1(k1);
- final long h2 = seed;
- return finalMix128(h1, h2, 8, hashOut);
- }
-
- /**
- * Returns a 128-bit hash of the input.
- * An empty or null input throws IllegalArgumentException.
- * @param in a String
- * @param seed A long valued seed.
- * @param hashOut A long array of size 2
- * @return the hash
- * @throws IllegalArgumentException if input is empty or null
- */
- public static long[] hash(final String in, final long seed, final long[] hashOut) {
- if ((in == null) || (in.length() == 0)) {
- throw new IllegalArgumentException("Input in is empty or null.");
- }
- final byte[] byteArr = in.getBytes(UTF_8);
- return hash(MemorySegment.ofArray(byteArr), 0L, byteArr.length, seed, hashOut);
- }
-
- //The worker method
-
- /**
- * Returns a 128-bit hash of the input as a long array of size 2.
- *
- * @param seg The input MemorySegment. Must be non-null and non-empty,
- * otherwise throws IllegalArgumentException.
- * @param offsetBytes the starting point within MemorySegment.
- * @param lengthBytes the total number of bytes to be hashed.
- * @param seed A long valued seed.
- * @param hashOut the size 2 long array for the resulting 128-bit hash
- * @return the hash.
- * @throws IllegalArgumentException if input MemorySegment is empty
- */
- public static long[] hash(final MemorySegment seg, final long offsetBytes, final long lengthBytes,
- final long seed, final long[] hashOut) {
- Objects.requireNonNull(seg, "Input MemorySegment must not be null");
- final long segCap = seg.byteSize();
- if (segCap == 0L) { throw new IllegalArgumentException("Input MemorySegment must not be empty."); }
-
- long cumOff = offsetBytes;
- long rem = lengthBytes;
-
- long h1 = seed;
- long h2 = seed;
-
- // Process the 128-bit blocks (the body) into the hash
- while (rem >= 16L) {
- final long k1 = seg.get(JAVA_LONG_UNALIGNED, cumOff); //0, 16, 32, ...
- final long k2 = seg.get(JAVA_LONG_UNALIGNED, cumOff + 8); //8, 24, 40, ...
-
- synchronized (MurmurHash3FFM.class) {
- cumOff += 16L;
- rem -= 16L;
- }
-
- h1 ^= mixK1(k1);
- h1 = Long.rotateLeft(h1, 27);
- h1 += h2;
- h1 = (h1 * 5) + 0x52dce729L;
-
- h2 ^= mixK2(k2);
- h2 = Long.rotateLeft(h2, 31);
- h2 += h1;
- h2 = (h2 * 5) + 0x38495ab5L;
- }
-
- // Get the tail (if any): 1 to 15 bytes
- if (rem > 0L) {
- long k1 = 0;
- long k2 = 0;
- switch ((int) rem) {
- case 15: {
- k2 ^= (seg.get(JAVA_BYTE, cumOff + 14) & 0xFFL) << 48;
- }
- //$FALL-THROUGH$
- case 14: {
- k2 ^= (seg.get(JAVA_SHORT_UNALIGNED, cumOff + 12) & 0xFFFFL) << 32;
- k2 ^= seg.get(JAVA_INT_UNALIGNED, cumOff + 8) & 0xFFFFFFFFL;
- k1 = seg.get(JAVA_LONG_UNALIGNED, cumOff);
- break;
- }
-
- case 13: {
- k2 ^= (seg.get(JAVA_BYTE, cumOff + 12) & 0xFFL) << 32;
- }
- //$FALL-THROUGH$
- case 12: {
- k2 ^= seg.get(JAVA_INT_UNALIGNED, cumOff + 8) & 0xFFFFFFFFL;
- k1 = seg.get(JAVA_LONG_UNALIGNED, cumOff);
- break;
- }
-
- case 11: {
- k2 ^= (seg.get(JAVA_BYTE, cumOff + 10) & 0xFFL) << 16;
- }
- //$FALL-THROUGH$
- case 10: {
- k2 ^= seg.get(JAVA_SHORT_UNALIGNED, cumOff + 8) & 0xFFFFL;
- k1 = seg.get(JAVA_LONG_UNALIGNED, cumOff);
- break;
- }
-
- case 9: {
- k2 ^= seg.get(JAVA_BYTE, cumOff + 8) & 0xFFL;
- }
- //$FALL-THROUGH$
- case 8: {
- k1 = seg.get(JAVA_LONG_UNALIGNED, cumOff);
- break;
- }
-
- case 7: {
- k1 ^= (seg.get(JAVA_BYTE, cumOff + 6) & 0xFFL) << 48;
- }
- //$FALL-THROUGH$
- case 6: {
- k1 ^= (seg.get(JAVA_SHORT_UNALIGNED, cumOff + 4) & 0xFFFFL) << 32;
- k1 ^= seg.get(JAVA_INT_UNALIGNED, cumOff) & 0xFFFFFFFFL;
- break;
- }
-
- case 5: {
- k1 ^= (seg.get(JAVA_BYTE, cumOff + 4) & 0xFFL) << 32;
- }
- //$FALL-THROUGH$
- case 4: {
- k1 ^= seg.get(JAVA_INT_UNALIGNED, cumOff) & 0xFFFFFFFFL;
- break;
- }
-
- case 3: {
- k1 ^= (seg.get(JAVA_BYTE, cumOff + 2) & 0xFFL) << 16;
- }
- //$FALL-THROUGH$
- case 2: {
- k1 ^= seg.get(JAVA_SHORT_UNALIGNED, cumOff) & 0xFFFFL;
- break;
- }
-
- case 1: {
- k1 ^= seg.get(JAVA_BYTE, cumOff) & 0xFFL;
- break;
- }
- default: break; //can't happen
- }
-
- h1 ^= mixK1(k1);
- h2 ^= mixK2(k2);
- }
- return finalMix128(h1, h2, lengthBytes, hashOut);
- }
-
- //--Helper methods----------------------------------------------------
-
- /**
- * Self mix of k1
- *
- * @param k1 input argument
- * @return mix
- */
- private static long mixK1(long k1) {
- k1 *= C1;
- k1 = Long.rotateLeft(k1, 31);
- k1 *= C2;
- return k1;
- }
-
- /**
- * Self mix of k2
- *
- * @param k2 input argument
- * @return mix
- */
- private static long mixK2(long k2) {
- k2 *= C2;
- k2 = Long.rotateLeft(k2, 33);
- k2 *= C1;
- return k2;
- }
-
- /**
- * Final self mix of h*.
- *
- * @param h input to final mix
- * @return mix
- */
- private static long finalMix64(long h) {
- h ^= h >>> 33;
- h *= 0xff51afd7ed558ccdL;
- h ^= h >>> 33;
- h *= 0xc4ceb9fe1a85ec53L;
- h ^= h >>> 33;
- return h;
- }
-
- /**
- * Finalization: Add the length into the hash and mix
- * @param h1 intermediate hash
- * @param h2 intermediate hash
- * @param lengthBytes the length in bytes
- * @param hashOut the output array of 2 longs
- * @return hashOut
- */
- private static long[] finalMix128(long h1, long h2, final long lengthBytes, final long[] hashOut) {
- h1 ^= lengthBytes;
- h2 ^= lengthBytes;
-
- h1 += h2;
- h2 += h1;
-
- h1 = finalMix64(h1);
- h2 = finalMix64(h2);
-
- h1 += h2;
- h2 += h1;
-
- hashOut[0] = h1;
- hashOut[1] = h2;
- return hashOut;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hash/XxHash.java b/src/main/java/org/apache/datasketches/hash/XxHash.java
deleted file mode 100644
index cb3f6e0df..000000000
--- a/src/main/java/org/apache/datasketches/hash/XxHash.java
+++ /dev/null
@@ -1,202 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hash;
-
-import static org.apache.datasketches.hash.XxHash64.hash;
-import static org.apache.datasketches.hash.XxHash64.hashBytes;
-import static org.apache.datasketches.hash.XxHash64.hashChars;
-import static org.apache.datasketches.hash.XxHash64.hashDoubles;
-import static org.apache.datasketches.hash.XxHash64.hashFloats;
-import static org.apache.datasketches.hash.XxHash64.hashInts;
-import static org.apache.datasketches.hash.XxHash64.hashLongs;
-import static org.apache.datasketches.hash.XxHash64.hashShorts;
-
-/**
- * The XxHash is a fast, non-cryptographic, 64-bit hash function that has
- * excellent avalanche and 2-way bit independence properties.
- * This java version adapted the C++ version and the OpenHFT/Zero-Allocation-Hashing implementation
- * referenced below as inspiration.
- *
- * The C++ source repository:
- *
- * https://github.com/Cyan4973/xxHash. It has a BSD 2-Clause License:
- *
- * http://www.opensource.org/licenses/bsd-license.php. See LICENSE.
- *
- * Portions of this code were adapted from
- *
- * OpenHFT/Zero-Allocation-Hashing, which has an Apache 2 license as does this site. See LICENSE.
- *
- * @author Lee Rhodes
- */
-public final class XxHash {
-
- private XxHash() { /* singleton */ }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetBytes starting at this offset
- * @param lengthBytes continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashByteArr(
- final byte[] arr,
- final int offsetBytes,
- final int lengthBytes,
- final long seed) {
- return hashBytes(arr, offsetBytes, lengthBytes, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetShorts starting at this offset
- * @param lengthShorts continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashShortArr(
- final short[] arr,
- final int offsetShorts,
- final int lengthShorts,
- final long seed) {
- return hashShorts(arr, offsetShorts, lengthShorts, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetChars starting at this offset
- * @param lengthChars continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashCharArr(
- final char[] arr,
- final int offsetChars,
- final int lengthChars,
- final long seed) {
- return hashChars(arr, offsetChars, lengthChars, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetInts starting at this offset
- * @param lengthInts continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashIntArr(
- final int[] arr,
- final int offsetInts,
- final int lengthInts,
- final long seed) {
- return hashInts(arr, offsetInts, lengthInts, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetLongs starting at this offset
- * @param lengthLongs continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashLongArr(
- final long[] arr,
- final int offsetLongs,
- final int lengthLongs,
- final long seed) {
- return hashLongs(arr, offsetLongs, lengthLongs, seed);
- }
-
- /**
- * Returns a 64-bit hash from a single long. This method has been optimized for speed when only
- * a single hash of a long is required.
- * @param in A long.
- * @param seed A long valued seed.
- * @return the hash.
- */
- public static long hashLong(
- final long in,
- final long seed) {
- return hash(in, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetFloats starting at this offset
- * @param lengthFloats continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashFloatArr(
- final float[] arr,
- final int offsetFloats,
- final int lengthFloats,
- final long seed) {
- return hashFloats(arr, offsetFloats, lengthFloats, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetDoubles starting at this offset
- * @param lengthDoubles continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashDoubleArr(
- final double[] arr,
- final int offsetDoubles,
- final int lengthDoubles,
- final long seed) {
- return hashDoubles(arr, offsetDoubles, lengthDoubles, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param str the given string
- * @param offsetChars starting at this offset
- * @param lengthChars continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashString(
- final String str,
- final int offsetChars,
- final int lengthChars,
- final long seed) {
- return XxHash64.hashString(str, offsetChars, lengthChars, seed);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hash/XxHash64.java b/src/main/java/org/apache/datasketches/hash/XxHash64.java
deleted file mode 100644
index a908b6aaf..000000000
--- a/src/main/java/org/apache/datasketches/hash/XxHash64.java
+++ /dev/null
@@ -1,314 +0,0 @@
- /*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hash;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_LONG_UNALIGNED;
-
-import java.lang.foreign.MemorySegment;
-
-/**
- * The XxHash is a fast, non-cryptographic, 64-bit hash function that has
- * excellent avalanche and 2-way bit independence properties.
- * This java version adapted from the C++ version and the OpenHFT/Zero-Allocation-Hashing implementation
- * referenced below as inspiration.
- *
- * The C++ source repository:
- *
- * https://github.com/Cyan4973/xxHash. It has a BSD 2-Clause License:
- *
- * http://www.opensource.org/licenses/bsd-license.php. See LICENSE.
- *
- * Portions of this code were adapted from
- *
- * OpenHFT/Zero-Allocation-Hashing, which has an Apache 2 license as does this site. See LICENSE.
- *
- * @author Lee Rhodes
- */
-public final class XxHash64 {
- // Unsigned, 64-bit primes
- private static final long P1 = -7046029288634856825L;
- private static final long P2 = -4417276706812531889L;
- private static final long P3 = 1609587929392839161L;
- private static final long P4 = -8796714831421723037L;
- private static final long P5 = 2870177450012600261L;
- //shift constants
- private static final byte SHORT_SHIFT = 1;
- private static final byte CHAR_SHIFT = 1;
- private static final byte INT_SHIFT = 2;
- private static final byte LONG_SHIFT = 3;
- private static final byte FLOAT_SHIFT = 2;
- private static final byte DOUBLE_SHIFT = 3;
-
- private XxHash64() { }
-
- /**
- * Returns the 64-bit hash of the sequence of bytes in the given MemorySegment
- *
- * @param seg A reference to the relevant MemorySegment.
- * @param offsetBytes offset in bytes in the given segment.
- * @param lengthBytes the length in bytes to be hashed
- * @param seed a given seed
- * @return the 64-bit hash of the sequence of bytes.
- */
- public static long hash(final MemorySegment seg, long offsetBytes, final long lengthBytes, final long seed) {
- long hash;
- long remaining = lengthBytes;
-
- if (remaining >= 32) {
- long v1 = seed + P1 + P2;
- long v2 = seed + P2;
- long v3 = seed;
- long v4 = seed - P1;
-
- do {
- v1 += seg.get(JAVA_LONG_UNALIGNED, offsetBytes) * P2;
- v1 = Long.rotateLeft(v1, 31);
- v1 *= P1;
-
- v2 += seg.get(JAVA_LONG_UNALIGNED, offsetBytes + 8L) * P2;
- v2 = Long.rotateLeft(v2, 31);
- v2 *= P1;
-
- v3 += seg.get(JAVA_LONG_UNALIGNED, offsetBytes + 16L) * P2;
- v3 = Long.rotateLeft(v3, 31);
- v3 *= P1;
-
- v4 += seg.get(JAVA_LONG_UNALIGNED, offsetBytes + 24L) * P2;
- v4 = Long.rotateLeft(v4, 31);
- v4 *= P1;
-
- offsetBytes += 32;
- remaining -= 32;
- } while (remaining >= 32);
-
- hash = Long.rotateLeft(v1, 1)
- + Long.rotateLeft(v2, 7)
- + Long.rotateLeft(v3, 12)
- + Long.rotateLeft(v4, 18);
-
- v1 *= P2;
- v1 = Long.rotateLeft(v1, 31);
- v1 *= P1;
- hash ^= v1;
- hash = (hash * P1) + P4;
-
- v2 *= P2;
- v2 = Long.rotateLeft(v2, 31);
- v2 *= P1;
- hash ^= v2;
- hash = (hash * P1) + P4;
-
- v3 *= P2;
- v3 = Long.rotateLeft(v3, 31);
- v3 *= P1;
- hash ^= v3;
- hash = (hash * P1) + P4;
-
- v4 *= P2;
- v4 = Long.rotateLeft(v4, 31);
- v4 *= P1;
- hash ^= v4;
- hash = (hash * P1) + P4;
- } //end remaining >= 32
- else {
- hash = seed + P5;
- }
-
- hash += lengthBytes;
-
- while (remaining >= 8) {
- long k1 = seg.get(JAVA_LONG_UNALIGNED, offsetBytes);
- k1 *= P2;
- k1 = Long.rotateLeft(k1, 31);
- k1 *= P1;
- hash ^= k1;
- hash = (Long.rotateLeft(hash, 27) * P1) + P4;
- offsetBytes += 8;
- remaining -= 8;
- }
-
- if (remaining >= 4) { //treat as unsigned ints
- hash ^= (seg.get(JAVA_INT_UNALIGNED, offsetBytes) & 0XFFFF_FFFFL) * P1;
- hash = (Long.rotateLeft(hash, 23) * P2) + P3;
- offsetBytes += 4;
- remaining -= 4;
- }
-
- while (remaining != 0) { //treat as unsigned bytes
- hash ^= (seg.get(JAVA_BYTE, offsetBytes) & 0XFFL) * P5;
- hash = Long.rotateLeft(hash, 11) * P1;
- --remaining;
- ++offsetBytes;
- }
-
- return finalize(hash);
- }
-
- /**
- * Returns a 64-bit hash from a single long. This method has been optimized for speed when only
- * a single hash of a long is required.
- * @param in A long.
- * @param seed A long valued seed.
- * @return the hash.
- */
- public static long hash(final long in, final long seed) {
- long hash = seed + P5;
- hash += 8;
- long k1 = in;
- k1 *= P2;
- k1 = Long.rotateLeft(k1, 31);
- k1 *= P1;
- hash ^= k1;
- hash = (Long.rotateLeft(hash, 27) * P1) + P4;
- return finalize(hash);
- }
-
- private static long finalize(long hash) {
- hash ^= hash >>> 33;
- hash *= P2;
- hash ^= hash >>> 29;
- hash *= P3;
- hash ^= hash >>> 32;
- return hash;
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetBytes starting at this offset
- * @param lengthBytes continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashBytes(final byte[] arr, final int offsetBytes,
- final int lengthBytes, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(arr);
- return hash(seg, offsetBytes, lengthBytes, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetShorts starting at this offset
- * @param lengthShorts continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashShorts(final short[] arr, final int offsetShorts,
- final int lengthShorts, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(arr);
- return hash(seg, (offsetShorts << SHORT_SHIFT), lengthShorts << SHORT_SHIFT, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetChars starting at this offset
- * @param lengthChars continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashChars(final char[] arr, final int offsetChars,
- final int lengthChars, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(arr);
- return hash(seg, offsetChars << CHAR_SHIFT, lengthChars << CHAR_SHIFT, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetInts starting at this offset
- * @param lengthInts continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashInts(final int[] arr, final int offsetInts,
- final int lengthInts, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(arr);
- return hash(seg, offsetInts << INT_SHIFT, lengthInts << INT_SHIFT, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetLongs starting at this offset
- * @param lengthLongs continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashLongs(final long[] arr, final int offsetLongs,
- final int lengthLongs, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(arr);
- return hash(seg, offsetLongs << LONG_SHIFT, lengthLongs << LONG_SHIFT, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetFloats starting at this offset
- * @param lengthFloats continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashFloats(final float[] arr, final int offsetFloats,
- final int lengthFloats, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(arr);
- return hash(seg, offsetFloats << FLOAT_SHIFT, lengthFloats << FLOAT_SHIFT, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param arr the given array
- * @param offsetDoubles starting at this offset
- * @param lengthDoubles continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashDoubles(final double[] arr, final int offsetDoubles,
- final int lengthDoubles, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(arr);
- return hash(seg, offsetDoubles << DOUBLE_SHIFT, lengthDoubles << DOUBLE_SHIFT, seed);
- }
-
- /**
- * Hash the given arr starting at the given offset and continuing for the given length using the
- * given seed.
- * @param str the given string
- * @param offsetChars starting at this offset
- * @param lengthChars continuing for this length
- * @param seed the given seed
- * @return the hash
- */
- public static long hashString(final String str, final int offsetChars,
- final int lengthChars, final long seed) {
- final MemorySegment seg = MemorySegment.ofArray(str.toCharArray());
- return hash(seg, offsetChars << CHAR_SHIFT, lengthChars << CHAR_SHIFT, seed);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hash/package-info.java b/src/main/java/org/apache/datasketches/hash/package-info.java
deleted file mode 100644
index 5744b2776..000000000
--- a/src/main/java/org/apache/datasketches/hash/package-info.java
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * The hash package contains a high-performing and extended Java implementations
- * of Austin Appleby's 128-bit MurmurHash3 hash function originally coded in C.
- * This core MurmurHash3.java class is used throughout many of the sketch classes for consistency
- * and as long as the user specifies the same seed will result in coordinated hash operations.
- * This package also contains an adaptor class that extends the basic class with more functions
- * commonly associated with hashing.
- */
-package org.apache.datasketches.hash;
diff --git a/src/main/java/org/apache/datasketches/hll/AbstractCoupons.java b/src/main/java/org/apache/datasketches/hll/AbstractCoupons.java
deleted file mode 100644
index 62727ebed..000000000
--- a/src/main/java/org/apache/datasketches/hll/AbstractCoupons.java
+++ /dev/null
@@ -1,160 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.Math.max;
-import static org.apache.datasketches.hll.HllUtil.COUPON_RSE;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.KEY_MASK_26;
-import static org.apache.datasketches.hll.ToByteArrayImpl.toCouponByteArray;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * @author Lee Rhodes
- */
-abstract class AbstractCoupons extends HllSketchImpl {
-
- AbstractCoupons(final int lgConfigK, final TgtHllType tgtHllType, final CurMode curMode) {
- super(lgConfigK, tgtHllType, curMode);
- }
-
- @Override
- double getCompositeEstimate() {
- return getEstimate();
- }
-
- abstract int getCouponCount();
-
- abstract int[] getCouponIntArr();
-
- /**
- * This is the estimator for the Coupon List mode and Coupon Hash Set mode.
- *
- * Note: This is an approximation to the true mapping from numCoupons to N,
- * which has a range of validity roughly from 0 to 6 million coupons. The k of the implied coupon sketch, which must not be confused with the k of the HLL
- * sketch. In this application k is always 2^26, which is the number of address bits of the
- * 32-bit coupon. The KxQ registers serve dual purposes: They are used in the HIP estimator and in
- * the "raw" HLL estimator defined in the Flajolet, et al, 2007 HLL paper. In order to do this,
- * the way the KxQ registers are computed here differ from how they are defined in the paper. The paper Fig 2 defines This is only relevant for HLL_4 sketches that have been configured for off-heap
- * using MemorySegment. For on-heap sketches or unions this will return false.
- *
- * It is rare, but possible, the the off-heap MemorySegment that has been allocated to an HLL_4
- * sketch may not be large enough. If this should happen, the sketch makes a request for more
- * space from the owner of the resource and then moves itself to this new location. This all
- * happens transparently to the user. This method provides a means for the user to
- * inquire of the sketch if it has, in fact, moved itself.
- *
- * @param seg the given MemorySegment
- * @return true if the given MemorySegment refers to the same underlying resource as this HllSketch or
- * HllUnion.
- */
- @Override
- public abstract boolean isSameResource(MemorySegment seg);
-
- /**
- * Resets to empty, but does not change the configured values of lgConfigK and tgtHllType.
- */
- public abstract void reset();
-
- /**
- * Serializes this sketch as a byte array in compact form. The compact form is smaller in size
- * than the updatable form and read-only. It can be used in HllUnion operations as follows:
- * The sketch "wrapping" operation skips actual deserialization thus is quite fast. However,
- * any attempt to update the derived HllSketch will result in a Read-only exception. Note that in some cases, based on the state of the sketch, the compact form is
- * indistiguishable from the updatable form. In these cases the updatable form is returned
- * and the compact flag bit will not be set. Note: About 2X faster performance can be obtained by first converting the String to a
- * char[] and updating the sketch with that. This bypasses the complexity of the Java UTF_8
- * encoding. This, of course, will not produce the same internal hash values as updating directly
- * with a String. So be consistent! Unioning two sketches, one fed with strings and the other
- * fed with char[] will be meaningless.
- * This method will not modify the position, mark, limit, or byte order of the buffer. Little-endian order is preferred, but not required. This method may perform better if the provided byte
- * buffer is in little-endian order. Note: this will not produce the same output hash values as the update(String)
- * method but will be a little faster as it avoids the complexity of the UTF8 encoding. In general, it is always true that for HLL_6 and HLL_8, curMin is always 0, and numAtCurMin
- * is the number of zero slots. For these two types there is no need to track curMin nor to track
- * numAtCurMin once all the slots are filled.
- *
- * @param absHllArr an instance of AbstractHllArray
- * @return pair values representing numAtCurMin and curMin
- */
- static int curMinAndNum(final AbstractHllArray absHllArr) {
- int curMin = 64;
- int numAtCurMin = 0;
- final PairIterator itr = absHllArr.iterator();
- while (itr.nextAll()) {
- final int v = itr.getValue();
- if (v > curMin) { continue; }
- if (v < curMin) {
- curMin = v;
- numAtCurMin = 1;
- } else {
- numAtCurMin++;
- }
- }
- return HllUtil.pair(numAtCurMin, curMin);
- }
-
- static Hll6Array convertToHll6(final AbstractHllArray srcAbsHllArr) {
- final int lgConfigK = srcAbsHllArr.lgConfigK;
- final Hll6Array hll6Array = new Hll6Array(lgConfigK);
- hll6Array.putOutOfOrder(srcAbsHllArr.isOutOfOrder());
- int numZeros = 1 << lgConfigK;
- final PairIterator itr = srcAbsHllArr.iterator();
- while (itr.nextAll()) {
- if (itr.getValue() != EMPTY) {
- numZeros--;
- hll6Array.couponUpdate(itr.getPair()); //couponUpdate creates KxQ registers
- }
- }
- hll6Array.putNumAtCurMin(numZeros);
- hll6Array.putHipAccum(srcAbsHllArr.getHipAccum()); //intentional overwrite
- hll6Array.putRebuildCurMinNumKxQFlag(false);
- return hll6Array;
- }
-
- static Hll8Array convertToHll8(final AbstractHllArray srcAbsHllArr) {
- final int lgConfigK = srcAbsHllArr.lgConfigK;
- final Hll8Array hll8Array = new Hll8Array(lgConfigK);
- hll8Array.putOutOfOrder(srcAbsHllArr.isOutOfOrder());
- int numZeros = 1 << lgConfigK;
- final PairIterator itr = srcAbsHllArr.iterator();
- while (itr.nextAll()) {
- if (itr.getValue() != EMPTY) {
- numZeros--;
- hll8Array.couponUpdate(itr.getPair()); //creates KxQ registers
- }
- }
- hll8Array.putNumAtCurMin(numZeros);
- hll8Array.putHipAccum(srcAbsHllArr.getHipAccum()); //intentional overwrite
- hll8Array.putRebuildCurMinNumKxQFlag(false);
- return hll8Array;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/CouponHashSet.java b/src/main/java/org/apache/datasketches/hll/CouponHashSet.java
deleted file mode 100644
index 4ba67c265..000000000
--- a/src/main/java/org/apache/datasketches/hll/CouponHashSet.java
+++ /dev/null
@@ -1,165 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.LG_INIT_SET_SIZE;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_DENOM;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_NUMER;
-import static org.apache.datasketches.hll.PreambleUtil.HASH_SET_INT_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.HASH_SET_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.LIST_INT_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.computeLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.extractCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractCurMode;
-import static org.apache.datasketches.hll.PreambleUtil.extractHashSetCount;
-import static org.apache.datasketches.hll.PreambleUtil.extractInt;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgK;
-import static org.apache.datasketches.hll.PreambleUtil.extractTgtHllType;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class CouponHashSet extends CouponList {
-
- /**
- * Constructs this sketch with the intent of loading it with data
- * @param lgConfigK the configured Lg K
- * @param tgtHllType the new target Hll type
- */
- CouponHashSet(final int lgConfigK, final TgtHllType tgtHllType) {
- super(lgConfigK, tgtHllType, CurMode.SET);
- assert lgConfigK > 7;
- }
-
- /**
- * Copy constructor
- * @param that another CouponHashSet
- */
- CouponHashSet(final CouponHashSet that) {
- super(that);
- }
-
- /**
- * Copy As constructor.
- * @param that another CouponHashSet
- * @param tgtHllType the new target Hll type
- */
- CouponHashSet(final CouponHashSet that, final TgtHllType tgtHllType) {
- super(that, tgtHllType);
- }
-
- //will also accept List, but results in a Set
- static CouponHashSet heapifySet(final MemorySegment seg) {
- final int lgConfigK = extractLgK(seg);
- final TgtHllType tgtHllType = extractTgtHllType(seg);
-
- final CurMode curMode = extractCurMode(seg);
- final int segArrStart = (curMode == CurMode.LIST) ? LIST_INT_ARR_START : HASH_SET_INT_ARR_START;
- final CouponHashSet set = new CouponHashSet(lgConfigK, tgtHllType);
- final boolean segIsCompact = extractCompactFlag(seg);
- final int couponCount = extractHashSetCount(seg);
- int lgCouponArrInts = extractLgArr(seg);
- if (lgCouponArrInts < LG_INIT_SET_SIZE) {
- lgCouponArrInts = computeLgArr(seg, couponCount, lgConfigK);
- }
- if (segIsCompact) {
- for (int i = 0; i < couponCount; i++) {
- set.couponUpdate(extractInt(seg, segArrStart + (i << 2)));
- }
- } else { //updatable
- set.couponCount = couponCount;
- set.lgCouponArrInts = lgCouponArrInts;
- final int couponArrInts = 1 << lgCouponArrInts;
- set.couponIntArr = new int[couponArrInts];
- MemorySegment.copy(seg, JAVA_INT_UNALIGNED, HASH_SET_INT_ARR_START, set.couponIntArr, 0, couponArrInts);
- }
- return set;
- }
-
- @Override
- CouponHashSet copy() {
- return new CouponHashSet(this);
- }
-
- @Override
- CouponHashSet copyAs(final TgtHllType tgtHllType) {
- return new CouponHashSet(this, tgtHllType);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- final int index = find(couponIntArr, lgCouponArrInts, coupon);
- if (index >= 0) {
- return this; //found duplicate, ignore
- }
- couponIntArr[~index] = coupon; //found empty
- couponCount++;
- if (checkGrowOrPromote()) {
- return promoteHeapListOrSetToHll(this);
- }
- return this;
- }
-
- @Override
- int getSegDataStart() {
- return HASH_SET_INT_ARR_START;
- }
-
- @Override
- int getPreInts() {
- return HASH_SET_PREINTS;
- }
-
- private boolean checkGrowOrPromote() {
- if ((RESIZE_DENOM * couponCount) > (RESIZE_NUMER * (1 << lgCouponArrInts))) {
- if (lgCouponArrInts == (lgConfigK - 3)) { //at max size
- return true; // promote to HLL
- }
- couponIntArr = growHashSet(couponIntArr, ++lgCouponArrInts);
- }
- return false;
- }
-
- private static int[] growHashSet(final int[] coupIntArr, final int tgtLgCoupArrSize) {
- final int[] tgtCouponIntArr = new int[1 << tgtLgCoupArrSize]; //create tgt
- final int len = coupIntArr.length;
- for (int i = 0; i < len; i++) { //scan input arr for non-zero values
- final int fetched = coupIntArr[i];
- if (fetched != EMPTY) {
- final int idx = find(tgtCouponIntArr, tgtLgCoupArrSize, fetched); //find empty in tgt
- if (idx < 0) { //found EMPTY
- tgtCouponIntArr[~idx] = fetched; //insert
- continue;
- }
- throw new SketchesStateException("Error: found duplicate.");
- }
- }
- return tgtCouponIntArr;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/CouponList.java b/src/main/java/org/apache/datasketches/hll/CouponList.java
deleted file mode 100644
index bd70dd3ea..000000000
--- a/src/main/java/org/apache/datasketches/hll/CouponList.java
+++ /dev/null
@@ -1,237 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.LG_INIT_LIST_SIZE;
-import static org.apache.datasketches.hll.HllUtil.LG_INIT_SET_SIZE;
-import static org.apache.datasketches.hll.PreambleUtil.LIST_INT_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.LIST_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgK;
-import static org.apache.datasketches.hll.PreambleUtil.extractListCount;
-import static org.apache.datasketches.hll.PreambleUtil.extractTgtHllType;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-class CouponList extends AbstractCoupons {
- int lgCouponArrInts;
- int couponCount;
- int[] couponIntArr;
-
- private static int checkLgConfigK(final CurMode curMode, final int lgConfigK) {
- if (curMode == CurMode.SET) { assert lgConfigK > 7; }
- return lgConfigK;
- }
-
- /**
- * New instance constructor for LIST or SET.
- * @param lgConfigK the configured Lg K
- * @param tgtHllType the configured HLL target
- * @param curMode LIST or SET
- */
- CouponList(final int lgConfigK, final TgtHllType tgtHllType, final CurMode curMode) {
- super(checkLgConfigK(curMode, lgConfigK), tgtHllType, curMode);
- if (curMode == CurMode.LIST) {
- lgCouponArrInts = LG_INIT_LIST_SIZE;
- } else { //SET
- lgCouponArrInts = LG_INIT_SET_SIZE;
- }
- couponIntArr = new int[1 << lgCouponArrInts];
- couponCount = 0;
- }
-
- /**
- * Copy Constructor
- * @param that another CouponArray
- */
- CouponList(final CouponList that) {
- super(that.lgConfigK, that.tgtHllType, that.curMode);
- lgCouponArrInts = that.lgCouponArrInts;
- couponCount = that.couponCount;
- couponIntArr = that.couponIntArr.clone();
- }
-
- /**
- * Copy As constructor.
- * @param that another CouponList
- * @param tgtHllType the new target Hll type
- */ //also used by CouponHashSet
- CouponList(final CouponList that, final TgtHllType tgtHllType) {
- super(that.lgConfigK, tgtHllType, that.curMode);
- lgCouponArrInts = that.lgCouponArrInts;
- couponCount = that.couponCount;
- couponIntArr = that.couponIntArr.clone();
- }
-
- static final CouponList heapifyList(final MemorySegment seg) {
- final int lgConfigK = extractLgK(seg);
- final TgtHllType tgtHllType = extractTgtHllType(seg);
-
- final CouponList list = new CouponList(lgConfigK, tgtHllType, CurMode.LIST);
- final int couponCount = extractListCount(seg);
- MemorySegment.copy(seg, JAVA_INT_UNALIGNED, LIST_INT_ARR_START, list.couponIntArr, 0, couponCount);
- list.couponCount = couponCount;
- return list;
- }
-
- @Override
- CouponList copy() {
- return new CouponList(this);
- }
-
- @Override
- CouponList copyAs(final TgtHllType tgtHllType) {
- return new CouponList(this, tgtHllType);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- final int len = 1 << lgCouponArrInts;
- for (int i = 0; i < len; i++) { //search for empty slot
- final int couponAtIdx = couponIntArr[i];
- if (couponAtIdx == EMPTY) {
- couponIntArr[i] = coupon; //update
- couponCount++;
- if (couponCount >= len) { //array full
- if (lgConfigK < 8) {
- return promoteHeapListOrSetToHll(this); //oooFlag = false
- }
- return promoteHeapListToSet(this); //oooFlag = true
- }
- return this;
- }
- //cell not empty
- if (couponAtIdx == coupon) {
- return this; //duplicate
- }
- //cell not empty & not a duplicate, continue
- } //end for
- throw new SketchesStateException("Array invalid: no empties & no duplicates");
- }
-
- @Override
- int getCompactSerializationBytes() {
- return getSegDataStart() + (couponCount << 2);
- }
-
- @Override
- int getCouponCount() {
- return couponCount;
- }
-
- @Override
- int[] getCouponIntArr() {
- return couponIntArr;
- }
-
- @Override
- int getLgCouponArrInts() {
- return lgCouponArrInts;
- }
-
- @Override
- int getSegDataStart() {
- return LIST_INT_ARR_START;
- }
-
- @Override
- MemorySegment getMemorySegment() {
- return null;
- }
-
- @Override
- int getPreInts() {
- return LIST_PREINTS;
- }
-
- @Override
- boolean isCompact() {
- return false;
- }
-
- @Override
- boolean hasMemorySegment() {
- return false;
- }
-
- @Override
- boolean isOffHeap() {
- return false;
- }
-
- @Override
- boolean isSameResource(final MemorySegment seg) {
- return false;
- }
-
- @Override
- PairIterator iterator() {
- return new IntArrayPairIterator(couponIntArr, lgConfigK);
- }
-
- @Override
- void mergeTo(final HllSketch that) {
- final int arrLen = couponIntArr.length;
- for (int i = 0; i < arrLen; i++) {
- final int pair = couponIntArr[i];
- if (pair == 0) { continue; }
- that.couponUpdate(pair);
- }
- }
-
- @Override
- CouponList reset() {
- return new CouponList(lgConfigK, tgtHllType, CurMode.LIST);
- }
-
- static final HllSketchImpl promoteHeapListToSet(final CouponList list) {
- final int couponCount = list.couponCount;
- final int[] arr = list.couponIntArr;
- final CouponHashSet chSet = new CouponHashSet(list.lgConfigK, list.tgtHllType);
- for (int i = 0; i < couponCount; i++) {
- chSet.couponUpdate(arr[i]);
- }
- return chSet;
- }
-
- //Promotional move of coupons to an HllSketch from either List or Set.
- //called by CouponHashSet.couponUpdate()
- //called by CouponList.couponUpdate()
- static final HllSketchImpl promoteHeapListOrSetToHll(final CouponList src) {
- final HllArray tgtHllArr = HllArray.newHeapHll(src.lgConfigK, src.tgtHllType);
- final PairIterator srcItr = src.iterator();
- tgtHllArr.putKxQ0(1 << src.lgConfigK);
- while (srcItr.nextValid()) {
- tgtHllArr.couponUpdate(srcItr.getPair());
- }
- tgtHllArr.putHipAccum(src.getEstimate());
-
- tgtHllArr.putOutOfOrder(false);
- return tgtHllArr;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/CouponMapping.java b/src/main/java/org/apache/datasketches/hll/CouponMapping.java
deleted file mode 100644
index 25d927679..000000000
--- a/src/main/java/org/apache/datasketches/hll/CouponMapping.java
+++ /dev/null
@@ -1,58 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- *
- */
-final class CouponMapping {
-
- //Computed for Coupon lgK = 26 ONLY. Designed for the cubic interpolator function.
- static final double[] xArr = new double[] {
- 0.0, 1.0, 20.0, 400.0,
- 8000.0, 160000.0, 300000.0, 600000.0,
- 900000.0, 1200000.0, 1500000.0, 1800000.0,
- 2100000.0, 2400000.0, 2700000.0, 3000000.0,
- 3300000.0, 3600000.0, 3900000.0, 4200000.0,
- 4500000.0, 4800000.0, 5100000.0, 5400000.0,
- 5700000.0, 6000000.0, 6300000.0, 6600000.0,
- 6900000.0, 7200000.0, 7500000.0, 7800000.0,
- 8100000.0, 8400000.0, 8700000.0, 9000000.0,
- 9300000.0, 9600000.0, 9900000.0, 10200000.0
- };
-
- // CHECKSTYLE:OFF LineLength
- //Computed for Coupon lgK = 26 ONLY. Designed for the cubic interpolator function.
- static final double[] yArr = new double[] {
- 0.0000000000000000, 1.0000000000000000, 20.0000009437402611, 400.0003963713384110,
- 8000.1589294602090376, 160063.6067763759638183, 300223.7071597663452849, 600895.5933856170158833,
- 902016.8065120954997838, 1203588.4983199508860707, 1505611.8245524743106216, 1808087.9449319066479802,
- 2111018.0231759352609515, 2414403.2270142501220107, 2718244.7282051891088486, 3022543.7025524540804327,
- 3327301.3299219091422856, 3632518.7942584538832307, 3938197.2836029687896371, 4244337.9901093561202288,
- 4550942.1100616492331028, 4858010.8438911894336343, 5165545.3961938973516226, 5473546.9757476449012756,
- 5782016.7955296505242586, 6090956.0727340159937739, 6400366.0287892958149314, 6710247.8893762007355690,
- 7020602.8844453142955899, 7331432.2482349723577499, 7642737.2192891482263803, 7954519.0404754765331745,
- 8266778.9590033423155546, 8579518.2264420464634895, 8892738.0987390466034412, 9206439.8362383283674717,
- 9520624.7036988288164139, 9835293.9703129194676876, 10150448.9097250290215015, 10466090.8000503256917000
- };
- // CHECKSTYLE:ON LineLength
-}
diff --git a/src/main/java/org/apache/datasketches/hll/CubicInterpolation.java b/src/main/java/org/apache/datasketches/hll/CubicInterpolation.java
deleted file mode 100644
index 4bca1a02e..000000000
--- a/src/main/java/org/apache/datasketches/hll/CubicInterpolation.java
+++ /dev/null
@@ -1,165 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class CubicInterpolation {
-
- /**
- * Cubic interpolation using interpolation X and Y tables.
- *
- * @param xArr xArr
- * @param yArr yArr
- * @param x x
- * @return cubic interpolation
- */
- //Used by AbstractCoupons
- //In C: again-two-registers cubic_interpolate_using_table L1377
- static double usingXAndYTables(final double[] xArr, final double[] yArr,
- final double x) {
- assert (xArr.length >= 4) && (xArr.length == yArr.length);
- if ((x < xArr[0]) || (x > xArr[xArr.length - 1])) {
- throw new SketchesArgumentException("X value out of range: " + x);
- }
- if (x == xArr[xArr.length - 1]) {
- return yArr[yArr.length - 1]; // corner case
- }
- final int offset = findStraddle(xArr, x); //uses recursion
- assert (offset >= 0) && (offset <= (xArr.length - 2));
- if (offset == 0) {
- return interpolateUsingXAndYTables(xArr, yArr, offset, x); // corner case
- }
- if (offset == (xArr.length - 2)) {
- return interpolateUsingXAndYTables(xArr, yArr, offset - 2, x); // corner case
- }
- return interpolateUsingXAndYTables(xArr, yArr, offset - 1, x);
- }
-
- // In C: again-two-registers cubic_interpolate_aux L1368
- private static double interpolateUsingXAndYTables(final double[] xArr, final double[] yArr,
- final int offset, final double x) {
- return cubicInterpolate(xArr[offset], yArr[offset], xArr[offset + 1], yArr[offset + 1],
- xArr[offset + 2], yArr[offset + 2], xArr[offset + 3], yArr[offset + 3], x);
- }
-
- //Interpolate using X table and Y stride
-
- /**
- * Cubic interpolation using interpolation X table and Y stride.
- *
- * @param xArr The x array
- * @param yStride The y stride
- * @param x The value x
- * @return cubic interpolation
- */
- //In C: again-two-registers cubic_interpolate_with_x_arr_and_y_stride L1411
- // Used by HllEstimators
- static double usingXArrAndYStride(
- final double[] xArr, final double yStride, final double x) {
- final int xArrLen = xArr.length;
- final int xArrLenM1 = xArrLen - 1;
-
- final int offset;
- assert ((xArrLen >= 4) && (x >= xArr[0]) && (x <= xArr[xArrLenM1]));
-
- if (x == xArr[xArrLenM1]) { /* corner case */
- return (yStride * (xArrLenM1));
- }
-
- offset = findStraddle(xArr, x); //uses recursion
- final int xArrLenM2 = xArrLen - 2;
- assert ((offset >= 0) && (offset <= (xArrLenM2)));
-
- if (offset == 0) { /* corner case */
- return (interpolateUsingXArrAndYStride(xArr, yStride, (offset - 0), x));
- }
- else if (offset == xArrLenM2) { /* corner case */
- return (interpolateUsingXArrAndYStride(xArr, yStride, (offset - 2), x));
- }
- /* main case */
- return (interpolateUsingXArrAndYStride(xArr, yStride, (offset - 1), x));
- }
-
- //In C: again-two-registers cubic_interpolate_with_x_arr_and_y_stride_aux L1402
- private static double interpolateUsingXArrAndYStride(final double[] xArr, final double yStride,
- final int offset, final double x) {
- return cubicInterpolate(
- xArr[offset + 0], yStride * (offset + 0),
- xArr[offset + 1], yStride * (offset + 1),
- xArr[offset + 2], yStride * (offset + 2),
- xArr[offset + 3], yStride * (offset + 3),
- x);
- }
-
- //Cubic Interpolation used by both methods
-
- // Interpolate using the cubic curve that passes through the four given points, using the
- // Lagrange interpolation formula.
- // In C: again-two-registers cubic_interpolate_aux_aux L1346
- private static double cubicInterpolate(final double x0, final double y0, final double x1,
- final double y1, final double x2, final double y2, final double x3, final double y3,
- final double x) {
- final double l0Numer = (x - x1) * (x - x2) * (x - x3);
- final double l1Numer = (x - x0) * (x - x2) * (x - x3);
- final double l2Numer = (x - x0) * (x - x1) * (x - x3);
- final double l3Numer = (x - x0) * (x - x1) * (x - x2);
-
- final double l0Denom = (x0 - x1) * (x0 - x2) * (x0 - x3);
- final double l1Denom = (x1 - x0) * (x1 - x2) * (x1 - x3);
- final double l2Denom = (x2 - x0) * (x2 - x1) * (x2 - x3);
- final double l3Denom = (x3 - x0) * (x3 - x1) * (x3 - x2);
-
- final double term0 = (y0 * l0Numer) / l0Denom;
- final double term1 = (y1 * l1Numer) / l1Denom;
- final double term2 = (y2 * l2Numer) / l2Denom;
- final double term3 = (y3 * l3Numer) / l3Denom;
-
- return term0 + term1 + term2 + term3;
- }
-
- //In C: again-two-registers.c find_straddle L1335
- private static int findStraddle(final double[] xArr, final double x) {
- assert ((xArr.length >= 2) && (x >= xArr[0]) && (x <= xArr[xArr.length - 1]));
- return (recursiveFindStraddle(xArr, 0, xArr.length - 1, x));
- }
-
- //In C: again-two-registers.c find_straddle_aux L1322
- private static int recursiveFindStraddle(final double[] xArr, final int left, final int right,
- final double x) {
- final int middle;
- assert (left < right);
- assert ((xArr[left] <= x) && (x < xArr[right])); /* the invariant */
- if ((left + 1) == right) {
- return (left);
- }
- middle = left + ((right - left) / 2);
- if (xArr[middle] <= x) {
- return (recursiveFindStraddle(xArr, middle, right, x));
- } else {
- return (recursiveFindStraddle(xArr, left, middle, x));
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/CurMode.java b/src/main/java/org/apache/datasketches/hll/CurMode.java
deleted file mode 100644
index a31266ce4..000000000
--- a/src/main/java/org/apache/datasketches/hll/CurMode.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-/**
- * Represents the three fundamental modes of the HLL Sketch.
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-enum CurMode { LIST, SET, HLL; //do not change the order.
-
- public static final CurMode values[] = values();
-
- /**
- * Returns the CurMode given its ordinal
- * @param ordinal the order of appearance in the enum definition.
- * @return the CurMode given its ordinal
- */
- public static CurMode fromOrdinal(final int ordinal) {
- return values[ordinal];
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/DirectAuxHashMap.java b/src/main/java/org/apache/datasketches/hll/DirectAuxHashMap.java
deleted file mode 100644
index 0f18aa24a..000000000
--- a/src/main/java/org/apache/datasketches/hll/DirectAuxHashMap.java
+++ /dev/null
@@ -1,218 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.common.Util.clear;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_DENOM;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_NUMER;
-import static org.apache.datasketches.hll.HllUtil.noWriteAccess;
-import static org.apache.datasketches.hll.PreambleUtil.extractAuxCount;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.insertAuxCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgArr;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- */
-final class DirectAuxHashMap implements AuxHashMap {
- private final DirectHllArray host; //hosts the MemorySegment and read-only MemorySegment
- private final boolean readOnly;
-
- DirectAuxHashMap(final DirectHllArray host, final boolean initialize) {
- this.host = host;
- readOnly = (host.wseg == null);
- final int initLgArrInts = HllUtil.LG_AUX_ARR_INTS[host.lgConfigK];
-
- if (initialize) { //must be writable
- if (readOnly) { noWriteAccess(); }
- insertLgArr(host.wseg, initLgArrInts);
- clear(host.wseg, host.auxStart, 4 << initLgArrInts);
- } else if (extractLgArr(host.seg) < initLgArrInts) {
- if (readOnly) {
- throw new SketchesArgumentException(
- "Possible MemorySegment image corruption, incorrect LgArr field in preamble.");
- }
- //insert the correct LgArr value
- final int lgArr =
- PreambleUtil.computeLgArr(host.wseg, host.auxHashMap.getAuxCount(), host.lgConfigK);
- insertLgArr(host.wseg, lgArr);
- }
- }
-
- @Override
- public DirectAuxHashMap copy() { //a no-op
- return null;
- }
-
- @Override
- public int getAuxCount() {
- return extractAuxCount(host.seg);
- }
-
- @Override
- public int[] getAuxIntArr() {
- return null;
- }
-
- @Override
- public int getCompactSizeBytes() {
- return getAuxCount() << 2;
- }
-
- @Override
- public PairIterator getIterator() {
- return new IntMemorySegmentPairIterator(
- host.seg, host.auxStart, 1 << getLgAuxArrInts(), host.lgConfigK);
- }
-
- @Override
- public int getLgAuxArrInts() {
- return extractLgArr(host.seg);
- }
-
- @Override
- public int getUpdatableSizeBytes() {
- return 4 << getLgAuxArrInts();
- }
-
- @Override
- public boolean hasMemorySegment() {
- return host.hasMemorySegment();
- }
-
- @Override
- public boolean isOffHeap() {
- return host.isOffHeap();
- }
-
- @Override
- public boolean isSameResource(final MemorySegment seg) {
- return host.isSameResource(seg);
- }
-
- @Override
- public void mustAdd(final int slotNo, final int value) {
- if (readOnly) { noWriteAccess(); }
- final int index = find(host, slotNo);
- final int pair = HllUtil.pair(slotNo, value);
- if (index >= 0) {
- final String pairStr = HllUtil.pairString(pair);
- throw new SketchesStateException("Found a slotNo that should not be there: " + pairStr);
- }
- //Found empty entry
- host.wseg.set(JAVA_INT_UNALIGNED, host.auxStart + (~index << 2), pair);
- int auxCount = extractAuxCount(host.seg);
- insertAuxCount(host.wseg, ++auxCount);
- final int lgAuxArrInts = extractLgArr(host.seg);
- if ((RESIZE_DENOM * auxCount) > (RESIZE_NUMER * (1 << lgAuxArrInts))) {
- grow(host, lgAuxArrInts);
- }
- }
-
- @Override
- public int mustFindValueFor(final int slotNo) {
- final int index = find(host, slotNo);
- if (index >= 0) {
- final int pair = host.seg.get(JAVA_INT_UNALIGNED, host.auxStart + (index << 2));
- return HllUtil.getPairValue(pair);
- }
- throw new SketchesStateException("SlotNo not found: " + slotNo);
- }
-
- @Override
- public void mustReplace(final int slotNo, final int value) {
- if (readOnly) { noWriteAccess(); }
- final int index = find(host, slotNo);
- if (index >= 0) {
- host.wseg.set(JAVA_INT_UNALIGNED, host.auxStart + (index << 2), HllUtil.pair(slotNo, value));
- return;
- }
- final String pairStr = HllUtil.pairString(HllUtil.pair(slotNo, value));
- throw new SketchesStateException("Pair not found: " + pairStr);
- }
-
- //Searches the Aux arr hash table (embedded in MemorySegment) for an empty or a matching slotNo
- // depending on the context.
- //If entire entry is empty, returns one's complement of index = found empty.
- //If entry contains given slotNo, returns its index = found slotNo.
- //Continues searching.
- //If the probe comes back to original index, throws an exception.
- private static int find(final DirectHllArray host, final int slotNo) {
- final int lgAuxArrInts = extractLgArr(host.seg);
- assert lgAuxArrInts < host.lgConfigK : lgAuxArrInts;
- final int auxInts = 1 << lgAuxArrInts;
- final int auxArrMask = auxInts - 1;
- final int configKmask = (1 << host.lgConfigK) - 1;
- int probe = slotNo & auxArrMask;
- final int loopIndex = probe;
- do {
- final int arrVal = host.seg.get(JAVA_INT_UNALIGNED, host.auxStart + (probe << 2));
- if (arrVal == EMPTY) {
- return ~probe; //empty
- }
- else if (slotNo == (arrVal & configKmask)) { //found given slotNo
- return probe; //return aux array index
- }
- final int stride = (slotNo >>> lgAuxArrInts) | 1;
- probe = (probe + stride) & auxArrMask;
- } while (probe != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- private static void grow(final DirectHllArray host, final int oldLgAuxArrInts) {
- if (host.wseg == null) { noWriteAccess(); }
- final int oldAuxArrInts = 1 << oldLgAuxArrInts;
- final int[] oldIntArray = new int[oldAuxArrInts]; //buffer old aux data
- MemorySegment.copy(host.wseg, JAVA_INT_UNALIGNED, host.auxStart, oldIntArray, 0, oldAuxArrInts);
-
- insertLgArr(host.wseg, oldLgAuxArrInts + 1); //update LgArr field
-
- final long newAuxBytes = oldAuxArrInts << 3;
- final long requestBytes = host.auxStart + newAuxBytes;
- final long oldCapBytes = host.wseg.byteSize();
-
- if (requestBytes > oldCapBytes) {
- final MemorySegment newWseg = MemorySegment.ofArray(new byte[(int)requestBytes]);
- MemorySegment.copy(host.wseg, 0, newWseg, 0, host.auxStart);
-
- clear(newWseg, host.auxStart, newAuxBytes); //clear space for new aux data
- host.updateMemorySegment(newWseg);
- }
- //rehash into larger aux array
- final int configKmask = (1 << host.lgConfigK) - 1;
-
- for (int i = 0; i < oldAuxArrInts; i++) {
- final int fetched = oldIntArray[i];
- if (fetched != EMPTY) {
- //find empty in new array
- final int index = find(host, fetched & configKmask);
- host.wseg.set(JAVA_INT_UNALIGNED, host.auxStart + (~index << 2), fetched);
- }
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/DirectCouponHashSet.java b/src/main/java/org/apache/datasketches/hll/DirectCouponHashSet.java
deleted file mode 100644
index e00d4a888..000000000
--- a/src/main/java/org/apache/datasketches/hll/DirectCouponHashSet.java
+++ /dev/null
@@ -1,162 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.common.Util.clear;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.KEY_MASK_26;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_DENOM;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_NUMER;
-import static org.apache.datasketches.hll.HllUtil.noWriteAccess;
-import static org.apache.datasketches.hll.PreambleUtil.HASH_SET_INT_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.HASH_SET_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.LG_K_BYTE;
-import static org.apache.datasketches.hll.PreambleUtil.extractHashSetCount;
-import static org.apache.datasketches.hll.PreambleUtil.extractInt;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.insertHashSetCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertInt;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgArr;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesException;
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- */
-final class DirectCouponHashSet extends DirectCouponList {
-
- //Constructs this sketch with data.
- DirectCouponHashSet(final int lgConfigK, final TgtHllType tgtHllType, final MemorySegment wseg) {
- super(lgConfigK, tgtHllType, CurMode.SET, wseg);
- assert wseg.get(JAVA_BYTE, LG_K_BYTE) > 7;
- }
-
- //Constructs this sketch with read-only data, may be compact.
- DirectCouponHashSet(final int lgConfigK, final TgtHllType tgtHllType, final MemorySegment seg, final boolean readOnly) {
- super(lgConfigK, tgtHllType, CurMode.SET, seg, readOnly);
- assert seg.get(JAVA_BYTE, LG_K_BYTE) > 7;
- }
-
- @Override //returns on-heap Set
- CouponHashSet copy() {
- return CouponHashSet.heapifySet(seg);
- }
-
- @Override //returns on-heap Set
- CouponHashSet copyAs(final TgtHllType tgtHllType) {
- final CouponHashSet clist = CouponHashSet.heapifySet(seg);
- return new CouponHashSet(clist, tgtHllType);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- if (wseg == null) { noWriteAccess(); }
- //avoid array copy
- final int index = find(seg, getLgCouponArrInts(), coupon);
- if (index >= 0) {
- return this; //found duplicate, ignore
- }
- insertInt(wseg, HASH_SET_INT_ARR_START + (~index << 2), coupon);
- insertHashSetCount(wseg, getCouponCount() + 1);
- final boolean promote = checkGrowOrPromote();
- if (!promote) { return this; }
- return promoteListOrSetToHll(this);
- }
-
- @Override
- int getCouponCount() {
- return extractHashSetCount(seg);
- }
-
- @Override
- int getSegDataStart() {
- return HASH_SET_INT_ARR_START;
- }
-
- @Override
- int getPreInts() {
- return HASH_SET_PREINTS;
- }
-
- private boolean checkGrowOrPromote() {
- int lgCouponArrInts = getLgCouponArrInts();
- if ((RESIZE_DENOM * getCouponCount()) > (RESIZE_NUMER * (1 << lgCouponArrInts))) {
- if (lgCouponArrInts == (getLgConfigK() - 3)) {
- return true; // promote
- }
- insertLgArr(wseg, ++lgCouponArrInts);
- growHashSet(wseg, lgCouponArrInts);
- }
- return false;
- }
-
- //This could fail if the user has undersized the given MemorySegment
- // and not used the public methods for sizing the MemorySegment. See exception.
- private static void growHashSet(final MemorySegment wseg, final int tgtLgCouponArrSize) {
- final int tgtArrSize = 1 << tgtLgCouponArrSize;
- final int[] tgtCouponIntArr = new int[tgtArrSize];
- final int oldLen = 1 << extractLgArr(wseg);
- for (int i = 0; i < oldLen; i++) {
- final int fetched = extractInt(wseg, HASH_SET_INT_ARR_START + (i << 2));
- if (fetched != EMPTY) {
- final int idx = find(tgtCouponIntArr, tgtLgCouponArrSize, fetched);
- if (idx < 0) { //found EMPTY
- tgtCouponIntArr[~idx] = fetched; //insert
- continue;
- }
- throw new SketchesStateException("Error: found duplicate.");
- }
- }
- clear(wseg, HASH_SET_INT_ARR_START, tgtArrSize << 2);
- try { MemorySegment.copy(tgtCouponIntArr, 0, wseg, JAVA_INT_UNALIGNED, HASH_SET_INT_ARR_START, tgtArrSize); }
- catch (final IndexOutOfBoundsException e) {
- throw new SketchesException(
- "The MemorySegment is undersized. Use the public methods for properly sizing MemorySegment.", e);
- }
- }
-
- //Searches the Coupon hash table (embedded in MemorySegment) for an empty slot
- // or a duplicate depending on the context.
- //If entire entry is empty, returns one's complement of index = found empty.
- //If entry equals given coupon, returns its index = found duplicate coupon
- //Continues searching
- //If the probe comes back to original index, throws an exception.
- private static int find(final MemorySegment seg, final int lgArr,
- final int coupon) {
- final int arrMask = (1 << lgArr) - 1;
- int probe = coupon & arrMask;
- final int loopIndex = probe;
- do {
- final int couponAtIndex = extractInt(seg, HASH_SET_INT_ARR_START + (probe << 2));
- if (couponAtIndex == EMPTY) { return ~probe; } //empty
- else if (coupon == couponAtIndex) { return probe; } //duplicate
- final int stride = ((coupon & KEY_MASK_26) >>> lgArr) | 1;
- probe = (probe + stride) & arrMask;
- } while (probe != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/DirectCouponList.java b/src/main/java/org/apache/datasketches/hll/DirectCouponList.java
deleted file mode 100644
index 90a19a6f7..000000000
--- a/src/main/java/org/apache/datasketches/hll/DirectCouponList.java
+++ /dev/null
@@ -1,321 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.common.Util.clear;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.LG_INIT_LIST_SIZE;
-import static org.apache.datasketches.hll.HllUtil.LG_INIT_SET_SIZE;
-import static org.apache.datasketches.hll.HllUtil.noWriteAccess;
-import static org.apache.datasketches.hll.PreambleUtil.EMPTY_FLAG_MASK;
-import static org.apache.datasketches.hll.PreambleUtil.HASH_SET_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.LIST_INT_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.LIST_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.computeLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.extractCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractInt;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.extractListCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.insertCurMode;
-import static org.apache.datasketches.hll.PreambleUtil.insertEmptyFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertFamilyId;
-import static org.apache.datasketches.hll.PreambleUtil.insertFlags;
-import static org.apache.datasketches.hll.PreambleUtil.insertInt;
-import static org.apache.datasketches.hll.PreambleUtil.insertKxQ0;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgK;
-import static org.apache.datasketches.hll.PreambleUtil.insertListCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertModes;
-import static org.apache.datasketches.hll.PreambleUtil.insertNumAtCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.insertPreInts;
-import static org.apache.datasketches.hll.PreambleUtil.insertSerVer;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.MemorySegmentStatus;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- */
-class DirectCouponList extends AbstractCoupons {
- MemorySegment wseg;
- MemorySegment seg;
- final boolean compact;
-
- private static int checkSegCompactFlag(final MemorySegment wseg, final int lgConfigK) {
- assert !extractCompactFlag(wseg) : "Compact Flag must not be set.";
- return lgConfigK;
- }
-
- //called from newInstance, writableWrap and DirectCouponHashSet, must not be compact
- DirectCouponList(final int lgConfigK, final TgtHllType tgtHllType, final CurMode curMode, final MemorySegment wseg) {
- super(checkSegCompactFlag(wseg, lgConfigK), tgtHllType, curMode);
- this.wseg = wseg;
- seg = wseg;
- compact = extractCompactFlag(wseg);
- }
-
- //called from HllSketch.wrap and from DirectCouponHashSet constructor, may or may not be compact
- DirectCouponList(final int lgConfigK, final TgtHllType tgtHllType, final CurMode curMode, final MemorySegment seg,
- final boolean readOnly) {
- super(lgConfigK, tgtHllType, curMode);
- wseg = null;
- this.seg = readOnly ? seg.asReadOnly() : seg;
- compact = extractCompactFlag(seg);
- }
-
- /**
- * Standard factory for new DirectCouponList.
- * This initializes the given MemorySegment.
- * @param lgConfigK the configured Lg K
- * @param tgtHllType the configured HLL target
- * @param dstSeg the destination MemorySegment for the sketch.
- * @return a new DirectCouponList
- */
- static DirectCouponList newInstance(final int lgConfigK, final TgtHllType tgtHllType,
- final MemorySegment dstSeg) {
- insertPreInts(dstSeg, LIST_PREINTS);
- insertSerVer(dstSeg);
- insertFamilyId(dstSeg);
- insertLgK(dstSeg, lgConfigK);
- insertLgArr(dstSeg, LG_INIT_LIST_SIZE);
- insertFlags(dstSeg, EMPTY_FLAG_MASK); //empty and not compact
- insertListCount(dstSeg, 0);
- insertModes(dstSeg, tgtHllType, CurMode.LIST);
- return new DirectCouponList(lgConfigK, tgtHllType, CurMode.LIST, dstSeg);
- }
-
- @Override //returns on-heap List
- CouponList copy() {
- return CouponList.heapifyList(seg);
- }
-
- @Override //returns on-heap List
- CouponList copyAs(final TgtHllType tgtHllType) {
- final CouponList clist = CouponList.heapifyList(seg);
- return new CouponList(clist, tgtHllType);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- if (wseg == null) { noWriteAccess(); }
- final int len = 1 << getLgCouponArrInts();
- for (int i = 0; i < len; i++) { //search for empty slot and duplicates
- final int couponAtIdx = extractInt(seg, LIST_INT_ARR_START + (i << 2));
- if (couponAtIdx == EMPTY) {
- insertInt(wseg, LIST_INT_ARR_START + (i << 2), coupon);
- int couponCount = extractListCount(seg);
- insertListCount(wseg, ++couponCount);
- insertEmptyFlag(wseg, false); //only first time
- if (couponCount >= len) { //array full
- if (lgConfigK < 8) {
- return promoteListOrSetToHll(this);//oooFlag = false
- }
- return promoteListToSet(this); //oooFlag = true
- }
- return this;
- }
- //cell not empty
- if (couponAtIdx == coupon) { return this; } //return if duplicate
- //cell not empty & not a duplicate, continue
- } //end for
- throw new SketchesStateException("Invalid State: no empties & no duplicates");
- }
-
- @Override
- int getCompactSerializationBytes() {
- return getSegDataStart() + (getCouponCount() << 2);
- }
-
- @Override //Overridden by DirectCouponHashSet
- int getCouponCount() {
- return extractListCount(seg);
- }
-
- @Override
- int[] getCouponIntArr() { //here only to satisfy the abstract, should not be used
- return null;
- }
-
- @Override
- int getLgCouponArrInts() {
- final int lgArr = extractLgArr(seg);
- if (lgArr >= LG_INIT_LIST_SIZE) { return lgArr; }
- //early versions of compact images did not use this lgArr field
- final int coupons = getCouponCount();
- return computeLgArr(seg, coupons, lgConfigK);
- }
-
- @Override
- int getSegDataStart() {
- return LIST_INT_ARR_START;
- }
-
- @Override
- MemorySegment getMemorySegment() {
- return seg;
- }
-
- @Override
- int getPreInts() {
- return LIST_PREINTS;
- }
-
- @Override
- boolean isCompact() {
- return compact;
- }
-
- @Override
- boolean hasMemorySegment() {
- return seg.scope().isAlive();
- }
-
- @Override
- boolean isOffHeap() {
- return seg.isNative();
- }
-
- @Override
- boolean isSameResource(final MemorySegment seg) {
- return MemorySegmentStatus.isSameResource(this.seg, seg);
- }
-
- @Override
- PairIterator iterator() {
- final long dataStart = getSegDataStart();
- final int lenInts = (compact) ? getCouponCount() : 1 << getLgCouponArrInts();
- return new IntMemorySegmentPairIterator(seg, dataStart, lenInts, lgConfigK);
- }
-
- @Override
- void mergeTo(final HllSketch that) {
- final int lenInts = (compact) ? getCouponCount() : 1 << getLgCouponArrInts();
- final int dataStart = getSegDataStart();
- for (int i = 0; i < lenInts; i++) {
- final int pair = seg.get(JAVA_INT_UNALIGNED, dataStart + (i << 2));
- if (pair == 0) { continue; }
- that.couponUpdate(pair);
- }
- }
-
- @Override
- DirectCouponList reset() {
- if (wseg == null) {
- throw new SketchesArgumentException("Cannot reset a read-only sketch");
- }
- insertEmptyFlag(wseg, true);
- final int bytes = HllSketch.getMaxUpdatableSerializationBytes(lgConfigK, tgtHllType);
- clear(wseg, 0, bytes);
-
- return DirectCouponList.newInstance(lgConfigK, tgtHllType, wseg);
- }
-
- //Called by DirectCouponList.couponUpdate()
- static final DirectCouponHashSet promoteListToSet(final DirectCouponList src) {
- final MemorySegment wseg = src.wseg;
-
- //get the data from the current MemorySegment
- HllUtil.checkPreamble(wseg); //sanity check
- final int lgConfigK = src.lgConfigK;
- final TgtHllType tgtHllType = src.tgtHllType;
- final int srcOffset = LIST_INT_ARR_START;
- final int couponArrInts = 1 << src.getLgCouponArrInts();
- final int[] couponArr = new int[couponArrInts]; //buffer
- MemorySegment.copy(wseg, JAVA_INT_UNALIGNED, srcOffset, couponArr, 0, couponArrInts);
-
- //rewrite the MemorySegment image as a SET:
- insertPreInts(wseg, HASH_SET_PREINTS);
- //SerVer, FamID, LgK should be OK
- insertLgArr(wseg, LG_INIT_SET_SIZE);
- insertCurMin(wseg, 0); //was list count
- insertCurMode(wseg, CurMode.SET);
- //tgtHllType should already be ok
- final int maxBytes = HllSketch.getMaxUpdatableSerializationBytes(lgConfigK, tgtHllType);
- clear(wseg, LIST_INT_ARR_START, maxBytes - LIST_INT_ARR_START); //clear all past first 8
-
- //create the tgt
- final DirectCouponHashSet dchSet = new DirectCouponHashSet(src.lgConfigK, src.tgtHllType, src.wseg);
-
- //now reload the coupon data into the set
- for (int i = 0; i < couponArrInts; i++) {
- final int coupon = couponArr[i];
- if (coupon != EMPTY) {
- dchSet.couponUpdate(coupon);
- }
- }
- return dchSet;
- }
-
- static final DirectHllArray promoteListOrSetToHll(final DirectCouponList src) {
- final MemorySegment wseg = src.wseg;
-
- //get the data from the current list or set MemorySegment
- HllUtil.checkPreamble(wseg); //sanity check
- final int lgConfigK = src.lgConfigK;
- final TgtHllType tgtHllType = src.tgtHllType;
- final int srcSegDataStart = src.getSegDataStart();
- final double est = src.getEstimate();
- final int couponArrInts = 1 << src.getLgCouponArrInts();
- final int[] couponArr = new int[couponArrInts]; //buffer
- MemorySegment.copy(wseg, JAVA_INT_UNALIGNED, srcSegDataStart, couponArr, 0, couponArrInts);
-
- //rewrite the MemorySegment image as an HLL
- insertPreInts(wseg, HLL_PREINTS);
- //SerVer, FamID, LgK should be OK
- insertLgArr(wseg, 0); //no Aux possible yet
- insertFlags(wseg, 0); //clear all flags
- insertCurMin(wseg, 0);
- insertCurMode(wseg, CurMode.HLL);
- //tgtHllType should already be set
- //we update HipAccum at the end
- //clear KxQ0, KxQ1, NumAtCurMin, AuxCount, hllArray, auxArr
- final int maxBytes = HllSketch.getMaxUpdatableSerializationBytes(lgConfigK, tgtHllType);
- clear(wseg, LIST_INT_ARR_START, maxBytes - LIST_INT_ARR_START); //clear all past first 8
- insertNumAtCurMin(wseg, 1 << lgConfigK); //set numAtCurMin
- insertKxQ0(wseg, 1 << lgConfigK);
-
- //choose the tgt
- final DirectHllArray dirHllArr;
- if (tgtHllType == TgtHllType.HLL_4) {
- dirHllArr = new DirectHll4Array(lgConfigK, wseg);
- } else if (tgtHllType == TgtHllType.HLL_6) {
- dirHllArr = new DirectHll6Array(lgConfigK, wseg);
- } else { //Hll_8
- dirHllArr = new DirectHll8Array(lgConfigK, wseg);
- }
-
- //now load the coupon data into HLL
- for (int i = 0; i < couponArrInts; i++) {
- final int coupon = couponArr[i];
- if (coupon != EMPTY) {
- dirHllArr.couponUpdate(coupon);
- }
- }
- dirHllArr.putHipAccum(est);
- return dirHllArr;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/DirectHll4Array.java b/src/main/java/org/apache/datasketches/hll/DirectHll4Array.java
deleted file mode 100644
index 03eefdc01..000000000
--- a/src/main/java/org/apache/datasketches/hll/DirectHll4Array.java
+++ /dev/null
@@ -1,209 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.hll.HllUtil.AUX_TOKEN;
-import static org.apache.datasketches.hll.HllUtil.KEY_BITS_26;
-import static org.apache.datasketches.hll.HllUtil.LG_AUX_ARR_INTS;
-import static org.apache.datasketches.hll.HllUtil.hiNibbleMask;
-import static org.apache.datasketches.hll.HllUtil.loNibbleMask;
-import static org.apache.datasketches.hll.HllUtil.noWriteAccess;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.extractAuxCount;
-import static org.apache.datasketches.hll.PreambleUtil.extractCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertAuxCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertInt;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgArr;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- */
-final class DirectHll4Array extends DirectHllArray {
-
- //Called by HllSketch.writableWrap(), DirectCouponList.promoteListOrSetToHll
- DirectHll4Array(final int lgConfigK, final MemorySegment wseg) {
- super(lgConfigK, TgtHllType.HLL_4, wseg);
- if (extractAuxCount(seg) > 0) {
- putAuxHashMap(new DirectAuxHashMap(this, false), false);
- }
- }
-
- //Called by HllSketch.wrap(MemorySegment)
- DirectHll4Array(final int lgConfigK, final MemorySegment seg, final boolean readOnly) {
- super(lgConfigK, TgtHllType.HLL_4, seg, readOnly);
- final int auxCount = extractAuxCount(seg);
- if (auxCount > 0) {
- final boolean compact = extractCompactFlag(seg);
- final AuxHashMap auxHashMap;
- if (compact) {
- auxHashMap = HeapAuxHashMap.heapify(seg, auxStart, lgConfigK, auxCount, compact);
- } else {
- auxHashMap = new DirectAuxHashMap(this, false); //not compact
- }
- putAuxHashMap(auxHashMap, compact);
- }
- }
-
- @Override
- HllSketchImpl copy() {
- return Hll4Array.heapify(seg);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- if (wseg == null) { noWriteAccess(); }
- final int newValue = coupon >>> KEY_BITS_26;
- final int configKmask = (1 << getLgConfigK()) - 1;
- final int slotNo = coupon & configKmask;
- updateSlotWithKxQ(slotNo, newValue);
- return this;
- }
-
- @Override
- int getHllByteArrBytes() {
- return hll4ArrBytes(lgConfigK);
- }
-
- @Override
- int getNibble(final int slotNo) {
- final long offset = HLL_BYTE_ARR_START + (slotNo >>> 1);
- int theByte = seg.get(JAVA_BYTE, offset);
- if ((slotNo & 1) > 0) { //odd?
- theByte >>>= 4;
- }
- return theByte & loNibbleMask;
- }
-
- @Override
- int getSlotValue(final int slotNo) {
- final int nib = getNibble(slotNo);
- if (nib == AUX_TOKEN) {
- final AuxHashMap auxHashMap = getAuxHashMap();
- return auxHashMap.mustFindValueFor(slotNo); //auxHashMap cannot be null here
- } else {
- return nib + getCurMin();
- }
- }
-
- @Override
- int getUpdatableSerializationBytes() {
- final AuxHashMap auxHashMap = getAuxHashMap();
- final int auxBytes;
- if (auxHashMap == null) {
- auxBytes = 4 << LG_AUX_ARR_INTS[lgConfigK];
- } else {
- auxBytes = 4 << auxHashMap.getLgAuxArrInts();
- }
- return HLL_BYTE_ARR_START + getHllByteArrBytes() + auxBytes;
- }
-
- @Override
- PairIterator iterator() {
- return new DirectHll4Iterator(1 << lgConfigK);
- }
-
- @Override
- void putNibble(final int slotNo, final int nibValue) {
- final long offset = HLL_BYTE_ARR_START + (slotNo >>> 1);
- final int oldValue = seg.get(JAVA_BYTE, offset);
- final byte value = ((slotNo & 1) == 0) //even?
- ? (byte) ((oldValue & hiNibbleMask) | (nibValue & loNibbleMask)) //set low nibble
- : (byte) ((oldValue & loNibbleMask) | ((nibValue << 4) & hiNibbleMask)); //set high nibble
- wseg.set(JAVA_BYTE, offset, value);
- }
-
- @Override
- //Would be used by HllUnion, but not used because the gadget is always HLL8 type
- void updateSlotNoKxQ(final int slotNo, final int newValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Used by this couponUpdate()
- //updates HipAccum, CurMin, NumAtCurMin, KxQs and checks newValue > oldValue
- void updateSlotWithKxQ(final int slotNo, final int newValue) {
- Hll4Update.internalHll4Update(this, slotNo, newValue);
- }
-
- @Override
- byte[] toCompactByteArray() {
- final boolean srcSegIsCompact = extractCompactFlag(seg);
- final int totBytes = getCompactSerializationBytes();
- final byte[] byteArr = new byte[totBytes];
- final MemorySegment segOut = MemorySegment.ofArray(byteArr);
- if (srcSegIsCompact) { //seg is already consistent with result
- MemorySegment.copy(seg, 0, segOut, 0, totBytes);
- return byteArr;
- }
- //everything but the aux array is consistent
- MemorySegment.copy(seg, 0, segOut, 0, auxStart);
- if (auxHashMap != null) {
- final int auxCount = auxHashMap.getAuxCount();
- insertAuxCount(segOut, auxCount);
- insertLgArr(segOut, auxHashMap.getLgAuxArrInts()); //only used for direct HLL
- final PairIterator itr = auxHashMap.getIterator();
- int cnt = 0;
- while (itr.nextValid()) { //works whether src has compact MemorySegment or not
- insertInt(segOut, auxStart + (cnt++ << 2), itr.getPair());
- }
- assert cnt == auxCount;
- }
- insertCompactFlag(segOut, true);
- return byteArr;
- }
-
- @Override
- byte[] toUpdatableByteArray() {
- final boolean segIsCompact = extractCompactFlag(seg);
- final int totBytes = getUpdatableSerializationBytes();
- final byte[] byteArr = new byte[totBytes];
- final MemorySegment segOut = MemorySegment.ofArray(byteArr);
-
- if (!segIsCompact) { //both seg and target are updatable
- MemorySegment.copy(seg, 0, segOut, 0, totBytes);
-
- return byteArr;
- }
- //seg is compact, need to handle auxArr. Easiest way:
- final HllSketch heapSk = HllSketch.heapify(seg);
- return heapSk.toUpdatableByteArray();
- }
-
- //ITERATOR
-
- final class DirectHll4Iterator extends HllPairIterator {
-
- DirectHll4Iterator(final int lengthPairs) {
- super(lengthPairs);
- }
-
- @Override
- int value() {
- return getSlotValue(index);
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/DirectHll6Array.java b/src/main/java/org/apache/datasketches/hll/DirectHll6Array.java
deleted file mode 100644
index 4d35be674..000000000
--- a/src/main/java/org/apache/datasketches/hll/DirectHll6Array.java
+++ /dev/null
@@ -1,145 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_SHORT_UNALIGNED;
-import static org.apache.datasketches.hll.HllUtil.VAL_MASK_6;
-import static org.apache.datasketches.hll.HllUtil.noWriteAccess;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- */
-final class DirectHll6Array extends DirectHllArray {
-
- //Called by HllSketch.writableWrap(), DirectCouponList.promoteListOrSetToHll
- DirectHll6Array(final int lgConfigK, final MemorySegment wseg) {
- super(lgConfigK, TgtHllType.HLL_6, wseg);
- }
-
- //Called by HllSketch.wrap(MemorySegment)
- DirectHll6Array(final int lgConfigK, final MemorySegment seg, final boolean readOnly) {
- super(lgConfigK, TgtHllType.HLL_6, seg, readOnly);
- }
-
- @Override
- HllSketchImpl copy() {
- return Hll6Array.heapify(seg);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- if (wseg == null) { noWriteAccess(); }
- final int newValue = HllUtil.getPairValue(coupon);
- final int configKmask = (1 << getLgConfigK()) - 1;
- final int slotNo = HllUtil.getPairLow26(coupon) & configKmask;
- updateSlotWithKxQ(slotNo, newValue);
- return this;
- }
-
- @Override
- int getHllByteArrBytes() {
- return hll6ArrBytes(lgConfigK);
- }
-
- @Override
- int getNibble(final int slotNo) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override int getSlotValue(final int slotNo) {
- return get6Bit(seg, HLL_BYTE_ARR_START, slotNo);
- }
-
- @Override
- PairIterator iterator() {
- return new DirectHll6Iterator(1 << lgConfigK);
- }
-
- @Override
- void putNibble(final int slotNo, final int nibValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Would be used by HllUnion, but not used because the gadget is always HLL8 type
- void updateSlotNoKxQ(final int slotNo, final int newValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Used by this couponUpdate()
- //updates HipAccum, CurMin, NumAtCurMin, KxQs and checks newValue > oldValue
- void updateSlotWithKxQ(final int slotNo, final int newValue) {
- final int oldValue = getSlotValue(slotNo);
- if (newValue > oldValue) {
- put6Bit(wseg, HLL_BYTE_ARR_START, slotNo, newValue);
- hipAndKxQIncrementalUpdate(this, oldValue, newValue);
- if (oldValue == 0) {
- decNumAtCurMin(); //overloaded as num zeros
- assert getNumAtCurMin() >= 0;
- }
- }
- }
-
- //off-heap / direct
- private static void put6Bit(final MemorySegment wseg, final int offsetBytes, final int slotNo,
- final int newValue) {
- final int startBit = slotNo * 6;
- final int shift = startBit & 0X7;
- final int byteIdx = (startBit >>> 3) + offsetBytes;
- final int valShifted = (newValue & 0X3F) << shift;
- final int curMasked = wseg.get(JAVA_SHORT_UNALIGNED, byteIdx) & (~(VAL_MASK_6 << shift));
- final short insert = (short) (curMasked | valShifted);
- wseg.set(JAVA_SHORT_UNALIGNED, byteIdx, insert);
- }
-
- //off-heap / direct
- private static int get6Bit(final MemorySegment seg, final int offsetBytes, final int slotNo) {
- final int startBit = slotNo * 6;
- final int shift = startBit & 0X7;
- final int byteIdx = (startBit >>> 3) + offsetBytes;
- return (byte) ((seg.get(JAVA_SHORT_UNALIGNED, byteIdx) >>> shift) & 0X3F);
- }
-
- //ITERATOR
-
- final class DirectHll6Iterator extends HllPairIterator {
- int bitOffset;
-
- DirectHll6Iterator(final int lengthPairs) {
- super(lengthPairs);
- bitOffset = -6;
- }
-
- @Override
- int value() {
- bitOffset += 6;
- final int tmp = seg.get(JAVA_SHORT_UNALIGNED, HLL_BYTE_ARR_START + (bitOffset / 8));
- final int shift = (bitOffset % 8) & 0X7;
- return (tmp >>> shift) & VAL_MASK_6;
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/DirectHll8Array.java b/src/main/java/org/apache/datasketches/hll/DirectHll8Array.java
deleted file mode 100644
index 34714fb75..000000000
--- a/src/main/java/org/apache/datasketches/hll/DirectHll8Array.java
+++ /dev/null
@@ -1,127 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.hll.HllUtil.VAL_MASK_6;
-import static org.apache.datasketches.hll.HllUtil.noWriteAccess;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * Uses 8 bits per slot in a byte array.
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class DirectHll8Array extends DirectHllArray {
-
- //Called by HllSketch.writableWrap(), DirectCouponList.promoteListOrSetToHll
- DirectHll8Array(final int lgConfigK, final MemorySegment wseg) {
- super(lgConfigK, TgtHllType.HLL_8, wseg);
- }
-
- //Called by HllSketch.wrap(MemorySegment)
- DirectHll8Array(final int lgConfigK, final MemorySegment seg, final boolean readOnly) {
- super(lgConfigK, TgtHllType.HLL_8, seg, readOnly);
- }
-
- @Override
- HllSketchImpl copy() {
- return Hll8Array.heapify(seg);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- if (wseg == null) { noWriteAccess(); }
- final int newValue = HllUtil.getPairValue(coupon);
- final int configKmask = (1 << getLgConfigK()) - 1;
- final int slotNo = HllUtil.getPairLow26(coupon) & configKmask;
-
- updateSlotWithKxQ(slotNo, newValue);
- return this;
- }
-
- @Override
- int getHllByteArrBytes() {
- return hll8ArrBytes(lgConfigK);
- }
-
- @Override
- int getNibble(final int slotNo) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override int getSlotValue(final int slotNo) {
- return seg.get(JAVA_BYTE, HLL_BYTE_ARR_START + slotNo) & VAL_MASK_6;
- }
-
- @Override
- PairIterator iterator() {
- return new DirectHll8Iterator(1 << lgConfigK);
- }
-
- @Override
- void putNibble(final int slotNo, final int nibValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Used by HllUnion when source is not HLL8
- void updateSlotNoKxQ(final int slotNo, final int newValue) {
- final int oldValue = getSlotValue(slotNo);
- if (newValue > oldValue) {
- wseg.set(JAVA_BYTE, HLL_BYTE_ARR_START + slotNo, (byte) (newValue & VAL_MASK_6));
- }
- }
-
- @Override
- //Used by this couponUpdate()
- //updates HipAccum, CurMin, NumAtCurMin, KxQs and checks newValue > oldValue
- void updateSlotWithKxQ(final int slotNo, final int newValue) {
- final int oldValue = getSlotValue(slotNo);
- if (newValue > oldValue) {
- wseg.set(JAVA_BYTE, HLL_BYTE_ARR_START + slotNo, (byte) (newValue & VAL_MASK_6));
- hipAndKxQIncrementalUpdate(this, oldValue, newValue);
- if (oldValue == 0) {
- decNumAtCurMin();
- assert getNumAtCurMin() >= 0;
- }
- }
- }
-
- //ITERATOR
-
- final class DirectHll8Iterator extends HllPairIterator {
-
- DirectHll8Iterator(final int lengthPairs) {
- super(lengthPairs);
- }
-
- @Override
- int value() {
- final int tmp = seg.get(JAVA_BYTE, HLL_BYTE_ARR_START + index);
- return tmp & VAL_MASK_6;
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/DirectHllArray.java b/src/main/java/org/apache/datasketches/hll/DirectHllArray.java
deleted file mode 100644
index e0fb7594e..000000000
--- a/src/main/java/org/apache/datasketches/hll/DirectHllArray.java
+++ /dev/null
@@ -1,292 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_DOUBLE_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.common.Util.clear;
-import static org.apache.datasketches.hll.PreambleUtil.CUR_MIN_COUNT_INT;
-import static org.apache.datasketches.hll.PreambleUtil.HIP_ACCUM_DOUBLE;
-import static org.apache.datasketches.hll.PreambleUtil.extractCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.extractCurMode;
-import static org.apache.datasketches.hll.PreambleUtil.extractEmptyFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractHipAccum;
-import static org.apache.datasketches.hll.PreambleUtil.extractKxQ0;
-import static org.apache.datasketches.hll.PreambleUtil.extractKxQ1;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgK;
-import static org.apache.datasketches.hll.PreambleUtil.extractNumAtCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.extractOooFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractRebuildCurMinNumKxQFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractTgtHllType;
-import static org.apache.datasketches.hll.PreambleUtil.insertAuxCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.insertEmptyFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertHipAccum;
-import static org.apache.datasketches.hll.PreambleUtil.insertKxQ0;
-import static org.apache.datasketches.hll.PreambleUtil.insertKxQ1;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.insertNumAtCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.insertOooFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertRebuildCurMinNumKxQFlag;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.MemorySegmentStatus;
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * @author Lee Rhodes
- */
-abstract class DirectHllArray extends AbstractHllArray {
- MemorySegment wseg; //used for writable direct
- MemorySegment seg; //used for compact, read-only direct
- Object segObj; //used temporarily for byte-array
- final boolean compact;
-
- private static int checkSegCompactFlag(final MemorySegment wseg, final int lgConfigK) {
- assert !extractCompactFlag(wseg);
- return lgConfigK;
- }
-
- //MemorySegment must be already initialized and may have data. Writable, must not be Compact.
- DirectHllArray(final int lgConfigK, final TgtHllType tgtHllType, final MemorySegment wseg) {
- super(checkSegCompactFlag(wseg, lgConfigK), tgtHllType, CurMode.HLL);
- this.wseg = wseg;
- seg = wseg;
- segObj = wseg.toArray(JAVA_BYTE);
- compact = extractCompactFlag(seg);
- insertEmptyFlag(wseg, false);
- }
-
- //MemorySegment must already be initialized and should have data. Read-only. May be Compact or not
- DirectHllArray(final int lgConfigK, final TgtHllType tgtHllType, final MemorySegment seg, final boolean readOnly) {
- super(lgConfigK, tgtHllType, CurMode.HLL);
- wseg = null;
- this.seg = readOnly ? seg.asReadOnly() : seg;
- segObj = seg.toArray(JAVA_BYTE);
- compact = extractCompactFlag(seg);
- }
-
- //only called by DirectAuxHashMap
- final void updateMemorySegment(final MemorySegment newWseg) {
- wseg = newWseg;
- seg = newWseg;
- segObj = wseg.toArray(JAVA_BYTE);
- }
-
- @Override
- void addToHipAccum(final double delta) {
- checkReadOnly(wseg);
- final double hipAccum = seg.get(JAVA_DOUBLE_UNALIGNED, HIP_ACCUM_DOUBLE);
- wseg.set(JAVA_DOUBLE_UNALIGNED, HIP_ACCUM_DOUBLE, hipAccum + delta);
- }
-
- @Override
- void decNumAtCurMin() {
- checkReadOnly(wseg);
- int numAtCurMin = seg.get(JAVA_INT_UNALIGNED, CUR_MIN_COUNT_INT);
- wseg.set(JAVA_INT_UNALIGNED, CUR_MIN_COUNT_INT, --numAtCurMin);
- }
-
- @Override
- int getCurMin() {
- return extractCurMin(seg);
- }
-
- @Override
- CurMode getCurMode() {
- return extractCurMode(seg);
- }
-
- @Override
- double getHipAccum() {
- return extractHipAccum(seg);
- }
-
- @Override
- double getKxQ0() {
- return extractKxQ0(seg);
- }
-
- @Override
- double getKxQ1() {
- return extractKxQ1(seg);
- }
-
- @Override
- int getLgConfigK() {
- return extractLgK(seg);
- }
-
- @Override
- MemorySegment getMemorySegment() {
- return seg;
- }
-
- @Override
- AuxHashMap getNewAuxHashMap() {
- return new DirectAuxHashMap(this, true);
- }
-
- @Override
- int getNumAtCurMin() {
- return extractNumAtCurMin(seg);
- }
-
- @Override
- TgtHllType getTgtHllType() {
- return extractTgtHllType(seg);
- }
-
- @Override
- boolean isCompact() {
- return compact;
- }
-
- @Override
- boolean isEmpty() {
- return extractEmptyFlag(seg);
- }
-
- @Override
- boolean hasMemorySegment() {
- return seg.scope().isAlive();
- }
-
- @Override
- boolean isOffHeap() {
- return seg.isNative();
- }
-
- @Override
- boolean isOutOfOrder() {
- return extractOooFlag(seg);
- }
-
- @Override
- boolean isSameResource(final MemorySegment seg) {
- return MemorySegmentStatus.isSameResource(this.seg, seg);
- }
-
- @Override
- boolean isRebuildCurMinNumKxQFlag() {
- return extractRebuildCurMinNumKxQFlag(seg);
- }
-
- @Override
- void putAuxHashMap(final AuxHashMap auxHashMap, final boolean compact) {
- if (auxHashMap instanceof HeapAuxHashMap) {
- if (compact) {
- this.auxHashMap = auxHashMap; //heap and compact
- } else { //heap and not compact
- final int[] auxArr = auxHashMap.getAuxIntArr();
- MemorySegment.copy(auxArr, 0, wseg, JAVA_INT_UNALIGNED, auxStart, auxArr.length);
- insertLgArr(wseg, auxHashMap.getLgAuxArrInts());
- insertAuxCount(wseg, auxHashMap.getAuxCount());
- this.auxHashMap = new DirectAuxHashMap(this, false);
- }
- } else { //DirectAuxHashMap
- assert !compact; //must not be compact
- this.auxHashMap = auxHashMap; //In case of read-only this works.
- }
- }
-
- @Override
- void putCurMin(final int curMin) {
- checkReadOnly(wseg);
- insertCurMin(wseg, curMin);
- }
-
- @Override
- void putEmptyFlag(final boolean empty) {
- checkReadOnly(wseg);
- insertEmptyFlag(wseg, empty);
- }
-
- @Override
- void putHipAccum(final double hipAccum) {
- checkReadOnly(wseg);
- insertHipAccum(wseg, hipAccum);
- }
-
- @Override
- void putKxQ0(final double kxq0) {
- checkReadOnly(wseg);
- insertKxQ0(wseg, kxq0);
- }
-
- @Override //called very very very rarely
- void putKxQ1(final double kxq1) {
- checkReadOnly(wseg);
- insertKxQ1(wseg, kxq1);
- }
-
- @Override
- void putNumAtCurMin(final int numAtCurMin) {
- checkReadOnly(wseg);
- insertNumAtCurMin(wseg, numAtCurMin);
- }
-
- @Override //not used on the direct side
- void putOutOfOrder(final boolean oooFlag) {
- if (oooFlag) { putHipAccum(0); }
- checkReadOnly(wseg);
- insertOooFlag(wseg, oooFlag);
- }
-
- @Override
- void putRebuildCurMinNumKxQFlag(final boolean rebuild) {
- checkReadOnly(wseg);
- insertRebuildCurMinNumKxQFlag(wseg, rebuild);
- }
-
- @Override //used by HLL6 and HLL8, overridden by HLL4
- byte[] toCompactByteArray() {
- return toUpdatableByteArray(); //indistinguishable for HLL6 and HLL8
- }
-
- @Override //used by HLL6 and HLL8, overridden by HLL4
- byte[] toUpdatableByteArray() {
- final int totBytes = getCompactSerializationBytes();
- final byte[] byteArr = new byte[totBytes];
- final MemorySegment segOut = MemorySegment.ofArray(byteArr);
- MemorySegment.copy(seg, 0, segOut, 0, totBytes);
- insertCompactFlag(segOut, false);
- return byteArr;
- }
-
- @Override
- HllSketchImpl reset() {
- checkReadOnly(wseg);
- insertEmptyFlag(wseg, true);
- final int bytes = HllSketch.getMaxUpdatableSerializationBytes(lgConfigK, tgtHllType);
- clear(wseg, 0, bytes);
- return DirectCouponList.newInstance(lgConfigK, tgtHllType, wseg);
- }
-
- private static final void checkReadOnly(final MemorySegment wseg) {
- if (wseg == null) {
- throw new SketchesArgumentException("Cannot modify a read-only sketch");
- }
- }
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HarmonicNumbers.java b/src/main/java/org/apache/datasketches/hll/HarmonicNumbers.java
deleted file mode 100644
index 1d236d2f6..000000000
--- a/src/main/java/org/apache/datasketches/hll/HarmonicNumbers.java
+++ /dev/null
@@ -1,94 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class HarmonicNumbers {
-
- /**
- * This is the estimator you would use for flat bit map random accessed, similar to a Bloom filter.
- * @param bitVectorLength the length of the bit vector in bits. Must be > 0.
- * @param numBitsSet the number of bits set in this bit vector. Must be ≥ 0 and ≤
- * bitVectorLength.
- * @return the estimate.
- */
- static double getBitMapEstimate(final int bitVectorLength, final int numBitsSet) {
- return (bitVectorLength * (harmonicNumber(bitVectorLength)
- - harmonicNumber(bitVectorLength - numBitsSet))); //converts to numZeros
- }
-
- //In C: two-registers.c Line 1170
- private static final int NUM_EXACT_HARMONIC_NUMBERS = 25;
-
- private static double[] tableOfExactHarmonicNumbers = {
- 0.0, // 0
- 1.0, // 1
- 1.5, // 2
- 11.0 / 6.0, // 3
- 25.0 / 12.0, // 4
- 137.0 / 60.0, // 5
- 49.0 / 20.0, // 6
- 363.0 / 140.0, // 7
- 761.0 / 280.0, // 8
- 7129.0 / 2520.0, // 9
- 7381.0 / 2520.0, // 10
- 83711.0 / 27720.0, // 11
- 86021.0 / 27720.0, // 12
- 1145993.0 / 360360.0, // 13
- 1171733.0 / 360360.0, // 14
- 1195757.0 / 360360.0, // 15
- 2436559.0 / 720720.0, // 16
- 42142223.0 / 12252240.0, // 17
- 14274301.0 / 4084080.0, // 18
- 275295799.0 / 77597520.0, // 19
- 55835135.0 / 15519504.0, // 20
- 18858053.0 / 5173168.0, // 21
- 19093197.0 / 5173168.0, // 22
- 444316699.0 / 118982864.0, // 23
- 1347822955.0 / 356948592.0 // 24
- };
-
- //In C: two-register.c Line 1202
- private static final double EULER_MASCHERONI_CONSTANT = 0.577215664901532860606512090082;
-
- private static double harmonicNumber(final long x_i) {
- if (x_i < NUM_EXACT_HARMONIC_NUMBERS) {
- return tableOfExactHarmonicNumbers[(int) x_i];
- } else {
- final double x = x_i;
- final double invSq = 1.0 / (x * x);
- double sum = Math.log(x) + EULER_MASCHERONI_CONSTANT + (1.0 / (2.0 * x));
- /* note: the number of terms included from this series expansion is appropriate
- for the size of the exact table (25) and the precision of doubles */
- double pow = invSq; /* now n^-2 */
- sum -= pow * (1.0 / 12.0);
- pow *= invSq; /* now n^-4 */
- sum += pow * (1.0 / 120.0);
- pow *= invSq; /* now n^-6 */
- sum -= pow * (1.0 / 252.0);
- pow *= invSq; /* now n^-8 */
- sum += pow * (1.0 / 240.0);
- return sum;
- }
- }
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HeapAuxHashMap.java b/src/main/java/org/apache/datasketches/hll/HeapAuxHashMap.java
deleted file mode 100644
index 3a1600676..000000000
--- a/src/main/java/org/apache/datasketches/hll/HeapAuxHashMap.java
+++ /dev/null
@@ -1,231 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_DENOM;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_NUMER;
-import static org.apache.datasketches.hll.PreambleUtil.extractInt;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgArr;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class HeapAuxHashMap implements AuxHashMap {
- private final int lgConfigK; //required for #slot bits
- private int lgAuxArrInts;
- private int auxCount;
- private int[] auxIntArr; //used by Hll4Array
-
- /**
- * Standard constructor
- * @param lgAuxArrInts the log size of the aux integer array
- * @param lgConfigK must be 7 to 21
- */
- HeapAuxHashMap(final int lgAuxArrInts, final int lgConfigK) {
- this.lgConfigK = lgConfigK;
- this.lgAuxArrInts = lgAuxArrInts;
- auxIntArr = new int[1 << lgAuxArrInts];
- }
-
- /**
- * Copy constructor
- * @param that another AuxHashMap
- */
- HeapAuxHashMap(final HeapAuxHashMap that) {
- lgConfigK = that.lgConfigK;
- lgAuxArrInts = that.lgAuxArrInts;
- auxCount = that.auxCount;
- auxIntArr = that.auxIntArr.clone();
- }
-
- static HeapAuxHashMap heapify(final MemorySegment seg, final long offset, final int lgConfigK,
- final int auxCount, final boolean srcCompact) {
- final int lgAuxArrInts;
- final HeapAuxHashMap auxMap;
- if (srcCompact) { //early versions did not use LgArr byte field
- lgAuxArrInts = PreambleUtil.computeLgArr(seg, auxCount, lgConfigK);
- } else { //updatable
- lgAuxArrInts = extractLgArr(seg);
- }
- auxMap = new HeapAuxHashMap(lgAuxArrInts, lgConfigK);
-
- final int configKmask = (1 << lgConfigK) - 1;
-
- if (srcCompact) {
- for (int i = 0; i < auxCount; i++) {
- final int pair = extractInt(seg, offset + (i << 2));
- final int slotNo = HllUtil.getPairLow26(pair) & configKmask;
- final int value = HllUtil.getPairValue(pair);
- auxMap.mustAdd(slotNo, value); //increments count
- }
- } else { //updatable
- final int auxArrInts = 1 << lgAuxArrInts;
- for (int i = 0; i < auxArrInts; i++) {
- final int pair = extractInt(seg, offset + (i << 2));
- if (pair == EMPTY) { continue; }
- final int slotNo = HllUtil.getPairLow26(pair) & configKmask;
- final int value = HllUtil.getPairValue(pair);
- auxMap.mustAdd(slotNo, value); //increments count
- }
- }
- return auxMap;
- }
-
- @Override
- public HeapAuxHashMap copy() {
- return new HeapAuxHashMap(this);
- }
-
- @Override
- public int getAuxCount() {
- return auxCount;
- }
-
- @Override
- public int[] getAuxIntArr() {
- return auxIntArr;
- }
-
- @Override
- public int getCompactSizeBytes() {
- return auxCount << 2;
- }
-
- @Override
- public PairIterator getIterator() {
- return new IntArrayPairIterator(auxIntArr, lgConfigK);
- }
-
- @Override
- public int getLgAuxArrInts() {
- return lgAuxArrInts;
- }
-
- @Override
- public int getUpdatableSizeBytes() {
- return 4 << lgAuxArrInts;
- }
-
- @Override
- public boolean hasMemorySegment() {
- return false;
- }
-
- @Override
- public boolean isOffHeap() {
- return false;
- }
-
- @Override
- public boolean isSameResource(final MemorySegment seg) {
- return false;
- }
-
- //In C: two-registers.c Line 300.
- @Override
- public void mustAdd(final int slotNo, final int value) {
- final int index = find(auxIntArr, lgAuxArrInts, lgConfigK, slotNo);
- final int pair = HllUtil.pair(slotNo, value);
- if (index >= 0) {
- final String pairStr = HllUtil.pairString(pair);
- throw new SketchesStateException("Found a slotNo that should not be there: " + pairStr);
- }
- //Found empty entry
- auxIntArr[~index] = pair;
- auxCount++;
- checkGrow();
- }
-
- //In C: two-registers.c Line 205
- @Override
- public int mustFindValueFor(final int slotNo) {
- final int index = find(auxIntArr, lgAuxArrInts, lgConfigK, slotNo);
- if (index >= 0) {
- return HllUtil.getPairValue(auxIntArr[index]);
- }
- throw new SketchesStateException("SlotNo not found: " + slotNo);
- }
-
- //In C: two-registers.c Line 321.
- @Override
- public void mustReplace(final int slotNo, final int value) {
- final int idx = find(auxIntArr, lgAuxArrInts, lgConfigK, slotNo);
- if (idx >= 0) {
- auxIntArr[idx] = HllUtil.pair(slotNo, value); //replace
- return;
- }
- final String pairStr = HllUtil.pairString(HllUtil.pair(slotNo, value));
- throw new SketchesStateException("Pair not found: " + pairStr);
- }
-
- //Searches the Aux arr hash table for an empty or a matching slotNo depending on the context.
- //If entire entry is empty, returns one's complement of index = found empty.
- //If entry contains given slotNo, returns its index = found slotNo.
- //Continues searching.
- //If the probe comes back to original index, throws an exception.
- private static int find(final int[] auxArr, final int lgAuxArrInts, final int lgConfigK,
- final int slotNo) {
- assert lgAuxArrInts < lgConfigK;
- final int auxArrMask = (1 << lgAuxArrInts) - 1;
- final int configKmask = (1 << lgConfigK) - 1;
- int probe = slotNo & auxArrMask;
- final int loopIndex = probe;
- do {
- final int arrVal = auxArr[probe];
- if (arrVal == EMPTY) { //Compares on entire entry
- return ~probe; //empty
- }
- else if (slotNo == (arrVal & configKmask)) { //Compares only on slotNo
- return probe; //found given slotNo, return probe = index into aux array
- }
- final int stride = (slotNo >>> lgAuxArrInts) | 1;
- probe = (probe + stride) & auxArrMask;
- } while (probe != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- private void checkGrow() {
- if ((RESIZE_DENOM * auxCount) > (RESIZE_NUMER * auxIntArr.length)) {
- growAuxSpace();
- }
- }
-
- private void growAuxSpace() {
- final int[] oldArray = auxIntArr;
- final int configKmask = (1 << lgConfigK) - 1;
- auxIntArr = new int[1 << ++lgAuxArrInts];
- for (int i = 0; i < oldArray.length; i++) {
- final int fetched = oldArray[i];
- if (fetched != EMPTY) {
- //find empty in new array
- final int idx = find(auxIntArr, lgAuxArrInts, lgConfigK, fetched & configKmask);
- auxIntArr[~idx] = fetched;
- }
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/Hll4Array.java b/src/main/java/org/apache/datasketches/hll/Hll4Array.java
deleted file mode 100644
index f6295fa78..000000000
--- a/src/main/java/org/apache/datasketches/hll/Hll4Array.java
+++ /dev/null
@@ -1,180 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.hll.HllUtil.AUX_TOKEN;
-import static org.apache.datasketches.hll.HllUtil.KEY_BITS_26;
-import static org.apache.datasketches.hll.HllUtil.LG_AUX_ARR_INTS;
-import static org.apache.datasketches.hll.HllUtil.hiNibbleMask;
-import static org.apache.datasketches.hll.HllUtil.loNibbleMask;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.extractAuxCount;
-import static org.apache.datasketches.hll.PreambleUtil.extractCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgK;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * Uses 4 bits per slot in a packed byte array.
- * @author Lee Rhodes
- */
-final class Hll4Array extends HllArray {
-
- /**
- * Standard constructor for new instance
- * @param lgConfigK the configured Lg K
- */
- Hll4Array(final int lgConfigK) {
- super(lgConfigK, TgtHllType.HLL_4);
- hllByteArr = new byte[hll4ArrBytes(lgConfigK)];
- }
-
- /**
- * Copy constructor
- * @param that another Hll4Array
- */
- Hll4Array(final Hll4Array that) {
- super(that);
- }
-
- static Hll4Array heapify(final MemorySegment seg) {
- final int lgConfigK = extractLgK(seg);
- final Hll4Array hll4Array = new Hll4Array(lgConfigK);
- HllArray.extractCommonHll(seg, hll4Array);
-
- //load AuxHashMap
- final int auxStart = hll4Array.auxStart;
- final int auxCount = extractAuxCount(seg);
- final boolean compact = extractCompactFlag(seg);
- HeapAuxHashMap auxHashMap = null;
- if (auxCount > 0) {
- auxHashMap = HeapAuxHashMap.heapify(seg, auxStart, lgConfigK, auxCount, compact);
- }
- hll4Array.putAuxHashMap(auxHashMap, false);
- return hll4Array;
- }
-
- @Override
- Hll4Array copy() {
- return new Hll4Array(this);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- final int newValue = coupon >>> KEY_BITS_26;
- final int configKmask = (1 << getLgConfigK()) - 1;
- final int slotNo = coupon & configKmask;
- updateSlotWithKxQ(slotNo, newValue);
- return this;
- }
-
- @Override
- int getNibble(final int slotNo) {
- int theByte = hllByteArr[slotNo >>> 1];
- if ((slotNo & 1) > 0) { //odd?
- theByte >>>= 4;
- }
- return theByte & loNibbleMask;
- }
-
- @Override
- int getSlotValue(final int slotNo) {
- final int nib = getNibble(slotNo);
- if (nib == AUX_TOKEN) {
- final AuxHashMap auxHashMap = getAuxHashMap();
- return auxHashMap.mustFindValueFor(slotNo); //auxHashMap cannot be null here
- } else {
- return nib + getCurMin();
- }
- }
-
- @Override
- int getUpdatableSerializationBytes() {
- final AuxHashMap auxHashMap = getAuxHashMap();
- final int auxBytes;
- if (auxHashMap == null) {
- auxBytes = 4 << LG_AUX_ARR_INTS[lgConfigK];
- } else {
- auxBytes = 4 << auxHashMap.getLgAuxArrInts();
- }
- return HLL_BYTE_ARR_START + getHllByteArrBytes() + auxBytes;
- }
-
- @Override
- PairIterator iterator() {
- return new HeapHll4Iterator(1 << lgConfigK);
- }
-
- @Override
- void putNibble(final int slotNo, final int nibValue) {
- final int byteno = slotNo >>> 1;
- final int oldValue = hllByteArr[byteno];
- if ((slotNo & 1) == 0) { // set low nibble
- hllByteArr[byteno] = (byte) ((oldValue & hiNibbleMask) | (nibValue & loNibbleMask));
- } else { //set high nibble
- hllByteArr[byteno] = (byte) ((oldValue & loNibbleMask) | ((nibValue << 4) & hiNibbleMask));
- }
- }
-
- @Override
- //Would be used by HllUnion, but not used because the gadget is always HLL8 type
- void updateSlotNoKxQ(final int slotNo, final int newValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Used by this couponUpdate()
- //updates HipAccum, CurMin, NumAtCurMin, KxQs and checks newValue > oldValue
- void updateSlotWithKxQ(final int slotNo, final int newValue) {
- Hll4Update.internalHll4Update(this, slotNo, newValue);
- }
-
- @Override
- byte[] toCompactByteArray() {
- return ToByteArrayImpl.toHllByteArray(this, true);
- }
-
- //ITERATOR
-
- final class HeapHll4Iterator extends HllPairIterator {
-
- HeapHll4Iterator(final int lengthPairs) {
- super(lengthPairs);
- }
-
- @Override
- int value() {
- return getSlotValue(index);
- }
- }
-
- @Override
- MemorySegment getMemorySegment() {
- return null;
- }
-
- @Override
- boolean isSameResource(final MemorySegment seg) {
- return false;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/Hll4Update.java b/src/main/java/org/apache/datasketches/hll/Hll4Update.java
deleted file mode 100644
index a1b67275b..000000000
--- a/src/main/java/org/apache/datasketches/hll/Hll4Update.java
+++ /dev/null
@@ -1,201 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.hll.HllUtil.AUX_TOKEN;
-import static org.apache.datasketches.hll.HllUtil.LG_AUX_ARR_INTS;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * Update process common to Heap Hll 4 and Direct Hll 4
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class Hll4Update {
-
- //Uses lgConfigK, curMin, numAtCurMin, auxMap
- //Only called by Hll4Array and DirectHll4Array
- //In C: two-registers.c Line 836 in "hhb_abstract_set_slot_if_new_value_bigger" non-sparse
- static void internalHll4Update(final AbstractHllArray host, final int slotNo,
- final int newValue) {
- assert ((0 <= slotNo) && (slotNo < (1 << host.getLgConfigK())));
-
- final int curMin = host.getCurMin();
- final int rawStoredOldNibble = host.getNibble(slotNo); //could be 0
- final int lbOnOldValue = rawStoredOldNibble + curMin; //provable lower bound, could be 0
-
- if (newValue <= lbOnOldValue) { return; }
- //Thus: newValue > lbOnOldValue AND newValue > curMin
-
- AuxHashMap auxHashMap; // = host.getAuxHashMap();
- final int actualOldValue;
- final int shiftedNewValue; //value - curMin
-
- //Based on whether we have an AUX_TOKEN and whether the shiftedNewValue is greater than
- // AUX_TOKEN, we have four cases for how to actually modify the data structure:
- // 1. (shiftedNewValue >= AUX_TOKEN) && (rawStoredOldNibble = AUX_TOKEN) //881:
- // The byte array already contains aux token
- // This is the case where old and new values are both exceptions.
- // Therefore, the 4-bit array already is AUX_TOKEN. Only need to update auxMap
- // 2. (shiftedNewValue < AUX_TOKEN) && (rawStoredOldNibble = AUX_TOKEN) //885
- // This is the (hypothetical) case where old value is an exception and the new one is not,
- // which is impossible given that curMin has not changed here and the newValue > oldValue.
- // 3. (shiftedNewValue >= AUX_TOKEN) && (rawStoredOldNibble < AUX_TOKEN) //892
- // This is the case where the old value is not an exception and the new value is.
- // Therefore the AUX_TOKEN must be stored in the 4-bit array and the new value
- // added to the exception table.
- // 4. (shiftedNewValue < AUX_TOKEN) && (rawStoredOldNibble < AUX_TOKEN) //897
- // This is the case where neither the old value nor the new value is an exception.
- // Therefore we just overwrite the 4-bit array with the shifted new value.
-
- if (rawStoredOldNibble == AUX_TOKEN) { //846 Note: This is rare and really hard to test!
- auxHashMap = host.getAuxHashMap(); //auxHashMap must already exist.
- assert auxHashMap != null;
- actualOldValue = auxHashMap.mustFindValueFor(slotNo);//lgtm [java/dereferenced-value-may-be-null]
- if (newValue <= actualOldValue) { return; }
- //We know that the array will be changed, but we haven't actually updated yet.
- AbstractHllArray.hipAndKxQIncrementalUpdate(host, actualOldValue, newValue);
- shiftedNewValue = newValue - curMin;
- assert (shiftedNewValue >= 0);
-
- if (shiftedNewValue >= AUX_TOKEN) { //CASE 1:
- auxHashMap.mustReplace(slotNo, newValue); //lgtm [java/dereferenced-value-may-be-null]
- }
- //else //CASE 2: impossible
-
- } else { //rawStoredOldNibble < AUX_TOKEN
- actualOldValue = lbOnOldValue;
- //We know that the array will be changed, but we haven't actually updated yet.
- AbstractHllArray.hipAndKxQIncrementalUpdate(host, actualOldValue, newValue);
- shiftedNewValue = newValue - curMin;
- assert (shiftedNewValue >= 0);
-
- if (shiftedNewValue >= AUX_TOKEN) { //CASE 3: //892
- host.putNibble(slotNo, AUX_TOKEN);
- auxHashMap = host.getAuxHashMap();
- if (auxHashMap == null) {
- auxHashMap = host.getNewAuxHashMap();
- host.putAuxHashMap(auxHashMap, false);
- }
- auxHashMap.mustAdd(slotNo, newValue);
- }
- else { // CASE 4: //897
- host.putNibble(slotNo, shiftedNewValue);
- }
- }
-
- // We just changed the HLL array, so it might be time to change curMin
- if (actualOldValue == curMin) { //908
- assert (host.getNumAtCurMin() >= 1);
- host.decNumAtCurMin();
- while (host.getNumAtCurMin() == 0) {
- //increases curMin by 1, and builds a new aux table,
- // shifts values in 4-bit table, and recounts curMin
- shiftToBiggerCurMin(host);
- }
- }
- }
-
- //This scheme only works with two double registers (2 kxq values).
- // HipAccum, kxq0 and kxq1 remain untouched.
- // This changes curMin, numAtCurMin, hllByteArr and auxMap.
- //Entering this routine assumes that all slots have valid nibbles > 0 and <= 15.
- //An AuxHashMap must exist if any values in the current hllByteArray are already 15.
- //In C: again-two-registers.c Lines 710 "hhb_shift_to_bigger_curmin"
- private static void shiftToBiggerCurMin(final AbstractHllArray host) {
- final int oldCurMin = host.getCurMin();
- final int newCurMin = oldCurMin + 1;
- final int lgConfigK = host.getLgConfigK();
- final int configK = 1 << lgConfigK;
- final int configKmask = configK - 1;
-
- int numAtNewCurMin = 0;
- int numAuxTokens = 0;
-
- // Walk through the slots of 4-bit array decrementing stored values by one unless it
- // equals AUX_TOKEN, where it is left alone but counted to be checked later.
- // If oldStoredValue is 0 it is an error.
- // If the decremented value is 0, we increment numAtNewCurMin.
- // Because getNibble is masked to 4 bits oldStoredValue can never be > 15 or negative
- for (int i = 0; i < configK; i++) { //724
- int oldStoredNibble = host.getNibble(i);
- if (oldStoredNibble == 0) {
- throw new SketchesStateException("Array slots cannot be 0 at this point.");
- }
- if (oldStoredNibble < AUX_TOKEN) {
- host.putNibble(i, --oldStoredNibble);
- if (oldStoredNibble == 0) { numAtNewCurMin++; }
- } else { //oldStoredNibble == AUX_TOKEN
- numAuxTokens++;
- assert host.getAuxHashMap() != null : "AuxHashMap cannot be null at this point.";
- }
- }
-
- //If old AuxHashMap exists, walk through it updating some slots and build a new AuxHashMap
- // if needed.
- AuxHashMap newAuxMap = null;
- final AuxHashMap oldAuxMap = host.getAuxHashMap();
- if (oldAuxMap != null) {
- int slotNum;
- int oldActualVal;
- int newShiftedVal;
-
- final PairIterator itr = oldAuxMap.getIterator();
- while (itr.nextValid()) {
- slotNum = itr.getKey() & configKmask;
- oldActualVal = itr.getValue();
- newShiftedVal = oldActualVal - newCurMin;
- assert newShiftedVal >= 0;
-
- assert host.getNibble(slotNum) == AUX_TOKEN
- : "Array slot != AUX_TOKEN: " + host.getNibble(slotNum);
- if (newShiftedVal < AUX_TOKEN) { //756
- assert (newShiftedVal == 14);
- // The former exception value isn't one anymore, so it stays out of new AuxHashMap.
- // Correct the AUX_TOKEN value in the HLL array to the newShiftedVal (14).
- host.putNibble(slotNum, newShiftedVal);
- numAuxTokens--;
- }
- else { //newShiftedVal >= AUX_TOKEN
- // the former exception remains an exception, so must be added to the newAuxMap
- if (newAuxMap == null) {
- //Note: even in the direct case we use a heap aux map temporarily
- newAuxMap = new HeapAuxHashMap(LG_AUX_ARR_INTS[lgConfigK], lgConfigK);
- }
- newAuxMap.mustAdd(slotNum, oldActualVal);
- }
- } //end scan of oldAuxMap
- } //end if (auxHashMap != null)
- else { //oldAuxMap == null
- assert numAuxTokens == 0 : "auxTokens: " + numAuxTokens;
- }
-
- if (newAuxMap != null) {
- assert newAuxMap.getAuxCount() == numAuxTokens : "auxCount: " + newAuxMap.getAuxCount()
- + ", HLL tokens: " + numAuxTokens;
- }
- host.putAuxHashMap(newAuxMap, false); //if we are direct, this will do the right thing
-
- host.putCurMin(newCurMin);
- host.putNumAtCurMin(numAtNewCurMin);
- } //end of shiftToBiggerCurMin
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/Hll6Array.java b/src/main/java/org/apache/datasketches/hll/Hll6Array.java
deleted file mode 100644
index a0ddcbf59..000000000
--- a/src/main/java/org/apache/datasketches/hll/Hll6Array.java
+++ /dev/null
@@ -1,155 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.common.ByteArrayUtil.getShortLE;
-import static org.apache.datasketches.common.ByteArrayUtil.putShortLE;
-import static org.apache.datasketches.hll.HllUtil.KEY_BITS_26;
-import static org.apache.datasketches.hll.HllUtil.VAL_MASK_6;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgK;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * Uses 6 bits per slot in a packed byte array.
- * @author Lee Rhodes
- */
-final class Hll6Array extends HllArray {
-
- /**
- * Standard constructor for new instance
- * @param lgConfigK the configured Lg K
- */
- Hll6Array(final int lgConfigK) {
- super(lgConfigK, TgtHllType.HLL_6);
- hllByteArr = new byte[hll6ArrBytes(lgConfigK)];
- }
-
- /**
- * Copy constructor
- * @param that another Hll6Array
- */
- Hll6Array(final Hll6Array that) {
- super(that);
- }
-
- static Hll6Array heapify(final MemorySegment seg) {
- final int lgConfigK = extractLgK(seg);
- final Hll6Array hll6Array = new Hll6Array(lgConfigK);
- HllArray.extractCommonHll(seg, hll6Array);
- return hll6Array;
- }
-
- @Override
- Hll6Array copy() {
- return new Hll6Array(this);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- final int newValue = coupon >>> KEY_BITS_26;
- final int configKmask = (1 << lgConfigK) - 1;
- final int slotNo = coupon & configKmask;
- updateSlotWithKxQ(slotNo, newValue);
- return this;
- }
-
- @Override
- int getNibble(final int slotNo) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override int getSlotValue(final int slotNo) {
- return get6Bit(hllByteArr, 0, slotNo);
- }
-
- @Override
- PairIterator iterator() {
- return new HeapHll6Iterator(1 << lgConfigK);
- }
-
- @Override
- void putNibble(final int slotNo, final int nibValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Would be used by HllUnion, but not used because the gadget is always HLL8 type
- void updateSlotNoKxQ(final int slotNo, final int newValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Used by this couponUpdate()
- //updates HipAccum, CurMin, NumAtCurMin, KxQs and checks newValue > oldValue
- void updateSlotWithKxQ(final int slotNo, final int newValue) {
- final int oldValue = getSlotValue(slotNo);
- if (newValue > oldValue) {
- put6Bit(hllByteArr, 0, slotNo, newValue);
- hipAndKxQIncrementalUpdate(this, oldValue, newValue);
- if (oldValue == 0) {
- numAtCurMin--; //interpret numAtCurMin as num Zeros
- assert getNumAtCurMin() >= 0;
- }
- }
- }
-
- //on-heap
- private static void put6Bit(final byte[] arr, final int offsetBytes, final int slotNo,
- final int newValue) {
- final int startBit = slotNo * 6;
- final int shift = startBit & 0X7;
- final int byteIdx = (startBit >>> 3) + offsetBytes;
- final int valShifted = (newValue & 0X3F) << shift;
- final int curMasked = getShortLE(arr, byteIdx) & (~(VAL_MASK_6 << shift));
- final short insert = (short) (curMasked | valShifted);
- putShortLE(arr, byteIdx, insert);
- }
-
- //on-heap
- private static int get6Bit(final byte[] arr, final int offsetBytes, final int slotNo) {
- final int startBit = slotNo * 6;
- final int shift = startBit & 0X7;
- final int byteIdx = (startBit >>> 3) + offsetBytes;
- return (byte) ((getShortLE(arr, byteIdx) >>> shift) & 0X3F);
- }
-
- //ITERATOR
-
- private final class HeapHll6Iterator extends HllPairIterator {
- int bitOffset;
-
- HeapHll6Iterator(final int lengthPairs) {
- super(lengthPairs);
- bitOffset = - 6;
- }
-
- @Override
- int value() {
- bitOffset += 6;
- final int tmp = getShortLE(hllByteArr, bitOffset / 8);
- final int shift = (bitOffset % 8) & 0X7;
- return (tmp >>> shift) & VAL_MASK_6;
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/Hll8Array.java b/src/main/java/org/apache/datasketches/hll/Hll8Array.java
deleted file mode 100644
index 423cebfee..000000000
--- a/src/main/java/org/apache/datasketches/hll/Hll8Array.java
+++ /dev/null
@@ -1,142 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.KEY_BITS_26;
-import static org.apache.datasketches.hll.HllUtil.VAL_MASK_6;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgK;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * Uses 8 bits per slot in a byte array.
- * @author Lee Rhodes
- */
-final class Hll8Array extends HllArray {
-
- /**
- * Standard constructor for new instance
- * @param lgConfigK the configured Lg K
- */
- Hll8Array(final int lgConfigK) {
- super(lgConfigK, TgtHllType.HLL_8);
- hllByteArr = new byte[hll8ArrBytes(lgConfigK)];
- }
-
- /**
- * Copy constructor
- * @param that another Hll8Array
- */
- Hll8Array(final Hll8Array that) {
- super(that);
- }
-
- static Hll8Array heapify(final MemorySegment seg) {
- final int lgConfigK = extractLgK(seg);
- final Hll8Array hll8Array = new Hll8Array(lgConfigK);
- HllArray.extractCommonHll(seg, hll8Array);
- return hll8Array;
- }
-
- @Override
- Hll8Array copy() {
- return new Hll8Array(this);
- }
-
- @Override
- HllSketchImpl couponUpdate(final int coupon) {
- final int newValue = coupon >>> KEY_BITS_26;
- final int configKmask = (1 << lgConfigK) - 1;
- final int slotNo = coupon & configKmask;
- updateSlotWithKxQ(slotNo, newValue);
- return this;
- }
-
- @Override
- int getNibble(final int slotNo) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override int getSlotValue(final int slotNo) {
- return hllByteArr[slotNo] & VAL_MASK_6;
- }
-
- @Override
- PairIterator iterator() {
- return new HeapHll8Iterator(1 << lgConfigK);
- }
-
- @Override
- void putNibble(final int slotNo, final int nibValue) {
- throw new SketchesStateException("Improper access.");
- }
-
- @Override
- //Used by HllUnion when source is not HLL8
- void updateSlotNoKxQ(final int slotNo, final int newValue) {
- final int oldValue = getSlotValue(slotNo);
- hllByteArr[slotNo] = (byte) Math.max(newValue, oldValue);
- }
-
- @Override
- //Used by this couponUpdate()
- //updates HipAccum, CurMin, NumAtCurMin, KxQs and checks newValue > oldValue
- void updateSlotWithKxQ(final int slotNo, final int newValue) {
- final int oldValue = getSlotValue(slotNo);
- if (newValue > oldValue) {
- hllByteArr[slotNo] = (byte) (newValue & VAL_MASK_6);
- hipAndKxQIncrementalUpdate(this, oldValue, newValue);
- if (oldValue == 0) {
- numAtCurMin--; //interpret numAtCurMin as num Zeros
- assert getNumAtCurMin() >= 0;
- }
- }
- }
-
- //ITERATOR
-
- final class HeapHll8Iterator extends HllPairIterator {
-
- HeapHll8Iterator(final int lengthPairs) {
- super(lengthPairs);
- }
-
- @Override
- int value() {
- return hllByteArr[index] & VAL_MASK_6;
- }
-
- @Override
- public boolean nextValid() {
- while (++index < lengthPairs) {
- value = hllByteArr[index] & VAL_MASK_6;
- if (value != EMPTY) {
- return true;
- }
- }
- return false;
- }
-
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HllArray.java b/src/main/java/org/apache/datasketches/hll/HllArray.java
deleted file mode 100644
index ba86cb1bc..000000000
--- a/src/main/java/org/apache/datasketches/hll/HllArray.java
+++ /dev/null
@@ -1,263 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.hll.HllUtil.LG_AUX_ARR_INTS;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.extractCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.extractEmptyFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractHipAccum;
-import static org.apache.datasketches.hll.PreambleUtil.extractKxQ0;
-import static org.apache.datasketches.hll.PreambleUtil.extractKxQ1;
-import static org.apache.datasketches.hll.PreambleUtil.extractNumAtCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.extractOooFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractRebuildCurMinNumKxQFlag;
-import static org.apache.datasketches.hll.TgtHllType.HLL_4;
-import static org.apache.datasketches.hll.TgtHllType.HLL_6;
-
-import java.lang.foreign.MemorySegment;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-abstract class HllArray extends AbstractHllArray {
- boolean oooFlag = false; //Out-Of-Order Flag
- boolean rebuildCurMinNumKxQ = false;
- int curMin; //always zero for Hll6 and Hll8, only used by Hll4Array
- int numAtCurMin; //# of values at curMin. If curMin = 0, it is # of zeros
- double hipAccum;
- double kxq0;
- double kxq1;
- byte[] hllByteArr = null; //init by sub-classes
-
- /**
- * Standard constructor for new instance
- * @param lgConfigK the configured Lg K
- * @param tgtHllType the type of target HLL sketch
- */
- HllArray(final int lgConfigK, final TgtHllType tgtHllType) {
- super(lgConfigK, tgtHllType, CurMode.HLL);
- curMin = 0;
- numAtCurMin = 1 << lgConfigK;
- hipAccum = 0;
- kxq0 = 1 << lgConfigK;
- kxq1 = 0;
- }
-
- /**
- * Copy constructor
- * @param that another HllArray
- */
- HllArray(final HllArray that) {
- super(that.getLgConfigK(), that.getTgtHllType(), CurMode.HLL);
- oooFlag = that.isOutOfOrder();
- rebuildCurMinNumKxQ = that.isRebuildCurMinNumKxQFlag();
- curMin = that.getCurMin();
- numAtCurMin = that.getNumAtCurMin();
- hipAccum = that.getHipAccum();
- kxq0 = that.getKxQ0();
- kxq1 = that.getKxQ1();
- hllByteArr = that.hllByteArr.clone(); //that.hllByteArr should never be null.
- final AuxHashMap thatAuxMap = that.getAuxHashMap();
- if (thatAuxMap != null) {
- putAuxHashMap(thatAuxMap.copy(), false);
- } else {
- putAuxHashMap(null, false);
- }
- }
-
- static final HllArray newHeapHll(final int lgConfigK, final TgtHllType tgtHllType) {
- if (tgtHllType == HLL_4) { return new Hll4Array(lgConfigK); }
- if (tgtHllType == HLL_6) { return new Hll6Array(lgConfigK); }
- return new Hll8Array(lgConfigK);
- }
-
- @Override
- void addToHipAccum(final double delta) {
- hipAccum += delta;
- }
-
- @Override
- void decNumAtCurMin() {
- numAtCurMin--;
- }
-
- @Override
- int getCurMin() {
- return curMin;
- }
-
- @Override
- CurMode getCurMode() {
- return curMode;
- }
-
- @Override
- double getHipAccum() {
- return hipAccum;
- }
-
- @Override
- int getHllByteArrBytes() {
- return hllByteArr.length;
- }
-
- @Override
- double getKxQ0() {
- return kxq0;
- }
-
- @Override
- double getKxQ1() {
- return kxq1;
- }
-
- @Override
- int getLgConfigK() {
- return lgConfigK;
- }
-
- @Override
- MemorySegment getMemorySegment() {
- return null;
- }
-
- @Override
- AuxHashMap getNewAuxHashMap() {
- return new HeapAuxHashMap(LG_AUX_ARR_INTS[lgConfigK], lgConfigK);
- }
-
- @Override
- int getNumAtCurMin() {
- return numAtCurMin;
- }
-
- @Override
- boolean isCompact() {
- return false;
- }
-
- @Override
- boolean isEmpty() {
- return false; //because there should be no normal way to create an HllArray that is empty
- }
-
- @Override
- boolean hasMemorySegment() {
- return false;
- }
-
- @Override
- boolean isOffHeap() {
- return false;
- }
-
- @Override
- boolean isOutOfOrder() {
- return oooFlag;
- }
-
- @Override
- boolean isSameResource(final MemorySegment seg) {
- return false;
- }
-
- @Override
- boolean isRebuildCurMinNumKxQFlag() {
- return rebuildCurMinNumKxQ;
- }
-
- @Override
- void putAuxHashMap(final AuxHashMap auxHashMap, final boolean compact) {
- this.auxHashMap = auxHashMap;
- }
-
- @Override
- void putCurMin(final int curMin) {
- this.curMin = curMin;
- }
-
- @Override
- void putEmptyFlag(final boolean empty) { /* meaningless for HllArray classes */ }
-
- @Override
- void putHipAccum(final double value) {
- hipAccum = value;
- }
-
- @Override
- void putKxQ0(final double kxq0) {
- this.kxq0 = kxq0;
- }
-
- @Override
- void putKxQ1(final double kxq1) {
- this.kxq1 = kxq1;
- }
-
- @Override
- void putNumAtCurMin(final int numAtCurMin) {
- this.numAtCurMin = numAtCurMin;
- }
-
- @Override
- void putOutOfOrder(final boolean oooFlag) {
- if (oooFlag) { putHipAccum(0); }
- this.oooFlag = oooFlag;
- }
-
- @Override
- void putRebuildCurMinNumKxQFlag(final boolean rebuild) {
- rebuildCurMinNumKxQ = rebuild;
- }
-
- @Override
- HllSketchImpl reset() {
- return new CouponList(lgConfigK, tgtHllType, CurMode.LIST);
- }
-
- @Override //used by HLL6 and HLL8, overridden by HLL4
- byte[] toCompactByteArray() {
- return toUpdatableByteArray(); //indistinguishable for HLL6 and HLL8
- }
-
- @Override //used by HLL4, HLL6 and HLL8
- byte[] toUpdatableByteArray() {
- return ToByteArrayImpl.toHllByteArray(this, false);
- }
-
- //used by heapify by all Heap HLL
- static final void extractCommonHll(final MemorySegment srcSeg, final HllArray hllArray) {
- hllArray.putOutOfOrder(extractOooFlag(srcSeg));
- hllArray.putEmptyFlag(extractEmptyFlag(srcSeg));
- hllArray.putCurMin(extractCurMin(srcSeg));
- hllArray.putHipAccum(extractHipAccum(srcSeg));
- hllArray.putKxQ0(extractKxQ0(srcSeg));
- hllArray.putKxQ1(extractKxQ1(srcSeg));
- hllArray.putNumAtCurMin(extractNumAtCurMin(srcSeg));
- hllArray.putRebuildCurMinNumKxQFlag(extractRebuildCurMinNumKxQFlag(srcSeg));
-
- //load Hll array
- MemorySegment.copy(srcSeg, JAVA_BYTE, HLL_BYTE_ARR_START, hllArray.hllByteArr, 0, hllArray.hllByteArr.length);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HllEstimators.java b/src/main/java/org/apache/datasketches/hll/HllEstimators.java
deleted file mode 100644
index 21bbd4249..000000000
--- a/src/main/java/org/apache/datasketches/hll/HllEstimators.java
+++ /dev/null
@@ -1,156 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.hll.HllUtil.MIN_LOG_K;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class HllEstimators {
-
- //HLL UPPER AND LOWER BOUNDS
-
- /*
- * The upper and lower bounds are not symmetric and thus are treated slightly differently.
- * For the lower bound, when the unique count is <= k, LB >= numNonZeros, where
- * numNonZeros = k - numAtCurMin AND curMin == 0.
- *
- * For HLL6 and HLL8, curMin is always 0 and numAtCurMin is initialized to k and is decremented
- * down for each valid update until it reaches 0, where it stays. Thus, for these two
- * isomorphs, when numAtCurMin = 0, means the true curMin is > 0 and the unique count must be
- * greater than k.
- *
- * HLL4 always maintains both curMin and numAtCurMin dynamically. Nonetheless, the rules for
- * the very small values <= k where curMin = 0 still apply.
- */
-
- static double hllLowerBound(final AbstractHllArray absHllArr, final int numStdDev) {
- final int lgConfigK = absHllArr.lgConfigK;
- final int configK = 1 << lgConfigK;
- final double numNonZeros =
- (absHllArr.getCurMin() == 0) ? configK - absHllArr.getNumAtCurMin() : configK;
- final double estimate = absHllArr.getEstimate();
- final boolean oooFlag = absHllArr.isOutOfOrder();
- final double relErr = BaseHllSketch.getRelErr(false, oooFlag, lgConfigK, numStdDev);
- return Math.max(estimate / (1.0 + relErr), numNonZeros);
- }
-
- static double hllUpperBound(final AbstractHllArray absHllArr, final int numStdDev) {
- final int lgConfigK = absHllArr.lgConfigK;
- final double estimate = absHllArr.getEstimate();
- final boolean oooFlag = absHllArr.isOutOfOrder();
- final double relErr = BaseHllSketch.getRelErr(true, oooFlag, lgConfigK, numStdDev);
- return estimate / (1.0 - relErr);
- }
-
- //THE HLL COMPOSITE ESTIMATOR
-
- /**
- * This is the (non-HIP) estimator.
- * It is called "composite" because multiple estimators are pasted together.
- * @param absHllArr an instance of the AbstractHllArray class.
- * @return the composite estimate
- */
- //In C: again-two-registers.c hhb_get_composite_estimate L1489
- static double hllCompositeEstimate(final AbstractHllArray absHllArr) {
- final int lgConfigK = absHllArr.getLgConfigK();
- final double rawEst = getHllRawEstimate(lgConfigK, absHllArr.getKxQ0() + absHllArr.getKxQ1());
-
- final double[] xArr = CompositeInterpolationXTable.xArrs[lgConfigK - MIN_LOG_K];
- final double yStride = CompositeInterpolationXTable.yStrides[lgConfigK - MIN_LOG_K];
- final int xArrLen = xArr.length;
-
- if (rawEst < xArr[0]) { return 0; }
-
- final int xArrLenM1 = xArrLen - 1;
-
- if (rawEst > xArr[xArrLenM1]) {
- final double finalY = yStride * (xArrLenM1);
- final double factor = finalY / xArr[xArrLenM1];
- return rawEst * factor;
- }
-
- final double adjEst =
- CubicInterpolation.usingXArrAndYStride(xArr, yStride, rawEst);
-
- // We need to completely avoid the linear_counting estimator if it might have a crazy value.
- // Empirical evidence suggests that the threshold 3*k will keep us safe if 2^4 <= k <= 2^21.
-
- if (adjEst > (3 << lgConfigK)) { return adjEst; }
- //Alternate call
- //if ((adjEst > (3 << lgConfigK)) || ((curMin != 0) || (numAtCurMin == 0)) ) { return adjEst; }
-
- final double linEst =
- getHllBitMapEstimate(lgConfigK, absHllArr.getCurMin(), absHllArr.getNumAtCurMin());
-
- // Bias is created when the value of an estimator is compared with a threshold to decide whether
- // to use that estimator or a different one.
- // We conjecture that less bias is created when the average of the two estimators
- // is compared with the threshold. Empirical measurements support this conjecture.
-
- final double avgEst = (adjEst + linEst) / 2.0;
-
- // The following constants comes from empirical measurements of the crossover point
- // between the average error of the linear estimator and the adjusted HLL estimator
- double crossOver = 0.64;
- if (lgConfigK == 4) { crossOver = 0.718; }
- else if (lgConfigK == 5) { crossOver = 0.672; }
-
- return (avgEst > (crossOver * (1 << lgConfigK))) ? adjEst : linEst;
- }
-
- /**
- * Estimator when N is small, roughly less than k log(k).
- * Refer to Wikipedia: Coupon Collector Problem
- * @param lgConfigK the current configured lgK of the sketch
- * @param curMin the current minimum value of the HLL window
- * @param numAtCurMin the current number of slots with the value curMin
- * @return the very low range estimate
- */
- //In C: again-two-registers.c hhb_get_improved_linear_counting_estimate L1274
- private static double getHllBitMapEstimate(
- final int lgConfigK, final int curMin, final int numAtCurMin) {
- final int configK = 1 << lgConfigK;
- final int numUnhitBuckets = (curMin == 0) ? numAtCurMin : 0;
-
- //This will eventually go away.
- if (numUnhitBuckets == 0) {
- return configK * Math.log(configK / 0.5);
- }
-
- final int numHitBuckets = configK - numUnhitBuckets;
- return HarmonicNumbers.getBitMapEstimate(configK, numHitBuckets);
- }
-
- //In C: again-two-registers.c hhb_get_raw_estimate L1167
- //This algorithm is from Flajolet's, et al, 2007 HLL paper, Fig 3.
- private static double getHllRawEstimate(final int lgConfigK, final double kxqSum) {
- final int configK = 1 << lgConfigK;
- final double correctionFactor;
- if (lgConfigK == 4) { correctionFactor = 0.673; }
- else if (lgConfigK == 5) { correctionFactor = 0.697; }
- else if (lgConfigK == 6) { correctionFactor = 0.709; }
- else { correctionFactor = 0.7213 / (1.0 + (1.079 / configK)); }
- return (correctionFactor * configK * configK) / kxqSum;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HllPairIterator.java b/src/main/java/org/apache/datasketches/hll/HllPairIterator.java
deleted file mode 100644
index 6b9f2f078..000000000
--- a/src/main/java/org/apache/datasketches/hll/HllPairIterator.java
+++ /dev/null
@@ -1,100 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.pair;
-
-/**
- * Iterates over an on-heap HLL byte array producing pairs of index, value.
- *
- * @author Lee Rhodes
- */
-abstract class HllPairIterator extends PairIterator {
- final int lengthPairs;
- int index;
- int value;
-
- //Used by Direct<4,6,8>Array, Heap<4,6,8>Array
- HllPairIterator(final int lengthPairs) {
- this.lengthPairs = lengthPairs;
- index = - 1;
- }
-
- @Override
- public String getHeader() {
- return String.format("%10s%6s", "Slot", "Value");
- }
-
- @Override
- public int getIndex() {
- return index;
- }
-
- @Override
- public int getKey() {
- return index;
- }
-
- @Override
- public int getPair() {
- return pair(index, value);
- }
-
- @Override
- public int getSlot() {
- return index;
- }
-
- @Override
- public String getString() {
- final int slot = getSlot();
- final int value = getValue();
- return String.format("%10d%6d", slot, value);
- }
-
- @Override
- public int getValue() {
- return value;
- }
-
- @Override
- public boolean nextAll() {
- if (++index < lengthPairs) {
- value = value();
- return true;
- }
- return false;
- }
-
- @Override
- public boolean nextValid() {
- while (++index < lengthPairs) {
- value = value();
- if (value != EMPTY) {
- return true;
- }
- }
- return false;
- }
-
- abstract int value();
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HllSketch.java b/src/main/java/org/apache/datasketches/hll/HllSketch.java
deleted file mode 100644
index 0ff0c1e97..000000000
--- a/src/main/java/org/apache/datasketches/hll/HllSketch.java
+++ /dev/null
@@ -1,566 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.common.Util.checkBounds;
-import static org.apache.datasketches.common.Util.clear;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.KEY_BITS_26;
-import static org.apache.datasketches.hll.HllUtil.LG_AUX_ARR_INTS;
-import static org.apache.datasketches.hll.HllUtil.checkPreamble;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.extractCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.extractLgK;
-import static org.apache.datasketches.hll.PreambleUtil.extractTgtHllType;
-
-import java.lang.foreign.MemorySegment;
-import java.util.Objects;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * The HllSketch is actually a collection of compact implementations of Phillipe Flajolet’s HyperLogLog (HLL)
- * sketch but with significantly improved error behavior and excellent speed performance.
- *
- * If the use case for sketching is primarily counting uniques and merging, the HLL sketch is the 2nd highest
- * performing in terms of accuracy for storage space consumed in the DataSketches library
- * (the new CPC sketch developed by Kevin J. Lang now beats HLL in terms of accuracy / space).
- * For large counts, HLL sketches can be 2 to 8 times smaller for the same accuracy than the DataSketches Theta
- * Sketches when serialized, but the Theta sketches can do set intersections and differences while HLL and CPC cannot.
- * The CPC sketch and HLL share similar use cases, but the CPC sketch is about 30 to 40% smaller than the HLL sketch
- * when serialized and larger than the HLL when active in a MemorySegment. Choose your weapons! A new HLL sketch is created with a simple constructor: All three different sketch types are targets in that the sketches start out in a warm-up mode that is small in
- * size and gradually grows as needed until the full HLL array is allocated. The HLL_4, HLL_6 and HLL_8 represent
- * different levels of compression of the final HLL array where the 4, 6 and 8 refer to the number of bits each
- * bucket of the HLL array is compressed down to.
- * The HLL_4 is the most compressed but generally slower than the other two, especially during union operations. All three types share the same API. Updating the HllSketch is very simple: Each of the presented integers above are first hashed into 128-bit hash values that are used by the sketch
- * HLL algorithm, so the above loop is essentially equivalent to using a random number generator initialized with a
- * seed so that the sequence is deterministic and random. Obtaining the cardinality results from the sketch is also simple: Which produces a console output something like this: The given dstSeg is checked for the required capacity as determined by
- * {@link #getMaxUpdatableSerializationBytes(int, TgtHllType)}.
- * @param lgConfigK The Log2 of K for the target HLL sketch. This value must be
- * between 4 and 21 inclusively.
- * @param tgtHllType the desired HLL type.
- * @param dstSeg the destination MemorySegment for the sketch.
- */
- public HllSketch(final int lgConfigK, final TgtHllType tgtHllType, final MemorySegment dstSeg) {
- Objects.requireNonNull(dstSeg, "Destination MemorySegment must not be null");
- final long minBytes = getMaxUpdatableSerializationBytes(lgConfigK, tgtHllType);
- final long capBytes = dstSeg.byteSize();
- HllUtil.checkSegSize(minBytes, capBytes);
- clear(dstSeg, 0, minBytes);
- hllSketchImpl = DirectCouponList.newInstance(lgConfigK, tgtHllType, dstSeg);
- }
-
- /**
- * Copy constructor used by copy().
- * @param that another HllSketch
- */
- HllSketch(final HllSketch that) {
- hllSketchImpl = that.hllSketchImpl.copy();
- }
-
- /**
- * Special constructor used by copyAs, heapify
- * @param that another HllSketchImpl, which must already be a copy
- */
- HllSketch(final HllSketchImpl that) {
- hllSketchImpl = that;
- }
-
- /**
- * Heapify the given byte array, which must be a valid HllSketch image and may have data.
- * @param byteArray the given byte array. This byteArray is not modified and is not retained
- * by the on-heap sketch.
- * @return an HllSketch on the java heap.
- */
- public static final HllSketch heapify(final byte[] byteArray) {
- return heapify(MemorySegment.ofArray(byteArray));
- }
-
- /**
- * Heapify the given MemorySegment, which must be a valid HllSketch image and may have data.
- * @param srcSeg the given MemorySegment, which is read-only.
- * @return an HllSketch on the java heap.
- */
- public static final HllSketch heapify(final MemorySegment srcSeg) {
- return heapify(srcSeg, true);
- }
-
- //used by HllUnion and above
- static final HllSketch heapify(final MemorySegment srcSeg, final boolean checkRebuild) {
- Objects.requireNonNull(srcSeg, "Source MemorySegment must not be null");
- checkBounds(0, 8, srcSeg.byteSize()); //need min 8 bytes
- final CurMode curMode = checkPreamble(srcSeg);
- final HllSketch heapSketch;
- if (curMode == CurMode.HLL) {
- final TgtHllType tgtHllType = extractTgtHllType(srcSeg);
- if (tgtHllType == TgtHllType.HLL_4) {
- heapSketch = new HllSketch(Hll4Array.heapify(srcSeg));
- } else if (tgtHllType == TgtHllType.HLL_6) {
- heapSketch = new HllSketch(Hll6Array.heapify(srcSeg));
- } else { //Hll_8
- heapSketch = new HllSketch(Hll8Array.heapify(srcSeg));
- if (checkRebuild) {
- HllUnion.checkRebuildCurMinNumKxQ(heapSketch);
- }
- }
- } else if (curMode == CurMode.LIST) {
- heapSketch = new HllSketch(CouponList.heapifyList(srcSeg));
- } else {
- heapSketch = new HllSketch(CouponHashSet.heapifySet(srcSeg));
- }
- return heapSketch;
- }
-
- /**
- * Wraps the given MemorySegment, which must be a image of a valid updatable sketch,
- * and may have data. What remains on the java heap is a
- * thin wrapper object that reads and writes to the given MemorySegment, which, depending on
- * how the user configures the MemorySegment, may actually reside on the Java heap or off-heap.
- *
- * The given dstSeg is checked for the required capacity as determined by
- * {@link #getMaxUpdatableSerializationBytes(int, TgtHllType)}.
- * @param srcWseg an writable image of a valid source sketch with data.
- * @return an HllSketch where the sketch data is in the given srcWseg.
- */
- public static final HllSketch writableWrap(final MemorySegment srcWseg) {
- if (srcWseg.isReadOnly()) { return wrap(srcWseg); }
- return writableWrap(srcWseg, true);
- }
-
- //used by HllUnion and above
- static final HllSketch writableWrap( final MemorySegment srcWseg, final boolean checkRebuild) {
- Objects.requireNonNull(srcWseg, "Source MemorySegment must not be null");
- checkBounds(0, 8, srcWseg.byteSize()); //need min 8 bytes
- if (extractCompactFlag(srcWseg)) {
- throw new SketchesArgumentException(
- "Cannot perform a writableWrap of a sketch image that is in compact form. "
- + "Compact sketches are by definition immutable.");
- }
- final int lgConfigK = extractLgK(srcWseg);
- final TgtHllType tgtHllType = extractTgtHllType(srcWseg);
- final long minBytes = getMaxUpdatableSerializationBytes(lgConfigK, tgtHllType);
- final long capBytes = srcWseg.byteSize();
- HllUtil.checkSegSize(minBytes, capBytes);
- final CurMode curMode = checkPreamble(srcWseg);
- final HllSketch directSketch;
- if (curMode == CurMode.HLL) {
- if (tgtHllType == TgtHllType.HLL_4) {
- directSketch = new HllSketch(new DirectHll4Array(lgConfigK, srcWseg));
- } else if (tgtHllType == TgtHllType.HLL_6) {
- directSketch = new HllSketch(new DirectHll6Array(lgConfigK, srcWseg));
- } else { //Hll_8
- directSketch = new HllSketch(new DirectHll8Array(lgConfigK, srcWseg));
- if (checkRebuild) { //HllUnion only uses HLL_8, we allow non-finalized from a HllUnion call.
- HllUnion.checkRebuildCurMinNumKxQ(directSketch);
- }
- }
- } else if (curMode == CurMode.LIST) {
- directSketch =
- new HllSketch(new DirectCouponList(lgConfigK, tgtHllType, curMode, srcWseg));
- } else { //SET
- directSketch =
- new HllSketch(new DirectCouponHashSet(lgConfigK, tgtHllType, srcWseg));
- }
- return directSketch;
- }
-
- /**
- * Wraps the given read-only MemorySegment that must be a image of a valid sketch,
- * which may be in compact or updatable form, and should have data. Any attempt to update the
- * given source MemorySegment will throw an exception.
- * @param srcSeg a read-only image of a valid source sketch.
- * @return an HllSketch, where the read-only data of the sketch is in the given srcSeg.
- *
- */
- public static final HllSketch wrap(final MemorySegment srcSeg) { //read only
- Objects.requireNonNull(srcSeg, "Source MemorySegment must not be null");
- checkBounds(0, 8, srcSeg.byteSize()); //need min 8 bytes
- final int lgConfigK = extractLgK(srcSeg);
- final TgtHllType tgtHllType = extractTgtHllType(srcSeg);
-
- final CurMode curMode = checkPreamble(srcSeg);
- final HllSketch directSketch;
- if (curMode == CurMode.HLL) {
- if (tgtHllType == TgtHllType.HLL_4) {
- directSketch = new HllSketch(new DirectHll4Array(lgConfigK, srcSeg, true));
- } else if (tgtHllType == TgtHllType.HLL_6) {
- directSketch = new HllSketch(new DirectHll6Array(lgConfigK, srcSeg, true));
- } else { //Hll_8
- directSketch = new HllSketch(new DirectHll8Array(lgConfigK, srcSeg, true));
- //rebuild if srcSeg came from a HllUnion and was not finalized, rather than throw exception.
- HllUnion.checkRebuildCurMinNumKxQ(directSketch);
- }
- } else if (curMode == CurMode.LIST) {
- directSketch =
- new HllSketch(new DirectCouponList(lgConfigK, tgtHllType, curMode, srcSeg, true));
- } else { //SET
- directSketch =
- new HllSketch(new DirectCouponHashSet(lgConfigK, tgtHllType, srcSeg, true));
- }
- return directSketch;
- }
-
- /**
- * Return a copy of this sketch onto the Java heap.
- * @return a copy of this sketch onto the Java heap.
- */
- public HllSketch copy() {
- return new HllSketch(this);
- }
-
- /**
- * Return a deep copy of this sketch onto the Java heap with the specified TgtHllType.
- * @param tgtHllType the TgtHllType enum
- * @return a deep copy of this sketch with the specified TgtHllType.
- */
- public HllSketch copyAs(final TgtHllType tgtHllType) {
- return new HllSketch(hllSketchImpl.copyAs(tgtHllType));
- }
-
- @Override
- public double getCompositeEstimate() {
- return hllSketchImpl.getCompositeEstimate();
- }
-
- @Override
- public double getEstimate() {
- return hllSketchImpl.getEstimate();
- }
-
- double getHipEstimate() {
- return hllSketchImpl.getHipEstimate();
- }
-
- @Override
- public int getLgConfigK() {
- return hllSketchImpl.getLgConfigK();
- }
-
- @Override
- public int getCompactSerializationBytes() {
- return hllSketchImpl.getCompactSerializationBytes();
- }
-
- @Override
- public double getLowerBound(final int numStdDev) {
- return hllSketchImpl.getLowerBound(numStdDev);
- }
-
- /**
- * Returns the maximum size in bytes that this sketch can grow to given lgConfigK.
- * However, for the HLL_4 sketch type, this value can be exceeded in extremely rare cases.
- * If exceeded, it will be larger by only a few percent.
- *
- * @param lgConfigK The Log2 of K for the target HLL sketch. This value must be
- * between 4 and 21 inclusively.
- * @param tgtHllType the desired Hll type
- * @return the maximum size in bytes that this sketch can grow to.
- */
- public static final int getMaxUpdatableSerializationBytes(final int lgConfigK,
- final TgtHllType tgtHllType) {
- final int arrBytes;
- if (tgtHllType == TgtHllType.HLL_4) {
- final int auxBytes = 4 << LG_AUX_ARR_INTS[lgConfigK];
- arrBytes = AbstractHllArray.hll4ArrBytes(lgConfigK) + auxBytes;
- }
- else if (tgtHllType == TgtHllType.HLL_6) {
- arrBytes = AbstractHllArray.hll6ArrBytes(lgConfigK);
- }
- else { //HLL_8
- arrBytes = AbstractHllArray.hll8ArrBytes(lgConfigK);
- }
- return HLL_BYTE_ARR_START + arrBytes;
- }
-
- MemorySegment getMemorySegment() {
- return hllSketchImpl.getMemorySegment();
- }
-
- @Override
- public TgtHllType getTgtHllType() {
- return hllSketchImpl.getTgtHllType();
- }
-
- @Override
- public int getUpdatableSerializationBytes() {
- return hllSketchImpl.getUpdatableSerializationBytes();
- }
-
- @Override
- public double getUpperBound(final int numStdDev) {
- return hllSketchImpl.getUpperBound(numStdDev);
- }
-
- @Override
- public boolean isCompact() {
- return hllSketchImpl.isCompact();
- }
-
- @Override
- public boolean isEmpty() {
- return hllSketchImpl.isEmpty();
- }
-
- @Override
- public boolean hasMemorySegment() {
- return hllSketchImpl.hasMemorySegment();
- }
-
- @Override
- public boolean isOffHeap() {
- return hllSketchImpl.isOffHeap();
- }
-
- @Override
- boolean isOutOfOrder() {
- return hllSketchImpl.isOutOfOrder();
- }
-
- @Override
- public boolean isSameResource(final MemorySegment seg) {
- return hllSketchImpl.isSameResource(seg);
- }
-
- void mergeTo(final HllSketch that) {
- hllSketchImpl.mergeTo(that);
- }
-
- HllSketch putOutOfOrderFlag(final boolean oooFlag) {
- hllSketchImpl.putOutOfOrder(oooFlag);
- return this;
- }
-
- @Override
- public void reset() {
- hllSketchImpl = hllSketchImpl.reset();
- }
-
- @Override
- public byte[] toCompactByteArray() {
- return hllSketchImpl.toCompactByteArray();
- }
-
- @Override
- public byte[] toUpdatableByteArray() {
- return hllSketchImpl.toUpdatableByteArray();
- }
-
- @Override
- public String toString(final boolean summary, final boolean detail, final boolean auxDetail,
- final boolean all) {
- final StringBuilder sb = new StringBuilder();
- if (summary) {
- sb.append("### HLL SKETCH SUMMARY: ").append(LS);
- sb.append(" Log Config K : ").append(getLgConfigK()).append(LS);
- sb.append(" Hll Target : ").append(getTgtHllType()).append(LS);
- sb.append(" Current Mode : ").append(getCurMode()).append(LS);
- sb.append(" MemorySegment : ").append(hasMemorySegment()).append(LS);
- sb.append(" LB : ").append(getLowerBound(1)).append(LS);
- sb.append(" Estimate : ").append(getEstimate()).append(LS);
- sb.append(" UB : ").append(getUpperBound(1)).append(LS);
- sb.append(" OutOfOrder Flag: ").append(isOutOfOrder()).append(LS);
- if (getCurMode() == CurMode.HLL) {
- final AbstractHllArray absHll = (AbstractHllArray) hllSketchImpl;
- sb.append(" CurMin : ").append(absHll.getCurMin()).append(LS);
- sb.append(" NumAtCurMin : ").append(absHll.getNumAtCurMin()).append(LS);
- sb.append(" HipAccum : ").append(absHll.getHipAccum()).append(LS);
- sb.append(" KxQ0 : ").append(absHll.getKxQ0()).append(LS);
- sb.append(" KxQ1 : ").append(absHll.getKxQ1()).append(LS);
- sb.append(" Rebuild KxQ Flg: ").append(absHll.isRebuildCurMinNumKxQFlag()).append(LS);
- } else {
- sb.append(" Coupon Count : ")
- .append(((AbstractCoupons)hllSketchImpl).getCouponCount()).append(LS);
- }
- }
- if (detail) {
- sb.append("### HLL SKETCH DATA DETAIL: ").append(LS);
- final PairIterator pitr = iterator();
- sb.append(pitr.getHeader()).append(LS);
- if (all) {
- while (pitr.nextAll()) {
- sb.append(pitr.getString()).append(LS);
- }
- } else {
- while (pitr.nextValid()) {
- sb.append(pitr.getString()).append(LS);
- }
- }
- }
- if (auxDetail && ((getCurMode() == CurMode.HLL) && (getTgtHllType() == TgtHllType.HLL_4))) {
- final AbstractHllArray absHll = (AbstractHllArray) hllSketchImpl;
- final PairIterator auxItr = absHll.getAuxIterator();
- if (auxItr != null) {
- sb.append("### HLL SKETCH AUX DETAIL: ").append(LS);
- sb.append(auxItr.getHeader()).append(LS);
- if (all) {
- while (auxItr.nextAll()) {
- sb.append(auxItr.getString()).append(LS);
- }
- } else {
- while (auxItr.nextValid()) {
- sb.append(auxItr.getString()).append(LS);
- }
- }
- }
- }
- return sb.toString();
- }
-
- /**
- * Returns a human readable string of the preamble of a byte array image of an HllSketch.
- * @param byteArr the given byte array
- * @return a human readable string of the preamble of a byte array image of an HllSketch.
- */
- public static String toString(final byte[] byteArr) {
- return PreambleUtil.toString(byteArr);
- }
-
- /**
- * Returns a human readable string of the preamble of a MemorySegment image of an HllSketch.
- * @param seg the given MemorySegment object
- * @return a human readable string of the preamble of a MemorySegment image of an HllSketch.
- */
- public static String toString(final MemorySegment seg) {
- return PreambleUtil.toString(seg);
- }
-
- //restricted methods
-
- /**
- * Returns a PairIterator over the key, value pairs of the HLL array.
- * @return a PairIterator over the key, value pairs of the HLL array.
- */
- PairIterator iterator() {
- return hllSketchImpl.iterator();
- }
-
- @Override
- CurMode getCurMode() {
- return hllSketchImpl.getCurMode();
- }
-
- @Override
- void couponUpdate(final int coupon) {
- if ((coupon >>> KEY_BITS_26 ) == EMPTY) { return; }
- hllSketchImpl = hllSketchImpl.couponUpdate(coupon);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HllSketchImpl.java b/src/main/java/org/apache/datasketches/hll/HllSketchImpl.java
deleted file mode 100644
index bcc74d255..000000000
--- a/src/main/java/org/apache/datasketches/hll/HllSketchImpl.java
+++ /dev/null
@@ -1,118 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import java.lang.foreign.MemorySegment;
-
-/**
- * The Abstract HllSketch implementation
- *
- * @author Lee Rhodes
- */
-abstract class HllSketchImpl {
- final int lgConfigK;
- final TgtHllType tgtHllType;
- final CurMode curMode;
-
- HllSketchImpl(final int lgConfigK, final TgtHllType tgtHllType, final CurMode curMode) {
- this.lgConfigK = lgConfigK;
- this.tgtHllType = tgtHllType;
- this.curMode = curMode;
- }
-
- /**
- * Returns a copy of this sketch on the Heap.
- * The LgConfigK, TgtHllType and CurMode are not changed.
- * @return Returns a copy of this sketch on the Heap.
- */
- abstract HllSketchImpl copy();
-
- /**
- * Returns a copy of this sketch on the Heap with the given TgtHllType.
- * The LgConfigK and CurMode are not changed.
- * @return Returns a copy of this sketch on the Heap with the given TgtHllType.
- */
- abstract HllSketchImpl copyAs(TgtHllType tgtHllType);
-
- abstract HllSketchImpl couponUpdate(int coupon);
-
- abstract int getCompactSerializationBytes();
-
- abstract double getCompositeEstimate();
-
- CurMode getCurMode() {
- return curMode;
- }
-
- abstract double getEstimate();
-
- abstract double getHipEstimate();
-
- int getLgConfigK() {
- return lgConfigK;
- }
-
- abstract double getLowerBound(int numStdDev);
-
- abstract int getSegDataStart();
-
- abstract MemorySegment getMemorySegment();
-
- abstract int getPreInts();
-
- TgtHllType getTgtHllType() {
- return tgtHllType;
- }
-
- abstract int getUpdatableSerializationBytes();
-
- abstract double getUpperBound(int numStdDev);
-
- abstract boolean isCompact();
-
- abstract boolean isEmpty();
-
- abstract boolean hasMemorySegment();
-
- abstract boolean isOffHeap();
-
- abstract boolean isOutOfOrder();
-
- abstract boolean isRebuildCurMinNumKxQFlag();
-
- abstract boolean isSameResource(MemorySegment seg);
-
- abstract PairIterator iterator();
-
- abstract void mergeTo(HllSketch that);
-
- abstract void putEmptyFlag(boolean empty);
-
- abstract void putOutOfOrder(boolean oooFlag);
-
- abstract void putRebuildCurMinNumKxQFlag(boolean rebuild);
-
- abstract HllSketchImpl reset();
-
- abstract byte[] toCompactByteArray();
-
- abstract byte[] toUpdatableByteArray();
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HllUnion.java b/src/main/java/org/apache/datasketches/hll/HllUnion.java
deleted file mode 100644
index 2738d68d7..000000000
--- a/src/main/java/org/apache/datasketches/hll/HllUnion.java
+++ /dev/null
@@ -1,834 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.common.Util.invPow2;
-import static org.apache.datasketches.hll.HllUtil.AUX_TOKEN;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-import static org.apache.datasketches.hll.HllUtil.loNibbleMask;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.extractTgtHllType;
-import static org.apache.datasketches.hll.TgtHllType.HLL_4;
-import static org.apache.datasketches.hll.TgtHllType.HLL_8;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * This performs union operations for all HllSketches. This HllUnion operator can be configured to be
- * on or off heap. The source sketch given to this HllUnion using the {@link #update(HllSketch)} can
- * be configured with any precision value lgConfigK (from 4 to 21), any TgtHllType
- * (HLL_4, HLL_6, HLL_8), and either on or off-heap; and it can be in either of the sparse modes
- * (LIST or SET), or the dense mode (HLL).
- *
- * Although the API for this HllUnion operator parallels many of the methods of the
- * HllSketch, the behavior of the HllUnion operator has some fundamental differences. First, this HllUnion operator is configured with a lgMaxK instead of the normal
- * lgConfigK. Generally, this HllUnion operator will inherit the lowest lgConfigK
- * less than lgMaxK that it has seen. However, the lgConfigK of incoming sketches that
- * are still in sparse are ignored. The lgMaxK provides the user the ability to specify the
- * largest maximum size for the HllUnion operation.
- *
- * Second, the user cannot specify the {@link TgtHllType} as an input parameter to the HllUnion.
- * Instead, it is specified for the sketch returned with {@link #getResult(TgtHllType)}.
- *
- * The following graph illustrates the HLL Merge speed. For a complete example of using the HllUnion operator
- * see HllUnion Example. The given dstSeg is checked for the required capacity as determined by
- * {@link HllSketch#getMaxUpdatableSerializationBytes(int, TgtHllType)}.
- * @param lgMaxK the desired maximum log-base-2 of K. This value must be
- * between 4 and 21 inclusively.
- * @param dstWseg the destination writable MemorySegment for the sketch.
- */
- public HllUnion(final int lgMaxK, final MemorySegment dstWseg) {
- this.lgMaxK = HllUtil.checkLgK(lgMaxK);
- gadget = new HllSketch(lgMaxK, HLL_8, dstWseg);
- }
-
- //used only by writableWrap
- private HllUnion(final HllSketch sketch) {
- lgMaxK = sketch.getLgConfigK();
- gadget = sketch;
- }
-
- /**
- * Construct a HllUnion operator populated with the given byte array image of an HllSketch.
- * @param byteArray the given byte array
- * @return a HllUnion operator populated with the given byte array image of an HllSketch.
- */
- public static final HllUnion heapify(final byte[] byteArray) {
- return heapify(MemorySegment.ofArray(byteArray));
- }
-
- /**
- * Construct a HllUnion operator populated with the given MemorySegment image of an HllSketch.
- * @param seg the given MemorySegment
- * @return a HllUnion operator populated with the given MemorySegment image of an HllSketch.
- */
- public static final HllUnion heapify(final MemorySegment seg) {
- final int lgK = HllUtil.checkLgK(seg.get(JAVA_BYTE, PreambleUtil.LG_K_BYTE));
- final HllSketch sk = HllSketch.heapify(seg, false); //allows non-finalized image
- final HllUnion union = new HllUnion(lgK);
- union.update(sk);
- return union;
- }
-
- /**
- * Wraps the given MemorySegment, which must be a image of a valid updatable HLL_8 sketch,
- * and may have data. What remains on the java heap is a
- * thin wrapper object that reads and writes to the given MemorySegment, which, depending on
- * how the user configures the MemorySegment, may actually reside on the Java heap or off-heap.
- *
- * The given dstSeg is checked for the required capacity as determined by
- * {@link HllSketch#getMaxUpdatableSerializationBytes(int, TgtHllType)}, and for the correct type.
- * @param srcWseg an writable image of a valid sketch with data.
- * @return a HllUnion operator where the sketch data is in the given dstSeg.
- */
- public static final HllUnion writableWrap(final MemorySegment srcWseg) {
- final TgtHllType tgtHllType = extractTgtHllType(srcWseg);
- if (tgtHllType != TgtHllType.HLL_8) {
- throw new SketchesArgumentException(
- "HllUnion can only wrap writable HLL_8 sketches that were the Gadget of a HllUnion.");
- }
- //allows writableWrap of non-finalized image
- return new HllUnion(HllSketch.writableWrap(srcWseg, false));
- }
-
- @Override
- public double getCompositeEstimate() {
- checkRebuildCurMinNumKxQ(gadget);
- return gadget.hllSketchImpl.getCompositeEstimate();
- }
-
- @Override
- CurMode getCurMode() {
- return gadget.getCurMode();
- }
-
- @Override
- public int getCompactSerializationBytes() {
- return gadget.getCompactSerializationBytes();
- }
-
- @Override
- public double getEstimate() {
- checkRebuildCurMinNumKxQ(gadget);
- return gadget.getEstimate();
- }
-
- /**
- * Gets the effective lgConfigK for the HllUnion operator, which may be less than
- * lgMaxK.
- * @return the lgConfigK.
- */
- @Override
- public int getLgConfigK() {
- return gadget.getLgConfigK();
- }
-
- @Override
- public double getLowerBound(final int numStdDev) {
- checkRebuildCurMinNumKxQ(gadget);
- return gadget.getLowerBound(numStdDev);
- }
-
- /**
- * Returns the maximum size in bytes that this HllUnion operator can grow to given a lgK.
- *
- * @param lgK The maximum Log2 of K for this HllUnion operator. This value must be
- * between 4 and 21 inclusively.
- * @return the maximum size in bytes that this HllUnion operator can grow to.
- */
- public static int getMaxSerializationBytes(final int lgK) {
- return HllSketch.getMaxUpdatableSerializationBytes(lgK, TgtHllType.HLL_8);
- }
-
- /**
- * Return the result of this HllUnion operator as an HLL_4 sketch.
- * @return the result of this HllUnion operator as an HLL_4 sketch.
- */
- public HllSketch getResult() {
- return getResult(HllSketch.DEFAULT_HLL_TYPE);
- }
-
- /**
- * Return the result of this HllUnion operator with the specified {@link TgtHllType}
- * @param tgtHllType the TgtHllType enum
- * @return the result of this HllUnion operator with the specified TgtHllType
- */
- public HllSketch getResult(final TgtHllType tgtHllType) {
- checkRebuildCurMinNumKxQ(gadget);
- return gadget.copyAs(tgtHllType);
- }
-
- @Override
- public TgtHllType getTgtHllType() {
- return TgtHllType.HLL_8;
- }
-
- @Override
- public int getUpdatableSerializationBytes() {
- return gadget.getUpdatableSerializationBytes();
- }
-
- @Override
- public double getUpperBound(final int numStdDev) {
- checkRebuildCurMinNumKxQ(gadget);
- return gadget.getUpperBound(numStdDev);
- }
-
- @Override
- public boolean isCompact() {
- return gadget.isCompact();
- }
-
- @Override
- public boolean isEmpty() {
- return gadget.isEmpty();
- }
-
- @Override
- public boolean hasMemorySegment() {
- return gadget.hasMemorySegment();
- }
-
- @Override
- public boolean isOffHeap() {
- return gadget.isOffHeap();
- }
-
- @Override
- boolean isOutOfOrder() {
- return gadget.isOutOfOrder();
- }
-
- @Override
- public boolean isSameResource(final MemorySegment seg) {
- return gadget.isSameResource(seg);
- }
-
- boolean isRebuildCurMinNumKxQFlag() {
- return gadget.hllSketchImpl.isRebuildCurMinNumKxQFlag();
- }
-
- void putRebuildCurMinNumKxQFlag(final boolean rebuild) {
- gadget.hllSketchImpl.putRebuildCurMinNumKxQFlag(rebuild);
- }
-
- /**
- * Resets to empty and retains the current lgK, but does not change the configured value of
- * lgMaxK.
- */
- @Override
- public void reset() {
- gadget.reset();
- }
-
- /**
- * Gets the serialization of this HllUnion operator as a byte array in compact form, which is
- * designed to be heapified only. It is not directly updatable.
- * For the HllUnion operator, this is the serialization of the internal state of
- * the HllUnion operator as a sketch.
- * @return the serialization of this HllUnion operator as a byte array.
- */
- @Override
- public byte[] toCompactByteArray() {
- checkRebuildCurMinNumKxQ(gadget);
- return gadget.toCompactByteArray();
- }
-
- @Override
- public byte[] toUpdatableByteArray() {
- checkRebuildCurMinNumKxQ(gadget);
- return gadget.toUpdatableByteArray();
- }
-
- @Override
- public String toString(final boolean summary, final boolean hllDetail,
- final boolean auxDetail, final boolean all) {
- final HllSketch clone = gadget.copy();
- checkRebuildCurMinNumKxQ(clone);
- return clone.toString(summary, hllDetail, auxDetail, all);
- }
-
- /**
- * Update this HllUnion operator with the given sketch.
- * @param sketch the given sketch.
- */
- public void update(final HllSketch sketch) {
- gadget.hllSketchImpl = unionImpl(sketch, gadget, lgMaxK);
- }
-
- /**
- * Update this HllUnion operator with the given HllUnion.
- * @param union the given HllUnion.
- */
- public void update(final HllUnion union) {
- gadget.hllSketchImpl = unionImpl(union.gadget, gadget, lgMaxK);
- }
-
- @Override
- void couponUpdate(final int coupon) {
- if (coupon == EMPTY) { return; }
- gadget.hllSketchImpl = gadget.hllSketchImpl.couponUpdate(coupon);
- }
-
- // HllUnion operator logic
-
- /**
- * Union the given source and destination sketches. This static method examines the state of
- * the current internal gadget and the incoming sketch and determines the optimum way to
- * perform the union. This may involve swapping the merge order, downsampling, transforming,
- * and / or copying one of the arguments and may completely replace the internals of the HllUnion.
- *
- * If the HllUnion gadget is empty, the source sketch is effectively copied to the HllUnion gadget
- * after any required transformations.
- *
- * The direction of the merge is reversed if the HllUnion gadget is in LIST or SET mode, and the
- * source sketch is in HLL mode. This is done to maintain maximum accuracy of the union process.
- *
- * The source sketch is downsampled if the source LgK is larger than maxLgK and in HLL mode.
- *
- * The HllUnion gadget is downsampled if both source and HllUnion gadget are in HLL mode
- * and the source LgK less than the HllUnion gadget LgK.
- *
- * @param source the given incoming sketch, which cannot be modified.
- * @param gadget the given gadget sketch, which has a target of HLL_8 and holds the result.
- * @param lgMaxK the maximum value of log2 K for this union operation.
- * @return the union of the two sketches in the form of the internal HllSketchImpl, which is
- * always in HLL_8 form.
- */
- private static HllSketchImpl unionImpl(final HllSketch source, final HllSketch gadget,
- final int lgMaxK) {
- assert gadget.getTgtHllType() == HLL_8;
- if ((source == null) || source.isEmpty()) {
- return gadget.hllSketchImpl;
- }
-
- final CurMode srcMode = source.getCurMode();
- if (srcMode == CurMode.LIST ) {
- source.mergeTo(gadget);
- return gadget.hllSketchImpl;
- }
-
- final int srcLgK = source.getLgConfigK();
- final int gadgetLgK = gadget.getLgConfigK();
- final boolean srcHasSeg = source.hasMemorySegment();
- final boolean gdtHasSeg = gadget.hasMemorySegment();
- final boolean gdtEmpty = gadget.isEmpty();
-
- if (srcMode == CurMode.SET ) {
- if (gdtEmpty && (srcLgK == gadgetLgK) && (!srcHasSeg) && (!gdtHasSeg)) {
- gadget.hllSketchImpl = source.copyAs(HLL_8).hllSketchImpl;
- return gadget.hllSketchImpl;
- }
- source.mergeTo(gadget);
- return gadget.hllSketchImpl;
- }
-
- //Hereafter, the source is in HLL mode.
- final int bit0 = gdtHasSeg ? 1 : 0;
- final int bits1_2 = (gdtEmpty ? 3 : gadget.getCurMode().ordinal()) << 1;
- final int bit3 = (srcLgK < gadgetLgK) ? 8 : 0;
- final int bit4 = (srcLgK > lgMaxK) ? 16 : 0;
- final int sw = bit4 | bit3 | bits1_2 | bit0;
- HllSketchImpl hllSketchImpl = null; //never returned as null
-
- switch (sw) {
- case 0: //src <= max, src >= gdt, gdtLIST, gdtHeap
- case 8: //src <= max, src < gdt, gdtLIST, gdtHeap
- case 2: //src <= max, src >= gdt, gdtSET, gdtHeap
- case 10://src <= max, src < gdt, gdtSET, gdtHeap
- { //Action: copy src, reverse merge w/autofold, ooof=src
- final HllSketch srcHll8Heap = source.copyAs(HLL_8);
- gadget.mergeTo(srcHll8Heap); //merge gdt(Hll8,heap,list/set) -> src(Hll8,heap,hll)
- hllSketchImpl = srcHll8Heap.hllSketchImpl;
- break;
- }
- case 16://src > max, src >= gdt, gdtList, gdtHeap
- case 18://src > max, src >= gdt, gdtSet, gdtHeap
- { //Action: downsample src to MaxLgK, reverse merge w/autofold, ooof=src
- final HllSketch srcHll8Heap = downsample(source, lgMaxK);
- gadget.mergeTo(srcHll8Heap); //merge gdt(Hll8,heap,list/set) -> src(Hll8,heap,hll)
- hllSketchImpl = srcHll8Heap.hllSketchImpl;
- break;
- }
-
- case 1: //src <= max, src >= gdt, gdtLIST, gdtMemorySegment
- case 9: //src <= max, src < gdt, gdtLIST, gdtMemorySegment
- case 3: //src <= max, src >= gdt, gdtSET, gdtMemorySegment
- case 11://src <= max, src < gdt, gdtSET, gdtMemorySegment
- { //Action: copy src, reverse merge w/autofold, use gdt MemorySegment, ooof=src
- final HllSketch srcHll8Heap = source.copyAs(HLL_8);
- gadget.mergeTo(srcHll8Heap); //merge gdt(Hll8,seg,list/set) -> src(Hll8,heap,hll)
- hllSketchImpl = useGadgetMemorySegment(gadget, srcHll8Heap, false).hllSketchImpl;
- break;
- }
- case 17://src > max, src >= gdt, gdtList, gdtMemorySegment
- case 19://src > max, src >= gdt, gdtSet, gdtMemorySegment
- { //Action: downsample src to MaxLgK, reverse merge w/autofold, use gdt MemorySegment, ooof=src
- final HllSketch srcHll8Heap = downsample(source, lgMaxK);
- gadget.mergeTo(srcHll8Heap); //merge gdt(Hll8,seg,list/set) -> src(Hll8,heap,hll), autofold
- hllSketchImpl = useGadgetMemorySegment(gadget, srcHll8Heap, false).hllSketchImpl;
- break;
- }
-
- case 4: //src <= max, src >= gdt, gdtHLL, gdtHeap
- case 20://src > max, src >= gdt, gdtHLL, gdtHeap
- case 5: //src <= max, src >= gdt, gdtHLL, gdtMemorySegment
- case 21://src > max, src >= gdt, gdtHLL, gdtMemorySegment
- { //Action: forward HLL merge w/autofold, ooof=True
- //merge src(Hll4,6,8,heap/seg,Mode=HLL) -> gdt(Hll8,heap,Mode=HLL)
- mergeHlltoHLLmode(source, gadget, srcLgK, gadgetLgK, srcHasSeg, gdtHasSeg);
- hllSketchImpl = gadget.putOutOfOrderFlag(true).hllSketchImpl;
- break;
- }
- case 12://src <= max, src < gdt, gdtHLL, gdtHeap
- { //Action: downsample gdt to srcLgK, forward HLL merge w/autofold, ooof=True
- final HllSketch gdtHll8Heap = downsample(gadget, srcLgK);
- //merge src(Hll4,6,8;heap/seg,Mode=HLL) -> gdt(Hll8,heap,hll)
- mergeHlltoHLLmode(source, gdtHll8Heap, srcLgK, gadgetLgK, srcHasSeg, false);
- hllSketchImpl = gdtHll8Heap.putOutOfOrderFlag(true).hllSketchImpl;
- break;
- }
- case 13://src <= max, src < gdt, gdtHLL, gdtMemorySegment
- { //Action: downsample gdt to srcLgK, forward HLL merge w/autofold, use gdt MemorySegment, ooof=True
- final HllSketch gdtHll8Heap = downsample(gadget, srcLgK);
- //merge src(Hll4,6,8;heap/seg;Mode=HLL) -> gdt(Hll8,heap,Mode=HLL)
- mergeHlltoHLLmode(source, gdtHll8Heap, srcLgK, gadgetLgK, srcHasSeg, false);
- hllSketchImpl = useGadgetMemorySegment(gadget, gdtHll8Heap, true).hllSketchImpl;
- break;
- }
-
- case 6: //src <= max, src >= gdt, gdtEmpty, gdtHeap
- case 14://src <= max, src < gdt, gdtEmpty, gdtHeap
- { //Action: copy src, replace gdt, ooof=src
- final HllSketch srcHll8Heap = source.copyAs(HLL_8);
- hllSketchImpl = srcHll8Heap.hllSketchImpl;
- break;
- }
- case 22://src > max, src >= gdt, gdtEmpty, gdtHeap
- { //Action: downsample src to lgMaxK, replace gdt, ooof=src
- final HllSketch srcHll8Heap = downsample(source, lgMaxK);
- hllSketchImpl = srcHll8Heap.hllSketchImpl;
- break;
- }
-
- case 7: //src <= max, src >= gdt, gdtEmpty, gdtMemorySegment
- case 15://src <= max, src < gdt, gdtEmpty, gdtMemorySegment
- { //Action: copy src, use gdt MemorySegment, ooof=src
- final HllSketch srcHll8Heap = source.copyAs(HLL_8);
- hllSketchImpl = useGadgetMemorySegment(gadget, srcHll8Heap, false).hllSketchImpl;
- break;
- }
- case 23://src > max, src >= gdt, gdtEmpty, gdtMemorySegment, replace seg, downsample src, ooof=src
- { //Action: downsample src to lgMaxK, use gdt MemorySegment, ooof=src
- final HllSketch srcHll8Heap = downsample(source, lgMaxK);
- hllSketchImpl = useGadgetMemorySegment(gadget, srcHll8Heap, false).hllSketchImpl;
- break;
- }
- default: return gadget.hllSketchImpl; //not possible
- }
- return hllSketchImpl;
- }
-
- private static final HllSketch useGadgetMemorySegment(
- final HllSketch gadget, final HllSketch hll8Heap, final boolean setOooFlag) {
- final MemorySegment wseg = gadget.getMemorySegment(); //use the gdt wseg
- final byte[] byteArr = hll8Heap.toUpdatableByteArray(); //serialize srcCopy
- MemorySegment.copy(byteArr, 0, wseg, JAVA_BYTE, 0, byteArr.length); //replace old data with new
- return (setOooFlag)
- ? HllSketch.writableWrap(wseg, false).putOutOfOrderFlag(true) //wrap, set oooflag, return
- : HllSketch.writableWrap(wseg, false); //wrap & return
- }
-
- private static final void mergeHlltoHLLmode(final HllSketch src, final HllSketch tgt,
- final int srcLgK, final int tgtLgK, final boolean srcHasSeg, final boolean tgtHasSeg) {
- final int sw = (tgtHasSeg ? 1 : 0) | (srcHasSeg ? 2 : 0)
- | ((srcLgK > tgtLgK) ? 4 : 0) | ((src.getTgtHllType() != HLL_8) ? 8 : 0);
- final int srcK = 1 << srcLgK;
-
- switch (sw) {
- case 0: { //HLL_8, srcLgK=tgtLgK, src=heap, tgt=heap
- final byte[] srcArr = ((Hll8Array) src.hllSketchImpl).hllByteArr;
- final byte[] tgtArr = ((Hll8Array) tgt.hllSketchImpl).hllByteArr;
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcArr[i];
- final byte tgtV = tgtArr[i];
- tgtArr[i] = (byte) Math.max(srcV, tgtV);
- }
- break;
- }
- case 1: { //HLL_8, srcLgK=tgtLgK, src=heap, tgt=seg
- final byte[] srcArr = ((Hll8Array) src.hllSketchImpl).hllByteArr;
- final MemorySegment tgtSeg = tgt.getMemorySegment();
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcArr[i];
- final byte tgtV = tgtSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i);
- tgtSeg.set(JAVA_BYTE, HLL_BYTE_ARR_START + i, (byte) Math.max(srcV, tgtV));
- }
- break;
- }
- case 2: { //HLL_8, srcLgK=tgtLgK, src=seg, tgt=heap
- final MemorySegment srcSeg = src.getMemorySegment();
- final byte[] tgtArr = ((Hll8Array) tgt.hllSketchImpl).hllByteArr;
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i);
- final byte tgtV = tgtArr[i];
- tgtArr[i] = (byte) Math.max(srcV, tgtV);
- }
- break;
- }
- case 3: { //HLL_8, srcLgK=tgtLgK, src=seg, tgt=seg
- final MemorySegment srcSeg = src.getMemorySegment();
- final MemorySegment tgtSeg = tgt.getMemorySegment();
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i);
- final byte tgtV = tgtSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i);
- tgtSeg.set(JAVA_BYTE, HLL_BYTE_ARR_START + i, (byte) Math.max(srcV, tgtV));
- }
- break;
- }
- case 4: { //HLL_8, srcLgK>tgtLgK, src=heap, tgt=heap
- final int tgtKmask = (1 << tgtLgK) - 1;
- final byte[] srcArr = ((Hll8Array) src.hllSketchImpl).hllByteArr;
- final byte[] tgtArr = ((Hll8Array) tgt.hllSketchImpl).hllByteArr;
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcArr[i];
- final int j = i & tgtKmask;
- final byte tgtV = tgtArr[j];
- tgtArr[j] = (byte) Math.max(srcV, tgtV);
- }
- break;
- }
- case 5: { //HLL_8, srcLgK>tgtLgK, src=heap, tgt=seg
- final int tgtKmask = (1 << tgtLgK) - 1;
- final byte[] srcArr = ((Hll8Array) src.hllSketchImpl).hllByteArr;
- final MemorySegment tgtSeg = tgt.getMemorySegment();
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcArr[i];
- final int j = i & tgtKmask;
- final byte tgtV = tgtSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + j);
- tgtSeg.set(JAVA_BYTE, HLL_BYTE_ARR_START + j, (byte) Math.max(srcV, tgtV));
- }
- break;
- }
- case 6: { //HLL_8, srcLgK>tgtLgK, src=seg, tgt=heap
- final int tgtKmask = (1 << tgtLgK) - 1;
- final MemorySegment srcSeg = src.getMemorySegment();
- final byte[] tgtArr = ((Hll8Array) tgt.hllSketchImpl).hllByteArr;
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i);
- final int j = i & tgtKmask;
- final byte tgtV = tgtArr[j];
- tgtArr[j] = (byte) Math.max(srcV, tgtV);
- }
- break;
- }
- case 7: { //HLL_8, srcLgK>tgtLgK, src=seg, tgt=seg
- final int tgtKmask = (1 << tgtLgK) - 1;
- final MemorySegment srcSeg = src.getMemorySegment();
- final MemorySegment tgtSeg = tgt.getMemorySegment();
- for (int i = 0; i < srcK; i++) {
- final byte srcV = srcSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i);
- final int j = i & tgtKmask;
- final byte tgtV = tgtSeg.get(JAVA_BYTE, HLL_BYTE_ARR_START + j);
- tgtSeg.set(JAVA_BYTE, HLL_BYTE_ARR_START + j, (byte) Math.max(srcV, tgtV));
- }
- break;
- }
- case 8: case 9:
- {
- //!HLL_8, srcLgK=tgtLgK, src=heap, tgt=heap/seg
- final AbstractHllArray tgtAbsHllArr = (AbstractHllArray)(tgt.hllSketchImpl);
- if (src.getTgtHllType() == HLL_4) {
- final Hll4Array src4 = (Hll4Array) src.hllSketchImpl;
- final AuxHashMap auxHashMap = src4.getAuxHashMap();
- final int curMin = src4.getCurMin();
- int i = 0;
- int j = 0;
- while (j < srcK) {
- final byte b = src4.hllByteArr[i++];
- int value = Byte.toUnsignedInt(b) & loNibbleMask;
- tgtAbsHllArr.updateSlotNoKxQ(j, value == AUX_TOKEN ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- value = Byte.toUnsignedInt(b) >>> 4;
- tgtAbsHllArr.updateSlotNoKxQ(j, value == AUX_TOKEN ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- }
- } else {
- final Hll6Array src6 = (Hll6Array) src.hllSketchImpl;
- int i = 0;
- int j = 0;
- while (j < srcK) {
- final byte b1 = src6.hllByteArr[i++];
- final byte b2 = src6.hllByteArr[i++];
- final byte b3 = src6.hllByteArr[i++];
- int value = Byte.toUnsignedInt(b1) & 0x3f;
- tgtAbsHllArr.updateSlotNoKxQ(j++, value);
- value = Byte.toUnsignedInt(b1) >>> 6;
- value |= (Byte.toUnsignedInt(b2) & 0x0f) << 2;
- tgtAbsHllArr.updateSlotNoKxQ(j++, value);
- value = Byte.toUnsignedInt(b2) >>> 4;
- value |= (Byte.toUnsignedInt(b3) & 3) << 4;
- tgtAbsHllArr.updateSlotNoKxQ(j++, value);
- value = Byte.toUnsignedInt(b3) >>> 2;
- tgtAbsHllArr.updateSlotNoKxQ(j++, value);
- }
- }
- break;
- }
- case 10: case 11:
- {
- //!HLL_8, srcLgK=tgtLgK, src=seg, tgt=heap/seg
- final AbstractHllArray tgtAbsHllArr = (AbstractHllArray)(tgt.hllSketchImpl);
- if (src.getTgtHllType() == HLL_4) {
- final DirectHll4Array src4 = (DirectHll4Array) src.hllSketchImpl;
- final AuxHashMap auxHashMap = src4.getAuxHashMap();
- final int curMin = src4.getCurMin();
- int i = 0;
- int j = 0;
- while (j < srcK) {
- final byte b = src4.seg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i++);
- int value = Byte.toUnsignedInt(b) & loNibbleMask;
- tgtAbsHllArr.updateSlotNoKxQ(j, value == AUX_TOKEN ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- value = Byte.toUnsignedInt(b) >>> 4;
- tgtAbsHllArr.updateSlotNoKxQ(j, value == AUX_TOKEN ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- }
- } else {
- final DirectHll6Array src6 = (DirectHll6Array) src.hllSketchImpl;
- int i = 0;
- int offset = HLL_BYTE_ARR_START;
- while (i < srcK) {
- final byte b1 = src6.seg.get(JAVA_BYTE, offset++);
- final byte b2 = src6.seg.get(JAVA_BYTE, offset++);
- final byte b3 = src6.seg.get(JAVA_BYTE, offset++);
- int value = Byte.toUnsignedInt(b1) & 0x3f;
- tgtAbsHllArr.updateSlotNoKxQ(i++, value);
- value = Byte.toUnsignedInt(b1) >>> 6;
- value |= (Byte.toUnsignedInt(b2) & 0x0f) << 2;
- tgtAbsHllArr.updateSlotNoKxQ(i++, value);
- value = Byte.toUnsignedInt(b2) >>> 4;
- value |= (Byte.toUnsignedInt(b3) & 3) << 4;
- tgtAbsHllArr.updateSlotNoKxQ(i++, value);
- value = Byte.toUnsignedInt(b3) >>> 2;
- tgtAbsHllArr.updateSlotNoKxQ(i++, value);
- }
- }
- break;
- }
- case 12: case 13:
- {
- //!HLL_8, srcLgK>tgtLgK, src=heap, tgt=heap/seg
- final int tgtKmask = (1 << tgtLgK) - 1;
- final AbstractHllArray tgtAbsHllArr = (AbstractHllArray)(tgt.hllSketchImpl);
- if (src.getTgtHllType() == HLL_4) {
- final Hll4Array src4 = (Hll4Array) src.hllSketchImpl;
- final AuxHashMap auxHashMap = src4.getAuxHashMap();
- final int curMin = src4.getCurMin();
- int i = 0;
- int j = 0;
- while (j < srcK) {
- final byte b = src4.hllByteArr[i++];
- int value = Byte.toUnsignedInt(b) & loNibbleMask;
- tgtAbsHllArr.updateSlotNoKxQ(j & tgtKmask, value == AUX_TOKEN
- ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- value = Byte.toUnsignedInt(b) >>> 4;
- tgtAbsHllArr.updateSlotNoKxQ(j & tgtKmask, value == AUX_TOKEN
- ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- }
- } else {
- final Hll6Array src6 = (Hll6Array) src.hllSketchImpl;
- int i = 0;
- int j = 0;
- while (j < srcK) {
- final byte b1 = src6.hllByteArr[i++];
- final byte b2 = src6.hllByteArr[i++];
- final byte b3 = src6.hllByteArr[i++];
- int value = Byte.toUnsignedInt(b1) & 0x3f;
- tgtAbsHllArr.updateSlotNoKxQ(j++ & tgtKmask, value);
- value = Byte.toUnsignedInt(b1) >>> 6;
- value |= (Byte.toUnsignedInt(b2) & 0x0f) << 2;
- tgtAbsHllArr.updateSlotNoKxQ(j++ & tgtKmask, value);
- value = Byte.toUnsignedInt(b2) >>> 4;
- value |= (Byte.toUnsignedInt(b3) & 3) << 4;
- tgtAbsHllArr.updateSlotNoKxQ(j++ & tgtKmask, value);
- value = Byte.toUnsignedInt(b3) >>> 2;
- tgtAbsHllArr.updateSlotNoKxQ(j++ & tgtKmask, value);
- }
- }
- break;
- }
- case 14: case 15:
- {
- //!HLL_8, srcLgK>tgtLgK, src=seg, tgt=heap/seg
- final int tgtKmask = (1 << tgtLgK) - 1;
- final AbstractHllArray tgtAbsHllArr = (AbstractHllArray)(tgt.hllSketchImpl);
- if (src.getTgtHllType() == HLL_4) {
- final DirectHll4Array src4 = (DirectHll4Array) src.hllSketchImpl;
- final AuxHashMap auxHashMap = src4.getAuxHashMap();
- final int curMin = src4.getCurMin();
- int i = 0;
- int j = 0;
- while (j < srcK) {
- final byte b = src4.seg.get(JAVA_BYTE, HLL_BYTE_ARR_START + i++);
- int value = Byte.toUnsignedInt(b) & loNibbleMask;
- tgtAbsHllArr.updateSlotNoKxQ(j & tgtKmask, value == AUX_TOKEN
- ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- value = Byte.toUnsignedInt(b) >>> 4;
- tgtAbsHllArr.updateSlotNoKxQ(j & tgtKmask, value == AUX_TOKEN
- ? auxHashMap.mustFindValueFor(j) : value + curMin);
- j++;
- }
- } else {
- final DirectHll6Array src6 = (DirectHll6Array) src.hllSketchImpl;
- int i = 0;
- int offset = HLL_BYTE_ARR_START;
- while (i < srcK) {
- final byte b1 = src6.seg.get(JAVA_BYTE, offset++);
- final byte b2 = src6.seg.get(JAVA_BYTE, offset++);
- final byte b3 = src6.seg.get(JAVA_BYTE, offset++);
- int value = Byte.toUnsignedInt(b1) & 0x3f;
- tgtAbsHllArr.updateSlotNoKxQ(i++ & tgtKmask, value);
- value = Byte.toUnsignedInt(b1) >>> 6;
- value |= (Byte.toUnsignedInt(b2) & 0x0f) << 2;
- tgtAbsHllArr.updateSlotNoKxQ(i++ & tgtKmask, value);
- value = Byte.toUnsignedInt(b2) >>> 4;
- value |= (Byte.toUnsignedInt(b3) & 3) << 4;
- tgtAbsHllArr.updateSlotNoKxQ(i++ & tgtKmask, value);
- value = Byte.toUnsignedInt(b3) >>> 2;
- tgtAbsHllArr.updateSlotNoKxQ(i++ & tgtKmask, value);
- }
- }
- break;
- }
- default: break; //not possible
- }
- tgt.hllSketchImpl.putRebuildCurMinNumKxQFlag(true);
- }
-
- //Used by HllUnion operator. Always copies or downsamples to Heap HLL_8.
- //Caller must ultimately manage oooFlag, as caller has more context.
- /**
- * Copies or downsamples the given candidate HLLmode sketch to tgtLgK, HLL_8, on the heap.
- *
- * @param candidate the HllSketch to downsample, must be in HLL mode.
- * @param tgtLgK the LgK to downsample to.
- * @return the downsampled HllSketch.
- */
- private static final HllSketch downsample(final HllSketch candidate, final int tgtLgK) {
- final AbstractHllArray candArr = (AbstractHllArray) candidate.hllSketchImpl;
- final HllArray tgtHllArr = HllArray.newHeapHll(tgtLgK, TgtHllType.HLL_8);
- final PairIterator candItr = candArr.iterator();
- while (candItr.nextValid()) {
- tgtHllArr.couponUpdate(candItr.getPair()); //rebuilds KxQ, etc.
- }
- //both of these are required for isomorphism
- tgtHllArr.putHipAccum(candArr.getHipAccum());
- tgtHllArr.putOutOfOrder(candidate.isOutOfOrder());
- tgtHllArr.putRebuildCurMinNumKxQFlag(false);
- return new HllSketch(tgtHllArr);
- }
-
- //Used to rebuild curMin, numAtCurMin and KxQ registers, due to high performance merge operation
- static final void checkRebuildCurMinNumKxQ(final HllSketch sketch) {
- final HllSketchImpl hllSketchImpl = sketch.hllSketchImpl;
- final CurMode curMode = sketch.getCurMode();
- final TgtHllType tgtHllType = sketch.getTgtHllType();
- final boolean rebuild = hllSketchImpl.isRebuildCurMinNumKxQFlag();
- if ( !rebuild || (curMode != CurMode.HLL) || (tgtHllType != HLL_8) ) { return; }
- final AbstractHllArray absHllArr = (AbstractHllArray)(hllSketchImpl);
- int curMin = 64;
- int numAtCurMin = 0;
- double kxq0 = 1 << absHllArr.getLgConfigK();
- double kxq1 = 0;
- final PairIterator itr = absHllArr.iterator();
- while (itr.nextAll()) {
- final int v = itr.getValue();
- if (v > 0) {
- if (v < 32) { kxq0 += invPow2(v) - 1.0; }
- else { kxq1 += invPow2(v) - 1.0; }
- }
- if (v > curMin) { continue; }
- if (v < curMin) {
- curMin = v;
- numAtCurMin = 1;
- } else {
- numAtCurMin++;
- }
- }
- absHllArr.putKxQ0(kxq0);
- absHllArr.putKxQ1(kxq1);
- absHllArr.putCurMin(curMin);
- absHllArr.putNumAtCurMin(numAtCurMin);
- absHllArr.putRebuildCurMinNumKxQFlag(false);
- //HipAccum is not affected
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/HllUtil.java b/src/main/java/org/apache/datasketches/hll/HllUtil.java
deleted file mode 100644
index 500f946f7..000000000
--- a/src/main/java/org/apache/datasketches/hll/HllUtil.java
+++ /dev/null
@@ -1,144 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.Math.log;
-import static java.lang.Math.sqrt;
-import static org.apache.datasketches.common.Util.checkBounds;
-import static org.apache.datasketches.hll.PreambleUtil.HASH_SET_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.LIST_PREINTS;
-import static org.apache.datasketches.hll.PreambleUtil.extractCurMode;
-import static org.apache.datasketches.hll.PreambleUtil.extractFamilyId;
-import static org.apache.datasketches.hll.PreambleUtil.extractPreInts;
-import static org.apache.datasketches.hll.PreambleUtil.extractSerVer;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesReadOnlyException;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class HllUtil {
- static final int KEY_BITS_26 = 26;
- static final int VAL_BITS_6 = 6;
- static final int KEY_MASK_26 = (1 << KEY_BITS_26) - 1;
- static final int VAL_MASK_6 = (1 << VAL_BITS_6) - 1;
- static final int EMPTY = 0;
- static final int MIN_LOG_K = 4;
- static final int MAX_LOG_K = 21;
-
- static final double HLL_HIP_RSE_FACTOR = sqrt(log(2.0)); //.8325546
- static final double HLL_NON_HIP_RSE_FACTOR = sqrt((3.0 * log(2.0)) - 1.0); //1.03896
- static final double COUPON_RSE_FACTOR = .409; //at transition point not the asymptote
-
- static final double COUPON_RSE = COUPON_RSE_FACTOR / (1 << 13);
-
- static final int LG_INIT_LIST_SIZE = 3;
- static final int LG_INIT_SET_SIZE = 5;
- static final int RESIZE_NUMER = 3;
- static final int RESIZE_DENOM = 4;
-
- static final int loNibbleMask = 0x0f;
- static final int hiNibbleMask = 0xf0;
- static final int AUX_TOKEN = 0xf;
-
- /**
- * Log2 table sizes for exceptions based on lgK from 0 to 26.
- * However, only lgK from 4 to 21 are used.
- */
- static final int[] LG_AUX_ARR_INTS = {
- 0, 2, 2, 2, 2, 2, 2, 3, 3, 3, //0 - 9
- 4, 4, 5, 5, 6, 7, 8, 9, 10, 11, //10 - 19
- 12, 13, 14, 15, 16, 17, 18 //20 - 26
- };
-
- //Checks
- static int checkLgK(final int lgK) {
- if ((lgK >= MIN_LOG_K) && (lgK <= MAX_LOG_K)) { return lgK; }
- throw new SketchesArgumentException(
- "Log K must be between 4 and 21, inclusive: " + lgK);
- }
-
- static void checkSegSize(final long minBytes, final long capBytes) {
- if (capBytes < minBytes) {
- throw new SketchesArgumentException(
- "Given MemorySegment is not large enough: " + capBytes);
- }
- }
-
- static void checkNumStdDev(final int numStdDev) {
- if ((numStdDev < 1) || (numStdDev > 3)) {
- throw new SketchesArgumentException(
- "NumStdDev may not be less than 1 or greater than 3.");
- }
- }
-
- static CurMode checkPreamble(final MemorySegment seg) {
- checkBounds(0, 8, seg.byteSize()); //need min 8 bytes
- final int preInts = extractPreInts(seg);
- checkBounds(0, (long)preInts * Integer.BYTES, seg.byteSize());
- final int serVer = extractSerVer(seg);
- final int famId = extractFamilyId(seg);
- final CurMode curMode = extractCurMode(seg);
- if (
- (famId != Family.HLL.getID())
- || (serVer != 1)
- || ((preInts != LIST_PREINTS) && (preInts != HASH_SET_PREINTS) && (preInts != HLL_PREINTS))
- || ((curMode == CurMode.LIST) && (preInts != LIST_PREINTS))
- || ((curMode == CurMode.SET) && (preInts != HASH_SET_PREINTS))
- || ((curMode == CurMode.HLL) && (preInts != HLL_PREINTS))
- ) {
- throw new SketchesArgumentException("Possible Corruption, Invalid Preamble:"
- + PreambleUtil.toString(seg));
- }
- return curMode;
- }
-
- //Exceptions
- static void noWriteAccess() {
- throw new SketchesReadOnlyException(
- "This sketch is compact or does not have write access to the underlying resource.");
- }
-
- //Used for thrown exceptions
- static String pairString(final int pair) {
- return "SlotNo: " + getPairLow26(pair) + ", Value: "
- + getPairValue(pair);
- }
-
- //Pairs
- static int pair(final int slotNo, final int value) {
- return (value << KEY_BITS_26) | (slotNo & KEY_MASK_26);
- }
-
- static int getPairLow26(final int coupon) {
- return coupon & KEY_MASK_26;
- }
-
- static int getPairValue(final int coupon) {
- return coupon >>> KEY_BITS_26;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/IntArrayPairIterator.java b/src/main/java/org/apache/datasketches/hll/IntArrayPairIterator.java
deleted file mode 100644
index 85603773c..000000000
--- a/src/main/java/org/apache/datasketches/hll/IntArrayPairIterator.java
+++ /dev/null
@@ -1,91 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-
-/**
- * Iterates over an on-heap integer array of pairs extracting
- * the components of the pair at a given index.
- *
- * @author Lee Rhodes
- */
-final class IntArrayPairIterator extends PairIterator {
- private final int[] array;
- private final int arrLen;
- private final int slotMask;
- private int index;
- private int pair;
-
- //used by CouponList, HeapAuxHashMap
- IntArrayPairIterator(final int[] array, final int lgConfigK) {
- this.array = array;
- slotMask = (1 << lgConfigK) - 1;
- arrLen = array.length;
- index = - 1;
- }
-
- @Override
- public int getIndex() {
- return index;
- }
-
- @Override
- public int getKey() {
- return HllUtil.getPairLow26(pair);
- }
-
- @Override
- public int getPair() {
- return pair;
- }
-
- @Override
- public int getSlot() {
- return getKey() & slotMask;
- }
-
- @Override
- public int getValue() {
- return HllUtil.getPairValue(pair);
- }
-
- @Override
- public boolean nextAll() {
- if (++index < arrLen) {
- pair = array[index];
- return true;
- }
- return false;
- }
-
- @Override
- public boolean nextValid() {
- while (++index < arrLen) {
- final int pair = array[index];
- if (pair != EMPTY) {
- this.pair = pair;
- return true;
- }
- }
- return false;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/IntMemorySegmentPairIterator.java b/src/main/java/org/apache/datasketches/hll/IntMemorySegmentPairIterator.java
deleted file mode 100644
index 7f510de18..000000000
--- a/src/main/java/org/apache/datasketches/hll/IntMemorySegmentPairIterator.java
+++ /dev/null
@@ -1,105 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.hll.HllUtil.EMPTY;
-
-import java.lang.foreign.MemorySegment;
-
-/**
- * Iterates within a given MemorySegment extracting integer pairs.
- *
- * @author Lee Rhodes
- */
-final class IntMemorySegmentPairIterator extends PairIterator {
- private final MemorySegment seg;
- private final long offsetBytes;
- private final int arrLen;
- private final int slotMask;
- private int index;
- private int pair;
-
- //Used by DirectAuxHashMap, DirectCouponList
- IntMemorySegmentPairIterator(
- final MemorySegment seg, final long offsetBytes, final int arrayLength, final int lgConfigK) {
- this.seg = seg;
- this.offsetBytes = offsetBytes;
- arrLen = arrayLength;
- slotMask = (1 << lgConfigK) - 1;
- index = -1;
- }
-
- IntMemorySegmentPairIterator(final byte[] byteArr, final long offsetBytes, final int lengthPairs,
- final int lgConfigK) {
- this(MemorySegment.ofArray(byteArr), offsetBytes, lengthPairs, lgConfigK);
- }
-
- @Override
- public int getIndex() {
- return index;
- }
-
- @Override
- public int getKey() {
- return HllUtil.getPairLow26(pair);
- }
-
- @Override
- public int getPair() {
- return pair;
- }
-
- @Override
- public int getSlot() {
- return getKey() & slotMask;
- }
-
- @Override
- public int getValue() {
- return HllUtil.getPairValue(pair);
- }
-
- @Override
- public boolean nextAll() {
- if (++index < arrLen) {
- pair = pair();
- return true;
- }
- return false;
- }
-
- @Override
- public boolean nextValid() {
- while (++index < arrLen) {
- final int pair = pair();
- if (pair != EMPTY) {
- this.pair = pair;
- return true;
- }
- }
- return false;
- }
-
- int pair() {
- return seg.get(JAVA_INT_UNALIGNED, offsetBytes + (index << 2));
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/PairIterator.java b/src/main/java/org/apache/datasketches/hll/PairIterator.java
deleted file mode 100644
index 0282152a3..000000000
--- a/src/main/java/org/apache/datasketches/hll/PairIterator.java
+++ /dev/null
@@ -1,98 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-/**
- * @author Lee Rhodes
- */
-abstract class PairIterator {
-
- /**
- * Gets the header string for a list of pairs
- * @return the header string for a list of pairs
- */
- String getHeader() {
- return String.format("%10s%10s%10s%6s", "Index", "Key", "Slot", "Value");
- }
-
- /**
- * In LIST and SET modes, this gets the iterating index into the integer array of HLL key/value
- * pairs.
- * In HLL mode, this is the iterating index into the hypothetical array of HLL values, which may
- * be physically contructed differently based on the compaction scheme (HLL_4, HLL_6, HLL_8).
- * @return the index.
- */
- abstract int getIndex();
-
- /**
- * Gets the key, the low 26 bits of an pair, and can be up to 26 bits in length.
- * @return the key
- */
- abstract int getKey();
-
- /**
- * Gets the key, value pair as a single int where the key is the lower 26 bits
- * and the value is in the upper 6 bits.
- * @return the key, value pair.
- */
- abstract int getPair();
-
- /**
- * Gets the target or actual HLL slot number, which is derived from the key and LgConfigK.
- * The slot number is the index into a hypothetical array of length K and has LgConfigK bits.
- * If in LIST or SET mode this is the index into the hypothetical target HLL array of size K.
- * In HLL mode, this will be the actual index into the hypothetical target HLL array of size K.
- * @return the target or actual HLL slot number.
- */
- abstract int getSlot();
-
- /**
- * Gets the current pair as a string
- * @return the current pair as a string
- */
- String getString() {
- final int index = getIndex();
- final int key = getKey();
- final int slot = getSlot();
- final int value = getValue();
- return String.format("%10d%10d%10d%6d", index, key, slot, value);
- }
-
- /**
- * Gets the HLL value of a particular slot or pair.
- * @return the HLL value of a particular slot or pair.
- */
- abstract int getValue();
-
- /**
- * Returns true at the next pair in sequence.
- * If false, the iteration is done.
- * @return true at the next pair in sequence.
- */
- abstract boolean nextAll();
-
- /**
- * Returns true at the next pair where getKey() and getValue() are valid.
- * If false, the iteration is done.
- * @return true at the next pair where getKey() and getValue() are valid.
- */
- abstract boolean nextValid();
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/PreambleUtil.java b/src/main/java/org/apache/datasketches/hll/PreambleUtil.java
deleted file mode 100644
index b86b65fc6..000000000
--- a/src/main/java/org/apache/datasketches/hll/PreambleUtil.java
+++ /dev/null
@@ -1,465 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_DOUBLE_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.common.Util.ceilingPowerOf2;
-import static org.apache.datasketches.common.Util.exactLog2OfLong;
-import static org.apache.datasketches.common.Util.zeroPad;
-import static org.apache.datasketches.hll.HllUtil.LG_AUX_ARR_INTS;
-import static org.apache.datasketches.hll.HllUtil.LG_INIT_SET_SIZE;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_DENOM;
-import static org.apache.datasketches.hll.HllUtil.RESIZE_NUMER;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.Family;
-
-//@formatter:off
-/**
- * These three target types are isomorphic representations of the same underlying HLL algorithm.
- * Thus, given the same value of lgConfigK and the same input, all three HLL target types
- * will produce identical estimates and have identical error distributions. The memory usage and the serialization size of the sketch during this early warmup phase starts
- * out very small (8 bytes, when empty) and then grows in increments of 4 bytes as required
- * until the full HLL array is allocated. This transition point occurs at about 10% of K for
- * sketches where lgConfigK is > 8. Accuracy for very small streams can be important because Big Data is often fragmented into millions of smaller
- * streams (or segments) that inevitably are power-law distributed in size. If you are sketching all these fragments,
- * as a general rule, more than 80% of your sketches will be very small, 20% will be much larger, and only a few will
- * be very large in cardinality. As a result of all of this work on accuracy, one will get a very smooth curve of the underlying accuracy of the
- * sketch once the statistical randomness is removed through multiple trials. This can be observed in the
- * following graph. The above graph has 7 curves. At y = 0, is the median line that hugs the x-axis so closely that it can't be seen.
- * The two curves, just above and just below the x-axis, correspond to +/- 1 standard deviation (SD) of error.
- * The distance between either one of this pair and the x-axis is also known as the Relative Standard Error (RSE).
- * This type of graph for illustrating sketch error we call a "pitchfork plot". The next two curves above and below correspond to +/- 2 SD, and
- * the top-most and bottom-most curves correspond to +/- 3 SD.
- * The chart grid lines are set at +/- multiples of Relative Standard Error (RSE) that correspond to +/- 1,2,3 SD.
- * Below the cardinality of about 512 there is no error at all. This is the point where this particular
- * sketch transitions from sparse to dense (or estimation) mode. In terms of accuracy, all three types, for the same lgConfigK, have the same error
- * distribution as a function of cardinality. The configuration parameter lgConfigK is the log-base-2 of K,
- * where K is the number of buckets or slots for the sketch. lgConfigK impacts both accuracy and
- * the size of the sketch in memory and when stored. Why is this important? Suppose you have been building a history of sketches of your customer's
- * data that go back a full year (or 5 or 10!) that were all configured with lgConfigK = 12. Because sketches
- * are so much smaller than the raw data it is possible that the raw data was discarded keeping only the sketches.
- * Even if you have the raw data, it might be very expensive and time consuming to reload and rebuild all your
- * sketches with a larger more accurate size, say, lgConfigK = 14.
- * This capability enables you to merge last year's data with this year's data built with larger sketches and still
- * have meaningful results. In other words, you can change your mind about what size sketch you need for your application at any time and
- * will not lose access to the data contained in your older historical sketches. This capability does come with a caveat: The resulting accuracy of the merged sketch will be the accuracy of the
- * smaller of the two sketches. Without this capability, you would either be stuck with the configuration you first
- * chose forever, or you would have to rebuild all your sketches from scratch, or worse, not be able to recover your
- * historical data. [1] Philippe Flajolet, et al,
-HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm.
- * DMTCS proc. AH, 2007, 127-146.
- *
- * [2] Edith Cohen,
-All-Distances Sketches, Revisited: HIP Estimators for Massive Graphs Analysis.
- * PODS'14, June 22-27, Snowbird, UT, USA.
- *
- * [3] Daniel Ting,
- *
-Streamed Approximate Counting of Distinct Elements, Beating Optimal Batch Methods.
- * KDD'14 August 24, 2014 New York, New York USA.
- *
- * [4] Kevin Lang,
- *
-Back to the Future: an Even More Nearly Optimal Cardinality Estimation Algorithm.
- * arXiv 1708.06839, August 22, 2017, Yahoo Research.
- *
- * [5] MemorySegment Component, See
- * JEP 454: Foreign Function And Memory API
- *
- * [6] MacBook Pro 2.3 GHz 8-Core Intel Core i9
- *
- * @see org.apache.datasketches.cpc.CpcSketch
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-package org.apache.datasketches.hll;
diff --git a/src/main/java/org/apache/datasketches/hllmap/CouponHashMap.java b/src/main/java/org/apache/datasketches/hllmap/CouponHashMap.java
deleted file mode 100644
index d69d65e7c..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/CouponHashMap.java
+++ /dev/null
@@ -1,332 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hllmap;
-
-import static org.apache.datasketches.common.Util.checkIfPowerOf2;
-import static org.apache.datasketches.common.Util.invPow2;
-
-import java.util.Arrays;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.hash.MurmurHash3;
-
-/**
- * Implements a key-value map where the value is a hash map of coupons.
- *
- * The outer map is implemented as a prime-sized, Open Address, Double Hash, with deletes, so
- * this table can grow and shrink. Each entry row has a 1-byte count where 255 is a marker for
- * "dirty" and zero is empty.
- *
- * The inner hash tables are implemented with linear probing or OASH and a load factor of 0.75.
- *
- * @author Lee Rhodes
- * @author Alexander Saydakov
- * @author Kevin Lang
- */
-final class CouponHashMap extends Map {
- private static final double INNER_LOAD_FACTOR = 0.75;
- private static final byte DELETED_KEY_MARKER = (byte) 255;
- private static final int BYTE_MASK = 0XFF;
- private static final int COUPON_K = 1024;
- private static final double RSE = 0.408 / Math.sqrt(1024);
-
- private final int maxCouponsPerKey_;
- private final int capacityCouponsPerKey_;
- private final int entrySizeBytes_;
-
- private int tableEntries_;
- private int capacityEntries_;
- private int numActiveKeys_;
- private int numDeletedKeys_;
-
- //Arrays
- private byte[] keysArr_;
- private short[] couponsArr_;
- private byte[] curCountsArr_; //also acts as a stateArr: 0 empty, 255 deleted
- private float[] invPow2SumArr_;
- private float[] hipEstAccumArr_;
-
- private CouponHashMap(final int keySizeBytes, final int maxCouponsPerKey) {
- super(keySizeBytes);
- maxCouponsPerKey_ = maxCouponsPerKey;
- capacityCouponsPerKey_ = (int)(maxCouponsPerKey * INNER_LOAD_FACTOR);
- entrySizeBytes_ = keySizeBytes + (maxCouponsPerKey * Short.BYTES) + 1 + 4 + 4;
- }
-
- static CouponHashMap getInstance(final int keySizeBytes, final int maxCouponsPerKey) {
- checkMaxCouponsPerKey(maxCouponsPerKey);
- final int tableEntries = COUPON_MAP_MIN_NUM_ENTRIES;
-
- final CouponHashMap map = new CouponHashMap(keySizeBytes, maxCouponsPerKey);
- map.tableEntries_ = tableEntries;
- map.capacityEntries_ = (int)(tableEntries * COUPON_MAP_GROW_TRIGGER_FACTOR);
- map.numActiveKeys_ = 0;
- map.numDeletedKeys_ = 0;
-
- map.keysArr_ = new byte[tableEntries * keySizeBytes];
- map.couponsArr_ = new short[tableEntries * maxCouponsPerKey];
- map.curCountsArr_ = new byte[tableEntries];
- map.invPow2SumArr_ = new float[tableEntries];
- map.hipEstAccumArr_ = new float[tableEntries];
- return map;
- }
-
- @Override
- double update(final byte[] key, final short coupon) {
- final int entryIndex = findOrInsertKey(key);
- return update(entryIndex, coupon); //negative when time to promote
- }
-
- @Override
- double update(final int entryIndex, final short coupon) {
- final int couponMapArrEntryIndex = entryIndex * maxCouponsPerKey_;
-
- int innerCouponIndex = (coupon & 0xFFFF) % maxCouponsPerKey_;
-
- while (couponsArr_[couponMapArrEntryIndex + innerCouponIndex] != 0) {
- if (couponsArr_[couponMapArrEntryIndex + innerCouponIndex] == coupon) {
- return hipEstAccumArr_[entryIndex]; //duplicate, returns the estimate
- }
- innerCouponIndex = (innerCouponIndex + 1) % maxCouponsPerKey_; //linear search
- }
- if (((curCountsArr_[entryIndex] + 1) & BYTE_MASK) > capacityCouponsPerKey_) {
- //returns the negative estimate, as signal to promote
- return -hipEstAccumArr_[entryIndex];
- }
-
- couponsArr_[couponMapArrEntryIndex + innerCouponIndex] = coupon; //insert
- curCountsArr_[entryIndex]++;
- //hip += k/qt; qt -= 1/2^(val);
- hipEstAccumArr_[entryIndex] += COUPON_K / invPow2SumArr_[entryIndex];
- invPow2SumArr_[entryIndex] -= invPow2(coupon16Value(coupon));
- return hipEstAccumArr_[entryIndex]; //returns the estimate
- }
-
- @Override
- double getEstimate(final byte[] key) {
- final int index = findKey(key);
- if (index < 0) { return 0; }
- return hipEstAccumArr_[index];
- }
-
- @Override
- double getUpperBound(final byte[] key) {
- return getEstimate(key) * (1 + RSE);
- }
-
- @Override
- double getLowerBound(final byte[] key) {
- return getEstimate(key) * (1 - RSE);
- }
-
- @Override
- void updateEstimate(final int entryIndex, final double estimate) {
- if (entryIndex < 0) {
- throw new SketchesArgumentException("Key not found.");
- }
- hipEstAccumArr_[entryIndex] = (float) estimate;
- }
-
- /**
- * Returns entryIndex if the given key is found. If not found, returns one's complement index
- * of an empty slot for insertion, which may be over a deleted key.
- * @param key the given key
- * @return the entryIndex
- */
- @Override
- int findKey(final byte[] key) {
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries_);
- int firstDeletedIndex = -1;
- final int loopIndex = entryIndex;
- do {
- if (curCountsArr_[entryIndex] == 0) {
- return firstDeletedIndex == -1 ? ~entryIndex : ~firstDeletedIndex; // found empty or deleted
- }
- if (curCountsArr_[entryIndex] == DELETED_KEY_MARKER) {
- if (firstDeletedIndex == -1) {
- firstDeletedIndex = entryIndex;
- }
- } else if (Map.arraysEqual(keysArr_, entryIndex * keySizeBytes_, key, 0, keySizeBytes_)) {
- return entryIndex; // found key
- }
- entryIndex = (entryIndex + getStride(hash[1], tableEntries_)) % tableEntries_;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- @Override
- int findOrInsertKey(final byte[] key) {
- int entryIndex = findKey(key);
- if (entryIndex < 0) { //key not found
- entryIndex = ~entryIndex;
- if (curCountsArr_[entryIndex] == DELETED_KEY_MARKER) { // reusing slot from a deleted key
- Arrays.fill(couponsArr_, entryIndex * maxCouponsPerKey_,
- (entryIndex + 1) * maxCouponsPerKey_, (short) 0);
- curCountsArr_[entryIndex] = 0;
- numDeletedKeys_--;
- }
- if ((numActiveKeys_ + numDeletedKeys_) >= capacityEntries_) {
- resize();
- entryIndex = ~findKey(key);
- assert entryIndex >= 0;
- }
- //insert new key
- System.arraycopy(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_);
- //initialize HIP: qt <- k; hip <- 0;
- invPow2SumArr_[entryIndex] = COUPON_K;
- hipEstAccumArr_[entryIndex] = 0;
- numActiveKeys_++;
- }
- return entryIndex;
- }
-
- @Override
- void deleteKey(final int entryIndex) {
- curCountsArr_[entryIndex] = DELETED_KEY_MARKER;
- numActiveKeys_--;
- numDeletedKeys_++;
- if ((numActiveKeys_ > COUPON_MAP_MIN_NUM_ENTRIES)
- && (numActiveKeys_ < (tableEntries_ * COUPON_MAP_SHRINK_TRIGGER_FACTOR))) {
- resize();
- }
- }
-
- @Override
- CouponsIterator getCouponsIterator(final int entryIndex) {
- return new CouponsIterator(couponsArr_, entryIndex * maxCouponsPerKey_, maxCouponsPerKey_);
- }
-
- @Override
- double getEntrySizeBytes() {
- return entrySizeBytes_;
- }
-
- @Override
- int getTableEntries() {
- return tableEntries_;
- }
-
- @Override
- int getCapacityEntries() {
- return capacityEntries_;
- }
-
- @Override
- int getCurrentCountEntries() {
- return numActiveKeys_ + numDeletedKeys_;
- }
-
- @Override
- long getTotalUsageBytes() {
- final long arrays = keysArr_.length
- + ((long) couponsArr_.length * Short.BYTES)
- + curCountsArr_.length
- + ((long) invPow2SumArr_.length * Float.BYTES)
- + ((long) hipEstAccumArr_.length * Float.BYTES);
- final long other = 4 * 5;
- return arrays + other;
- }
-
- @Override
- int getActiveEntries() {
- return numActiveKeys_;
- }
-
- @Override
- int getDeletedEntries() {
- return numDeletedKeys_;
- }
-
- @Override
- int getMaxCouponsPerEntry() {
- return maxCouponsPerKey_;
- }
-
- @Override
- int getCapacityCouponsPerEntry() {
- return capacityCouponsPerKey_;
- }
-
- private static final void checkMaxCouponsPerKey(final int maxCouponsPerKey) {
- checkIfPowerOf2(maxCouponsPerKey, "maxCouponsPerKey");
- final int cpk = maxCouponsPerKey;
- if ((cpk < 16) || (cpk > 256)) {
- throw new SketchesArgumentException(
- "Required: 16 <= maxCouponsPerKey <= 256 : " + maxCouponsPerKey);
- }
- }
-
- private void resize() {
- final byte[] oldKeysArr = keysArr_;
- final short[] oldCouponMapArr = couponsArr_;
- final byte[] oldCurCountsArr = curCountsArr_;
- final float[] oldInvPow2SumArr = invPow2SumArr_;
- final float[] oldHipEstAccumArr = hipEstAccumArr_;
- final int oldNumEntries = tableEntries_;
- tableEntries_ = Math.max(
- nextPrime((int) (numActiveKeys_ / COUPON_MAP_TARGET_FILL_FACTOR)),
- COUPON_MAP_MIN_NUM_ENTRIES
- );
- capacityEntries_ = (int)(tableEntries_ * COUPON_MAP_GROW_TRIGGER_FACTOR);
- keysArr_ = new byte[tableEntries_ * keySizeBytes_];
- couponsArr_ = new short[tableEntries_ * maxCouponsPerKey_];
- curCountsArr_ = new byte[tableEntries_];
- invPow2SumArr_ = new float[tableEntries_];
- hipEstAccumArr_ = new float[tableEntries_];
- numActiveKeys_ = 0;
- numDeletedKeys_ = 0;
- for (int i = 0; i < oldNumEntries; i++) {
- if ((oldCurCountsArr[i] != 0) && (oldCurCountsArr[i] != DELETED_KEY_MARKER)) {
- //extract an old valid key
- final byte[] key =
- Arrays.copyOfRange(oldKeysArr, i * keySizeBytes_, (i * keySizeBytes_) + keySizeBytes_);
- //insert the key and get its index
- final int index = insertKey(key);
- //copy the coupons array into that index
- System.arraycopy(oldCouponMapArr, i * maxCouponsPerKey_, couponsArr_,
- index * maxCouponsPerKey_, maxCouponsPerKey_);
- //transfer the count
- curCountsArr_[index] = oldCurCountsArr[i];
- //transfer the HIP registers
- invPow2SumArr_[index] = oldInvPow2SumArr[i];
- hipEstAccumArr_[index] = oldHipEstAccumArr[i];
- }
- }
- }
-
- // for internal use by resize, no resize check and no deleted key check here
- // no changes to HIP
- private int insertKey(final byte[] key) {
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries_);
- final int loopIndex = entryIndex;
- do {
- if (curCountsArr_[entryIndex] == 0) {
- System.arraycopy(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_);
- numActiveKeys_++;
- return entryIndex;
- }
- entryIndex = (entryIndex + getStride(hash[1], tableEntries_)) % tableEntries_;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hllmap/CouponTraverseMap.java b/src/main/java/org/apache/datasketches/hllmap/CouponTraverseMap.java
deleted file mode 100644
index ae5e0a523..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/CouponTraverseMap.java
+++ /dev/null
@@ -1,302 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hllmap;
-
-import java.util.Arrays;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.hash.MurmurHash3;
-
-/**
- * Implements a key-value map where the value is a simple array of coupons. Search operations are a
- * simple traverse of the consecutive coupons. Because of this, the maximum practical size of the
- * coupon array is about 8 coupons.
- *
- * The map is implemented as a prime-sized, Open Address, Double Hash, with deletes and a 1-bit
- * state array. The size of this map can grow or shrink.
- *
- * @author Lee Rhodes
- * @author Alexander Saydakov
- * @author Kevin Lang
- */
-final class CouponTraverseMap extends Map {
- private static final double RSE = 0.408 / Math.sqrt(1024);
- //private static int
- private final int maxCouponsPerKey_;
-
- private int tableEntries_;
- private int capacityEntries_;
- private int numActiveKeys_;
- private int numDeletedKeys_;
- private double entrySizeBytes_;
-
- //Arrays
- private byte[] keysArr_;
- private short[] couponsArr_;
-
- /**
- * Each entry row, associated with a key, also contains 3 double registers for accurately
- * tracking the HIP (Historical Inverse Probability) estimator. HLL implementations have multiple
- * estimators and the early estimators in this implementation are quite novel and provide superior
- * error performance over most other HLL implementations.
- *
- * @author Lee Rhodes
- * @author KevinLang
- * @author Alexander Saydakov
- */
-final class HllMap extends Map {
- private static final double LOAD_FACTOR = 15.0 / 16.0;
- private static final int HLL_INIT_NUM_ENTRIES = 157;
- private static final int HLL_INIT_NUM_ENTRIES_ARR_SIZE = (int) Math.ceil(HLL_INIT_NUM_ENTRIES / 8.0);
- private static final float HLL_RESIZE_FACTOR = 2.0F;
- private static final double RSE = sqrt(log(2.0)) / 32.0;
- private final int k_;
- private final int hllArrLongs_; //# of longs required to store the HLL array
-
- private int tableEntries_; //Full size of the table
- private int capacityEntries_; //max capacity entries defined by Load factor
- private int curCountEntries_; //current count of valid entries
- private float growthFactor_; //e.g., 1.2 to 2.0
- private double entrySizeBytes_;
-
- //Arrays
- private byte[] keysArr_; //keys of zero are allowed
- private long[] arrOfHllArr_;
- private double[] invPow2SumHiArr_;
- private double[] invPow2SumLoArr_;
- private double[] hipEstAccumArr_;
- private byte[] stateArr_;
-
- /**
- * Private constructor used to set all finals
- * @param keySizeBytes size of key in bytes
- * @param k size of HLL sketch
- */
- private HllMap(final int keySizeBytes, final int k) {
- super(keySizeBytes);
- k_ = k;
- hllArrLongs_ = (k / 10) + 1;
- }
-
- static HllMap getInstance(final int keySizeBytes, final int k) {
-
- final HllMap map = new HllMap(keySizeBytes, k);
- map.tableEntries_ = HLL_INIT_NUM_ENTRIES;
- map.capacityEntries_ = (int)(HLL_INIT_NUM_ENTRIES * LOAD_FACTOR);
- map.curCountEntries_ = 0;
- map.growthFactor_ = HLL_RESIZE_FACTOR;
- map.entrySizeBytes_ = updateEntrySizeBytes(map.tableEntries_, keySizeBytes, map.hllArrLongs_);
-
- map.keysArr_ = new byte[HLL_INIT_NUM_ENTRIES * map.keySizeBytes_];
- map.arrOfHllArr_ = new long[HLL_INIT_NUM_ENTRIES * map.hllArrLongs_];
- map.invPow2SumHiArr_ = new double[HLL_INIT_NUM_ENTRIES];
- map.invPow2SumLoArr_ = new double[HLL_INIT_NUM_ENTRIES];
- map.hipEstAccumArr_ = new double[HLL_INIT_NUM_ENTRIES];
- map.stateArr_ = new byte[HLL_INIT_NUM_ENTRIES_ARR_SIZE];
- return map;
- }
-
- @Override
- double update(final byte[] key, final short coupon) {
- final int entryIndex = findOrInsertKey(key);
- return update(entryIndex, coupon);
- }
-
- @Override
- double update(final int entryIndex, final short coupon) {
- updateHll(entryIndex, coupon); //update HLL array, updates HIP
- return hipEstAccumArr_[entryIndex];
- }
-
- @Override
- double getEstimate(final byte[] key) {
- if (key == null) { return Double.NaN; }
- final int entryIndex = findKey(key);
- if (entryIndex < 0) {
- return 0;
- }
- return hipEstAccumArr_[entryIndex];
- }
-
- @Override
- double getUpperBound(final byte[] key) {
- return getEstimate(key) * (1 + RSE);
- }
-
- @Override
- double getLowerBound(final byte[] key) {
- return getEstimate(key) * (1 - RSE);
- }
-
- @Override
- void updateEstimate(final int entryIndex, final double estimate) {
- hipEstAccumArr_[entryIndex] = estimate;
- }
-
- /**
- * Returns the entry index for the given key given the array of keys, if found.
- * Otherwise, returns the one's complement of first empty entry found;
- * @param key the key to search for
- * @return the entry index of the given key, or the one's complement of the index if not found.
- */
- @Override
- final int findKey(final byte[] key) {
- final int keyLen = key.length;
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries_);
- final int stride = getStride(hash[1], tableEntries_);
- final int loopIndex = entryIndex;
-
- do {
- if (isBitClear(stateArr_, entryIndex)) { //check if slot is empty
- return ~entryIndex;
- }
- if (arraysEqual(key, 0, keysArr_, entryIndex * keyLen, keyLen)) { //check for key match
- return entryIndex;
- }
- entryIndex = (entryIndex + stride) % tableEntries_;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- @Override
- int findOrInsertKey(final byte[] key) {
- int entryIndex = findKey(key);
- if (entryIndex < 0) { //key not found, initialize new row
- entryIndex = ~entryIndex;
- System.arraycopy(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_);
- setBit(stateArr_, entryIndex);
- invPow2SumHiArr_[entryIndex] = k_;
- invPow2SumLoArr_[entryIndex] = 0;
- hipEstAccumArr_[entryIndex] = 0;
- curCountEntries_++;
- if (curCountEntries_ > capacityEntries_) {
- resize();
- entryIndex = findKey(key);
- assert entryIndex >= 0;
- }
- }
- return entryIndex;
- }
-
- @Override
- double getEntrySizeBytes() {
- return entrySizeBytes_;
- }
-
- @Override
- int getTableEntries() {
- return tableEntries_;
- }
-
- @Override
- int getCapacityEntries() {
- return capacityEntries_;
- }
-
- @Override
- int getCurrentCountEntries() {
- return curCountEntries_;
- }
-
- @Override
- long getTotalUsageBytes() {
- final long arrays = keysArr_.length
- + ((long) arrOfHllArr_.length * Long.BYTES)
- + ((long) invPow2SumLoArr_.length * Double.BYTES)
- + ((long) invPow2SumHiArr_.length * Double.BYTES)
- + ((long) hipEstAccumArr_.length * Double.BYTES)
- + stateArr_.length;
- final long other = (5L * Integer.BYTES) + Float.BYTES + Double.BYTES;
- return arrays + other;
- }
-
- @Override
- CouponsIterator getCouponsIterator(final int index) {
- // not applicable
- return null;
- }
-
- @Override
- int getMaxCouponsPerEntry() {
- // not applicable
- return 0;
- }
-
- @Override
- int getCapacityCouponsPerEntry() {
- // not applicable
- return 0;
- }
-
- @Override
- int getActiveEntries() {
- return curCountEntries_;
- }
-
- @Override
- int getDeletedEntries() {
- return 0;
- }
-
- /**
- * Find the first empty slot for the given key.
- * Only used by resize, where it is known that the key does not exist in the table.
- * Throws an exception if no empty slots.
- * @param key the given key
- * @param tableEntries prime size of table
- * @param stateArr the valid bit array
- * @return the first empty slot for the given key
- */
- private static final int findEmpty(final byte[] key, final int tableEntries, final byte[] stateArr) {
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries);
- final int stride = getStride(hash[1], tableEntries);
- final int loopIndex = entryIndex;
-
- do {
- if (isBitClear(stateArr, entryIndex)) { //check if slot is empty
- return entryIndex;
- }
- entryIndex = (entryIndex + stride) % tableEntries;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("No empty slots.");
- }
-
- //This method is specifically tied to the HLL array layout
- @SuppressFBWarnings(value = "IM_MULTIPLYING_RESULT_OF_IREM", justification = "False Positive")
- private final boolean updateHll(final int entryIndex, final int coupon) {
- final int newValue = coupon16Value(coupon);
-
- final int hllIdx = coupon & (k_ - 1); //lower lgK bits
- final int longIdx = hllIdx / 10;
- final int shift = ((hllIdx % 10) * 6) & SIX_BIT_MASK;
-
- long hllLong = arrOfHllArr_[(entryIndex * hllArrLongs_) + longIdx];
- final int oldValue = (int)(hllLong >>> shift) & SIX_BIT_MASK;
- if (newValue <= oldValue) { return false; }
- // newValue > oldValue
-
- //update hipEstAccum BEFORE updating invPow2Sum
- final double invPow2Sum = invPow2SumHiArr_[entryIndex] + invPow2SumLoArr_[entryIndex];
- final double oneOverQ = k_ / invPow2Sum;
- hipEstAccumArr_[entryIndex] += oneOverQ;
-
- //update invPow2Sum
- if (oldValue < 32) { invPow2SumHiArr_[entryIndex] -= invPow2(oldValue); }
- else { invPow2SumLoArr_[entryIndex] -= invPow2(oldValue); }
- if (newValue < 32) { invPow2SumHiArr_[entryIndex] += invPow2(newValue); }
- else { invPow2SumLoArr_[entryIndex] += invPow2(newValue); }
-
- //insert the new value
- hllLong &= ~(0X3FL << shift); //zero out the 6-bit field
- hllLong |= ((long)newValue) << shift; //insert
- arrOfHllArr_[(entryIndex * hllArrLongs_) + longIdx] = hllLong;
- return true;
- }
-
- private final void resize() {
- final int newTableEntries = nextPrime((int)(tableEntries_ * growthFactor_));
- final int newCapacityEntries = (int)(newTableEntries * LOAD_FACTOR);
-
- final byte[] newKeysArr = new byte[newTableEntries * keySizeBytes_];
- final long[] newArrOfHllArr = new long[newTableEntries * hllArrLongs_];
- final double[] newInvPow2Sum1 = new double[newTableEntries];
- final double[] newInvPow2Sum2 = new double[newTableEntries];
- final double[] newHipEstAccum = new double[newTableEntries];
- final byte[] newStateArr = new byte[(int) Math.ceil(newTableEntries / 8.0)];
-
- for (int oldIndex = 0; oldIndex < tableEntries_; oldIndex++) {
- if (isBitClear(stateArr_, oldIndex)) { continue; }
- // extract an old key
- final byte[] key =
- Arrays.copyOfRange(keysArr_, oldIndex * keySizeBytes_, (oldIndex + 1) * keySizeBytes_);
- final int newIndex = findEmpty(key, newTableEntries, newStateArr);
- System.arraycopy(key, 0, newKeysArr, newIndex * keySizeBytes_, keySizeBytes_); //put key
- //put the rest of the row
- System.arraycopy(arrOfHllArr_, oldIndex * hllArrLongs_, newArrOfHllArr,
- newIndex * hllArrLongs_, hllArrLongs_);
- newInvPow2Sum1[newIndex] = invPow2SumHiArr_[oldIndex];
- newInvPow2Sum2[newIndex] = invPow2SumLoArr_[oldIndex];
- newHipEstAccum[newIndex] = hipEstAccumArr_[oldIndex];
- setBit(newStateArr, newIndex);
- }
- //restore into sketch
- tableEntries_ = newTableEntries;
- capacityEntries_ = newCapacityEntries;
- //curCountEntries_, growthFactor_ unchanged
- entrySizeBytes_ = updateEntrySizeBytes(tableEntries_, keySizeBytes_, hllArrLongs_);
-
- keysArr_ = newKeysArr;
- arrOfHllArr_ = newArrOfHllArr;
- invPow2SumHiArr_ = newInvPow2Sum1; //init to k
- invPow2SumLoArr_ = newInvPow2Sum2; //init to 0
- hipEstAccumArr_ = newHipEstAccum; //init to 0
- stateArr_ = newStateArr;
- }
-
- private static final double updateEntrySizeBytes(final int tableEntries, final int keySizeBytes,
- final int hllArrLongs) {
- final double byteFraction = Math.ceil(tableEntries / 8.0) / tableEntries;
- return keySizeBytes + ((double) hllArrLongs * Long.BYTES) + (3.0 * Double.BYTES) + byteFraction;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hllmap/Map.java b/src/main/java/org/apache/datasketches/hllmap/Map.java
deleted file mode 100644
index df730e3d4..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/Map.java
+++ /dev/null
@@ -1,260 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hllmap;
-
-import static org.apache.datasketches.common.Util.LS;
-
-import java.math.BigInteger;
-
-import org.apache.datasketches.hash.MurmurHash3;
-
-/**
- * Base class and API for all the maps.
- *
- * @author Lee Rhodes
- * @author Alexander Saydakov
- * @author Kevin Lang
- */
-abstract class Map {
-
- static final long SEED = 1234567890L;
- static final int SIX_BIT_MASK = 0X3F; // 6 bits
- static final int TEN_BIT_MASK = 0X3FF; //10 bits
-
- // These parameters are tuned as a set to avoid pathological resizing.
- // Consider modeling the behavior before changing any of them
- static final int COUPON_MAP_MIN_NUM_ENTRIES = 157;
- static final double COUPON_MAP_SHRINK_TRIGGER_FACTOR = 0.5;
- static final double COUPON_MAP_GROW_TRIGGER_FACTOR = 15.0 / 16.0;
- static final double COUPON_MAP_TARGET_FILL_FACTOR = 2.0 / 3.0;
- static final int COUPON_MAP_MIN_NUM_ENTRIES_ARR_SIZE = (int)Math.ceil(COUPON_MAP_MIN_NUM_ENTRIES / 8.0);
-
- final int keySizeBytes_;
-
- Map(final int keySizeBytes) {
- keySizeBytes_ = keySizeBytes;
- }
-
- /**
- * Update this map with a key and a coupon.
- * Return the cardinality estimate of all identifiers that have been associated with this key,
- * including this update.
- * @param key the dimensional criteria for measuring cardinality
- * @param coupon the property associated with the key for which cardinality is to be measured.
- * @return the cardinality estimate of all identifiers that have been associated with this key,
- * including this update.
- */
- abstract double update(byte[] key, short coupon);
-
- /**
- * Updates this map with an index and a coupon
- * @param index the given index
- * @param coupon the given coupon
- * @return the cardinality estimate of all identifiers that have been associated with this key,
- * including this update.
- */
- abstract double update(int index, short coupon);
-
- /**
- * Returns the estimate of the cardinality of identifiers associated with the given key.
- * @param key the given key
- * @return the estimate of the cardinality of identifiers associated with the given key.
- */
- abstract double getEstimate(byte[] key);
-
- /**
- * Update the internal estimate at the given index
- * @param index the given index
- * @param estimate the given estimate
- */
- void updateEstimate(final int index, final double estimate) { /* overridden by sub-classes */ }
-
- /**
- * Returns the upper bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key. This approximates the RSE with 68% confidence.
- * @param key the given key
- * @return the upper bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key.
- */
- abstract double getUpperBound(byte[] key);
-
- /**
- * Returns the lower bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key. This approximates the RSE with 68% confidence.
- * @param key the given key
- * @return the lower bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key.
- */
- abstract double getLowerBound(byte[] key);
-
- abstract int findKey(byte[] key);
-
- abstract int findOrInsertKey(byte[] key);
-
- abstract CouponsIterator getCouponsIterator(int index);
-
- abstract int getMaxCouponsPerEntry();
-
- abstract int getCapacityCouponsPerEntry();
-
- abstract int getActiveEntries();
-
- abstract int getDeletedEntries();
-
- abstract double getEntrySizeBytes();
-
- abstract int getTableEntries();
-
- abstract int getCapacityEntries();
-
- abstract int getCurrentCountEntries();
-
- abstract long getTotalUsageBytes();
-
- int getKeySizeBytes() {
- return keySizeBytes_;
- }
-
- /**
- * Delete the key at the given index
- * @param index the given index
- */
- void deleteKey(final int index) { /* overridden by sub-classes */ }
-
- /**
- * Returns The space consumed by this map is quite sensitive to the actual distribution of identifiers
- * per key, so you should characterize and or experiment with your typical input streams.
- * Nonetheless, our experiments on live streams of over 100M keys required about 1.4GB of space.
- * This is about 14 bytes per key for key storage and unique count storage.
- *
- * Given such highly-skewed distributions, using this map is far more efficient space-wise than
- * the alternative of dedicating an HLL sketch per key. Based on our use cases, after
- * subtracting the space required for key storage, the average bytes per key required for unique
- * count estimation ({@link #getAverageSketchBytesPerKey()}) is about 10.
- *
- * Internally, this map is implemented as a hierarchy of internal hash maps with progressively
- * increasing storage allocated for unique count estimation. As a key acquires more identifiers it
- * is "promoted" up to a higher internal map. The final map of keys is a map of compact HLL
- * sketches.
- *
- * The unique values in all the internal maps, except the final HLL map, are stored in a special
- * form called a coupon. A coupon is a 16-bit value that fully describes a k=1024 HLL bin.
- * It contains 10 bits of address and a 6-bit HLL value.
- *
- * All internal maps use a prime number size and Knuth's Open Addressing Double Hash (OADH)
- * search algorithm.
- *
- * The internal base map holds all the keys and each key is associated with one 16-bit value.
- * Initially, the value is a single coupon. Once the key is promoted, this 16-bit field contains a
- * reference to the internal map where the key is still active.
- *
- * The intermediate maps between the base map and the final HLL map are of two types.
- * The first few of these are called traverse maps where the coupons are
- * stored as unsorted arrays. After the traverse maps are the coupon hash maps, where the coupons
- * are stored in small OASH hash tables.
- *
- * All the intermediate maps support deletes and can dynamically grow and shrink as required by
- * the input stream.
- *
- * The sketch estimator algorithms are unbiased with a Relative Standard Error (RSE)
- * of about 2.6% with 68% confidence, or equivalently, about 5.2% with a 95% confidence.
- *
- * In a parallel package in the sketches-misc repository, there are 2 classes that can be used
- * from the command line to feed this mapping sketch piped from standard-in for experimental
- * evaluation. The first is ProcessIpStream, which processes simple IP/ID pairs and the second,
- * ProcessDistributionStream, which processes pairs that describe a distribution.
- * In this same package is the VariousMapRSETest class that was used to generate the error plots
- * for the web site. Please refer to the javadocs for those classes for more information.
- *
- * @author Lee Rhodes
- * @author Alexander Saydakov
- * @author Kevin Lang
- */
-public final class UniqueCountMap {
- private static final int NUM_LEVELS = 10; // total of single coupon + traverse + coupon maps + hll
- private static final int NUM_TRAVERSE_MAPS = 3;
- private static final int HLL_K = 1024;
- private static final int INITIAL_NUM_ENTRIES = 1000003;
- private static final int MIN_INITIAL_NUM_ENTRIES = 157;
- private final int keySizeBytes_;
-
- /** TraverseCouponMap or HashCouponMap instances */
- private final Map[] maps_;
-
- /**
- * Constructs a UniqueCountMap with an initial capacity of one million entries.
- * @param keySizeBytes must be at least 4 bytes to have sufficient entropy.
- */
- public UniqueCountMap(final int keySizeBytes) {
- this(INITIAL_NUM_ENTRIES, keySizeBytes);
- }
-
- /**
- * Constructs a UniqueCountMap with a given initial number of entries.
- *
- * @param initialNumEntries The initial number of entries provides a tradeoff between
- * wasted space, if too high, and wasted time resizing the table, if too low.
- * @param keySizeBytes must be at least 4 bytes to have sufficient entropy
- */
- public UniqueCountMap(final int initialNumEntries, final int keySizeBytes) {
- checkConstructorKeySize(keySizeBytes);
- final int initEntries = Math.max(initialNumEntries, MIN_INITIAL_NUM_ENTRIES);
- keySizeBytes_ = keySizeBytes;
- maps_ = new Map[NUM_LEVELS]; // includes base level and top level
- maps_[0] = SingleCouponMap.getInstance(initEntries, keySizeBytes);
- }
-
- /**
- * Updates the map with a given key and identifier and returns the estimate of the number of
- * unique identifiers encountered so far for the given key.
- * @param key the given key
- * @param identifier the given identifier for unique counting associated with the key
- * @return the estimate of the number of unique identifiers encountered so far for the given key.
- */
- public double update(final byte[] key, final byte[] identifier) {
- if (key == null) { return Double.NaN; }
- checkMethodKeySize(key);
- if (identifier == null) { return getEstimate(key); }
- final short coupon = (short) Map.coupon16(identifier);
-
- final int baseMapIndex = maps_[0].findOrInsertKey(key);
- final double baseMapEstimate = maps_[0].update(baseMapIndex, coupon);
- if (baseMapEstimate > 0) { return baseMapEstimate; }
- final int level = -(int) baseMapEstimate; // base map is level 0
- if (level == 0) {
- return promote(key, coupon, maps_[0], baseMapIndex, level, baseMapIndex, 0);
- }
-
- final Map map = maps_[level];
- final int index = map.findOrInsertKey(key);
- final double estimate = map.update(index, coupon);
- if (estimate > 0) { return estimate; }
- return promote(key, coupon, map, index, level, baseMapIndex, -estimate);
- }
-
- /**
- * Retrieves the current estimate of unique count for a given key.
- * @param key given key
- * @return estimate of unique count so far
- */
- public double getEstimate(final byte[] key) {
- if (key == null) { return Double.NaN; }
- checkMethodKeySize(key);
- final double est = maps_[0].getEstimate(key);
- if (est >= 0.0) { return est; }
- //key has been promoted
- final int level = -(int)est;
- final Map map = maps_[level];
- return map.getEstimate(key);
- }
-
- /**
- * Returns the upper bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key.
- * @param key the given key
- * @return the upper bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key.
- */
- public double getUpperBound(final byte[] key) {
- if (key == null) { return Double.NaN; }
- checkMethodKeySize(key);
- final double est = maps_[0].getEstimate(key);
- if (est >= 0.0) { return est; }
- //key has been promoted
- final int level = -(int)est;
- final Map map = maps_[level];
- return map.getUpperBound(key);
- }
-
- /**
- * Returns the lower bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key.
- * @param key the given key
- * @return the lower bound cardinality with respect to {@link #getEstimate(byte[])} associated
- * with the given key.
- */
- public double getLowerBound(final byte[] key) {
- if (key == null) { return Double.NaN; }
- checkMethodKeySize(key);
- final double est = maps_[0].getEstimate(key);
- if (est >= 0.0) { return est; }
- //key has been promoted
- final int level = -(int)est;
- final Map map = maps_[level];
- return map.getLowerBound(key);
- }
-
- /**
- * Returns the number of active, unique keys across all internal maps
- * @return the number of active, unique keys across all internal maps
- */
- public int getActiveEntries() {
- return maps_[0].getCurrentCountEntries();
- }
-
- /**
- * Returns total bytes used by all internal maps
- * @return total bytes used by all internal maps
- */
- public long getTotalUsageBytes() {
- long total = 0;
- for (int i = 0; i < maps_.length; i++) {
- if (maps_[i] != null) {
- total += maps_[i].getTotalUsageBytes();
- }
- }
- return total;
- }
-
- /**
- * Returns total bytes used for key storage
- * @return total bytes used for key storage
- */
- public long getKeyUsageBytes() {
- long total = 0;
- for (int i = 0; i < maps_.length; i++) {
- if (maps_[i] != null) {
- total += (long) (maps_[i].getActiveEntries()) * keySizeBytes_;
- }
- }
- return total;
- }
-
- /**
- * Returns the average bytes storage per key that is dedicated to sketching the unique counts.
- * @return the average bytes storage per key that is dedicated to sketching the unique counts.
- */
- public double getAverageSketchBytesPerKey() {
- return (double) (getTotalUsageBytes() - getKeyUsageBytes()) / getActiveEntries();
- }
-
- /**
- * Returns the number of active internal maps so far.
- * Only the base map is initialized in the constructor, so this method would return 1.
- * As more keys are promoted up to higher level maps, the return value would grow until the
- * last level HLL map is allocated.
- * @return the number of active levels so far
- */
- int getActiveMaps() {
- int levels = 0;
- final int iMapsLen = maps_.length;
- for (int i = 0; i < iMapsLen; i++) {
- if (maps_[i] != null) { levels++; }
- }
- return levels;
- }
-
- /**
- * Returns the base map
- * @return the base map
- */
- Map getBaseMap() {
- return maps_[0];
- }
-
- /**
- * Returns the top-level HllMap. It may be null.
- * @return the top-level HllMap.
- */
- Map getHllMap() {
- return maps_[maps_.length - 1];
- }
-
- /**
- * Returns a string with a human-readable summary of the UniqueCountMap and all the internal maps
- * @return human-readable summary
- */
- @Override
- public String toString() {
- final long totKeys = getActiveEntries();
- final long totBytes = getTotalUsageBytes();
- final long keyBytes = getKeyUsageBytes();
- final double avgValBytesPerKey = getAverageSketchBytesPerKey();
-
- final String ksb = Map.fmtLong(keySizeBytes_);
- final String alvls = Map.fmtLong(getActiveMaps());
- final String tKeys = Map.fmtLong(totKeys);
- final String tBytes = Map.fmtLong(totBytes);
- final String kBytes = Map.fmtLong(keyBytes);
- final String avgValBytes = Map.fmtDouble(avgValBytesPerKey);
-
-
- final StringBuilder sb = new StringBuilder();
- final String thisSimpleName = this.getClass().getSimpleName();
- sb.append("## ").append(thisSimpleName).append(" SUMMARY: ").append(LS);
- sb.append(" Key Size Bytes : ").append(ksb).append(LS);
- sb.append(" Active Map Levels : ").append(alvls).append(LS);
- sb.append(" Total keys : ").append(tKeys).append(LS);
- sb.append(" Total Usage Bytes : ").append(tBytes).append(LS);
- sb.append(" Total Key Usage Bytes : ").append(kBytes).append(LS);
- sb.append(" Avg Sketch Usage Bytes/Key : ").append(avgValBytes).append(LS);
- sb.append(LS);
- for (int i = 0; i < maps_.length; i++) {
- final Map cMap = maps_[i];
- if (cMap != null) {
- sb.append(cMap.toString());
- sb.append(LS);
- }
- }
- sb.append("## ").append("END UNIQUE COUNT MAP SUMMARY");
- sb.append(LS);
- return sb.toString();
- }
-
- private void setLevelInBaseMap(final int index, final int level) {
- ((SingleCouponMap) maps_[0]).setLevel(index, level);
- }
-
- private double promote(final byte[] key, final short coupon, final Map fromMap, final int fromIndex,
- final int fromLevel, final int baseMapIndex, final double estimate) {
- final Map newMap = getMapForLevel(fromLevel + 1);
- final int newMapIndex = newMap.findOrInsertKey(key);
- final CouponsIterator it = fromMap.getCouponsIterator(fromIndex);
- while (it.next()) {
- final double est = newMap.update(newMapIndex, it.getValue());
- assert est > 0;
- }
- fromMap.deleteKey(fromIndex);
- newMap.updateEstimate(newMapIndex, estimate);
- final double newEstimate = newMap.update(newMapIndex, coupon);
- setLevelInBaseMap(baseMapIndex, fromLevel + 1);
- assert newEstimate > 0; // this must be positive since we have just promoted
- return newEstimate;
- }
-
- private Map getMapForLevel(final int level) {
- if (maps_[level] == null) {
- final int newLevelCapacity = 1 << level;
- if (level <= NUM_TRAVERSE_MAPS) {
- maps_[level] = CouponTraverseMap.getInstance(keySizeBytes_, newLevelCapacity);
- } else if (level < (maps_.length - 1)) {
- maps_[level] = CouponHashMap.getInstance(keySizeBytes_, newLevelCapacity);
- } else {
- maps_[level] = HllMap.getInstance(keySizeBytes_, HLL_K);
- }
- }
- return maps_[level];
- }
-
- private static final void checkConstructorKeySize(final int keySizeBytes) {
- if (keySizeBytes < 4) {
- throw new SketchesArgumentException("KeySizeBytes must be >= 4: " + keySizeBytes);
- }
- }
-
- private final void checkMethodKeySize(final byte[] key) {
- if (key.length != keySizeBytes_) {
- throw new SketchesArgumentException("Key size must be " + keySizeBytes_ + " bytes.");
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hllmap/package-info.java b/src/main/java/org/apache/datasketches/hllmap/package-info.java
deleted file mode 100644
index 20428b04f..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/package-info.java
+++ /dev/null
@@ -1,38 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * The hllmap package contains a space efficient HLL mapping sketch of keys to approximate unique
- * count of identifiers. For example, counting the number of unique users (identifiers) per IP
- * address.
- *
- * In cases where the number of keys is very large, having an individual HLL sketch per key may
- * not be practical. If the distribution of values per key is highly skewed where the vast
- * majority of keys have only a few values then this mapping sketch will make sense as it will be
- * far more space efficient than dedicating individual HLL sketches per key.
- *
- * From our own testing, sketching 100 million IPv4 addresses with such a
- * highly skewed distribution of identifiers per IP uses only 1.4GB of memory. This translates to
- * an average of about 10 bytes per IP allocated to the equivalent of a full k=1024 HLL sketch
- * and provides an RSE of less than 2.5%. Your results will vary depending on the actual
- * distribution of identifiers per key.
- *
- * @see org.apache.datasketches.hllmap.UniqueCountMap
- */
-package org.apache.datasketches.hllmap;
diff --git a/src/main/java/org/apache/datasketches/kll/KllDirectCompactItemsSketch.java b/src/main/java/org/apache/datasketches/kll/KllDirectCompactItemsSketch.java
deleted file mode 100644
index 477a7fc5e..000000000
--- a/src/main/java/org/apache/datasketches/kll/KllDirectCompactItemsSketch.java
+++ /dev/null
@@ -1,307 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.kll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.kll.KllPreambleUtil.DATA_START_ADR;
-import static org.apache.datasketches.kll.KllPreambleUtil.DATA_START_ADR_SINGLE_ITEM;
-import static org.apache.datasketches.kll.KllPreambleUtil.getMemorySegmentK;
-import static org.apache.datasketches.kll.KllPreambleUtil.getMemorySegmentLevelZeroSortedFlag;
-import static org.apache.datasketches.kll.KllPreambleUtil.getMemorySegmentM;
-import static org.apache.datasketches.kll.KllPreambleUtil.getMemorySegmentMinK;
-import static org.apache.datasketches.kll.KllPreambleUtil.getMemorySegmentN;
-import static org.apache.datasketches.kll.KllSketch.SketchStructure.COMPACT_EMPTY;
-import static org.apache.datasketches.kll.KllSketch.SketchStructure.COMPACT_SINGLE;
-
-import java.lang.foreign.MemorySegment;
-import java.lang.reflect.Array;
-import java.util.Comparator;
-
-import org.apache.datasketches.common.ArrayOfBooleansSerDe;
-import org.apache.datasketches.common.ArrayOfItemsSerDe;
-import org.apache.datasketches.common.MemorySegmentRequest;
-import org.apache.datasketches.common.MemorySegmentStatus;
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * This class implements an off-heap, read-only KllItemsSketch using MemorySegment.
- *
- * Please refer to the documentation in the package-info: Please refer to the documentation in the package-info: Please refer to the documentation in the package-info: Please refer to the documentation in the package-info: Here is what we do for each level: It can be proved that generalCompress returns a sketch that satisfies the space constraints
- * no matter how much data is passed in.
- * We are pretty sure that it works correctly when inBuf and outBuf are the same.
- * All levels except for level zero must be sorted before calling this, and will still be
- * sorted afterwards.
- * Level zero is not required to be sorted before, and may not be sorted afterwards. This trashes inBuf and inLevels and modifies outBuf and outLevels. The given MemorySegment must be writable and it must contain a KllDoublesSketch in updatable form.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. The given MemorySegment must be writable and it must contain a KllDoublesSketch in updatable form.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. It is up to the user to optionally extend this interface if more flexible
- * handling of requests for more capacity is required. The parameter k will not change. Note: a single occurrence of a NaN in the array will force this method to use the conventional update path
- * rather than the fast update path. Here is what we do for each level: It can be proved that generalCompress returns a sketch that satisfies the space constraints
- * no matter how much data is passed in.
- * We are pretty sure that it works correctly when inBuf and outBuf are the same.
- * All levels except for level zero must be sorted before calling this, and will still be
- * sorted afterwards.
- * Level zero is not required to be sorted before, and may not be sorted afterwards. This trashes inBuf and inLevels and modifies outBuf and outLevels. The given MemorySegment must be writable and it must contain a KllFloatsSketch in updatable form.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. The given MemorySegment must be writable and it must contain a KllFloatsSketch in updatable form.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. It is up to the user to optionally extend this interface if more flexible
- * handling of requests for more capacity is required. The parameter k will not change. Note: a single occurrence of a NaN in the array will force this method to use the conventional update path
- * rather than the fast update path. Please refer to the documentation in the package-info: Please refer to the documentation in the package-info: Please refer to the documentation in the package-info: Please refer to the documentation in the package-info: Here is what we do for each level: It can be proved that generalCompress returns a sketch that satisfies the space constraints
- * no matter how much data is passed in.
- * We are pretty sure that it works correctly when inBuf and outBuf are the same.
- * All levels except for level zero must be sorted before calling this, and will still be
- * sorted afterwards.
- * Level zero is not required to be sorted before, and may not be sorted afterwards. This trashes inBuf and inLevels and modifies outBuf and outLevels. Here is what we do for each level: It can be proved that generalCompress returns a sketch that satisfies the space constraints
- * no matter how much data is passed in.
- * We are pretty sure that it works correctly when inBuf and outBuf are the same.
- * All levels except for level zero must be sorted before calling this, and will still be
- * sorted afterwards.
- * Level zero is not required to be sorted before, and may not be sorted afterwards. This trashes inBuf and inLevels and modifies outBuf and outLevels. The given MemorySegment must be writable and it must contain a KllLongsSketch in updatable form.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. The given MemorySegment must be writable and it must contain a KllLongsSketch in updatable form.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. It is up to the user to optionally extend this interface if more flexible
- * handling of requests for more capacity is required. The parameter k will not change. The intent of the design of this class was to isolate the detailed knowledge of the bit and
- * byte layout of the serialized form of the sketches derived from the base sketch classes into one place.
- * This allows the possibility of the introduction of different serialization
- * schemes with minimal impact on the rest of the library. KLL is an implementation of a very compact quantiles sketch with lazy compaction scheme
- * and nearly optimal accuracy per retained quantile. Reference Optimal Quantile Approximation in Streams. The default k of 200 yields a "single-sided" epsilon of about 1.33% and a
- * "double-sided" (PMF) epsilon of about 1.65%, with a confidence of 99%. Prototype example of the recommended iteration loop: The code included here does work fine for moderate sized partitioning tasks.
- * As an example, using the test code in the test branch with the partitioning task of splitting
- * a data set of 1 billion items into 324 partitions of size 3M items completed in under 3 minutes, which was
- * performed on a single CPU. For much larger partitioning tasks, it is recommended that this code be leveraged into a
- * parallelized systems environment. This range of data may or may not be subsequently further partitioned. Note: It is easy to prove that the following simplified code which launches multiple waves of
- * carry propagation does exactly the same amount of merging work (including the work of
- * allocating fresh buffers) as the more complicated and seemingly more efficient approach that
- * tracks a single carry propagation wave through both sketches.
- *
- * This simplified code probably does do slightly more "outer loop" work, but I am pretty
- * sure that even that is within a constant factor of the more complicated code, plus the
- * total amount of "outer loop" work is at least a factor of K smaller than the total amount of
- * merging work, which is identical in the two approaches.
- *
- * Note: a two-way merge that doesn't modify either of its two inputs could be implemented
- * by making a deep copy of the larger sketch and then merging the smaller one into it.
- * However, it was decided not to do this.
- *
- * @param src The source sketch
- * @param tgt The target sketch
- */
- static void mergeInto(final QuantilesDoublesSketch src, final UpdatableQuantilesDoublesSketch tgt) {
- final int srcK = src.getK();
- final int tgtK = tgt.getK();
- final long srcN = src.getN();
- final long tgtN = tgt.getN();
-
- if (srcK != tgtK) {
- downSamplingMergeInto(src, tgt);
- return;
- }
- //The remainder of this code is for the case where the k's are equal
-
- final DoublesSketchAccessor srcSketchBuf = DoublesSketchAccessor.wrap(src, false);
- final long nFinal = tgtN + srcN;
-
- for (int i = 0; i < srcSketchBuf.numItems(); i++) { // update only the base buffer
- tgt.update(srcSketchBuf.get(i));
- }
-
- final int spaceNeeded = DoublesUpdateImpl.getRequiredItemCapacity(tgtK, nFinal);
- final int tgtCombBufItemCap = tgt.getCombinedBufferItemCapacity();
- if (spaceNeeded > tgtCombBufItemCap) { //copies base buffer plus current levels
- tgt.growCombinedBuffer(tgtCombBufItemCap, spaceNeeded);
- }
-
- final DoublesArrayAccessor scratch2KAcc = DoublesArrayAccessor.initialize(2 * tgtK);
-
- long srcBitPattern = src.getBitPattern();
- assert srcBitPattern == (srcN / (2L * srcK));
-
- final DoublesSketchAccessor tgtSketchBuf = DoublesSketchAccessor.wrap(tgt, true);
- long newTgtBitPattern = tgt.getBitPattern();
-
- for (int srcLvl = 0; srcBitPattern != 0L; srcLvl++, srcBitPattern >>>= 1) {
- if ((srcBitPattern & 1L) > 0L) {
- newTgtBitPattern = DoublesUpdateImpl.inPlacePropagateCarry(
- srcLvl,
- srcSketchBuf.setLevel(srcLvl),
- scratch2KAcc,
- false,
- tgtK,
- tgtSketchBuf,
- newTgtBitPattern
- );
- }
- }
-
- if (tgt.hasMemorySegment() && (nFinal > 0)) {
- final MemorySegment seg = tgt.getMemorySegment();
- clearBits(seg, FLAGS_BYTE, (byte) EMPTY_FLAG_MASK);
- }
-
- tgt.putN(nFinal);
- tgt.putBitPattern(newTgtBitPattern); // no-op if direct
-
- assert (tgt.getN() / (2L * tgtK)) == tgt.getBitPattern(); // internal consistency check
-
- double srcMax = src.getMaxItem();
- srcMax = Double.isNaN(srcMax) ? Double.NEGATIVE_INFINITY : srcMax;
- double srcMin = src.getMinItem();
- srcMin = Double.isNaN(srcMin) ? Double.POSITIVE_INFINITY : srcMin;
-
- double tgtMax = tgt.getMaxItem();
- tgtMax = Double.isNaN(tgtMax) ? Double.NEGATIVE_INFINITY : tgtMax;
- double tgtMin = tgt.getMinItem();
- tgtMin = Double.isNaN(tgtMin) ? Double.POSITIVE_INFINITY : tgtMin;
-
- tgt.putMaxItem(Math.max(srcMax, tgtMax));
- tgt.putMinItem(Math.min(srcMin, tgtMin));
- }
-
- /**
- * Merges the source sketch into the target sketch that can have a smaller K.
- * However, it is required that the ratio of the two K's be a power of 2.
- * I.e., source.getK() = target.getK() * 2^(nonnegative integer).
- * The source is not modified.
- *
- * @param src The source sketch
- * @param tgt The target sketch
- */
- //also used by QuantilesDoublesSketch, QuantilesDoublesUnionImpl and HeapDoublesSketchTest
- static void downSamplingMergeInto(final QuantilesDoublesSketch src, final UpdatableQuantilesDoublesSketch tgt) {
- final int sourceK = src.getK();
- final int targetK = tgt.getK();
- final long tgtN = tgt.getN();
-
- if ((sourceK % targetK) != 0) {
- throw new SketchesArgumentException(
- "source.getK() must equal target.getK() * 2^(nonnegative integer).");
- }
-
- final int downFactor = sourceK / targetK;
- checkIfPowerOf2(downFactor, "source.getK()/target.getK() ratio");
- final int lgDownFactor = Integer.numberOfTrailingZeros(downFactor);
-
- if (src.isEmpty()) { return; }
-
- final DoublesSketchAccessor srcSketchBuf = DoublesSketchAccessor.wrap(src, false);
- final long nFinal = tgtN + src.getN();
-
- for (int i = 0; i < srcSketchBuf.numItems(); i++) { // update only the base buffer
- tgt.update(srcSketchBuf.get(i));
- }
-
- final int spaceNeeded = DoublesUpdateImpl.getRequiredItemCapacity(targetK, nFinal);
- final int curCombBufCap = tgt.getCombinedBufferItemCapacity();
- if (spaceNeeded > curCombBufCap) { //copies base buffer plus current levels
- tgt.growCombinedBuffer(curCombBufCap, spaceNeeded);
- }
-
- //working scratch buffers
- final DoublesArrayAccessor scratch2KAcc = DoublesArrayAccessor.initialize(2 * targetK);
- final DoublesArrayAccessor downScratchKAcc = DoublesArrayAccessor.initialize(targetK);
-
- final DoublesSketchAccessor tgtSketchBuf = DoublesSketchAccessor.wrap(tgt, true);
-
- long srcBitPattern = src.getBitPattern();
- long newTgtBitPattern = tgt.getBitPattern();
- for (int srcLvl = 0; srcBitPattern != 0L; srcLvl++, srcBitPattern >>>= 1) {
- if ((srcBitPattern & 1L) > 0L) {
- justZipWithStride(
- srcSketchBuf.setLevel(srcLvl),
- downScratchKAcc,
- targetK,
- downFactor
- );
- newTgtBitPattern = DoublesUpdateImpl.inPlacePropagateCarry(
- srcLvl + lgDownFactor, //starting level
- downScratchKAcc, //optSrcKBuf,
- scratch2KAcc, //size2KBuf,
- false, //do mergeInto version
- targetK,
- tgtSketchBuf,
- newTgtBitPattern
- );
-
- tgt.putBitPattern(newTgtBitPattern); //off-heap is a no-op
- }
- }
- if (tgt.hasMemorySegment() && (nFinal > 0)) {
- final MemorySegment seg = tgt.getMemorySegment();
- clearBits(seg, FLAGS_BYTE, (byte) EMPTY_FLAG_MASK);
- }
- tgt.putN(nFinal);
-
- assert (tgt.getN() / (2L * targetK)) == newTgtBitPattern; // internal consistency check
-
- double srcMax = src.getMaxItem();
- srcMax = Double.isNaN(srcMax) ? Double.NEGATIVE_INFINITY : srcMax;
- double srcMin = src.getMinItem();
- srcMin = Double.isNaN(srcMin) ? Double.POSITIVE_INFINITY : srcMin;
-
- double tgtMax = tgt.getMaxItem();
- tgtMax = Double.isNaN(tgtMax) ? Double.NEGATIVE_INFINITY : tgtMax;
- double tgtMin = tgt.getMinItem();
- tgtMin = Double.isNaN(tgtMin) ? Double.POSITIVE_INFINITY : tgtMin;
-
- if (srcMax > tgtMax) { tgt.putMaxItem(srcMax); }
- if (srcMin < tgtMin) { tgt.putMinItem(srcMin); }
- }
-
- private static void justZipWithStride(
- final DoublesBufferAccessor bufA, // input
- final DoublesBufferAccessor bufC, // output
- final int kC, // number of items that should be in the output
- final int stride) {
- final int randomOffset = QuantilesDoublesSketch.rand.nextInt(stride);
- for (int a = randomOffset, c = 0; c < kC; a += stride, c++ ) {
- bufC.set(c, bufA.get(a));
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/quantiles/DoublesSketchAccessor.java b/src/main/java/org/apache/datasketches/quantiles/DoublesSketchAccessor.java
deleted file mode 100644
index 4c4bdd5cd..000000000
--- a/src/main/java/org/apache/datasketches/quantiles/DoublesSketchAccessor.java
+++ /dev/null
@@ -1,178 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.quantiles;
-
-import static org.apache.datasketches.quantiles.PreambleUtil.COMBINED_BUFFER;
-
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * This allows access to package-private levels and data in whatever doubles quantiles sketch you give
- * it: on-heap, off-heap; compact and non-compact
- * @author Jon Malkin
- */
-abstract class DoublesSketchAccessor extends DoublesBufferAccessor {
- static final int BB_LVL_IDX = -1;
-
- final QuantilesDoublesSketch ds_;
- final boolean forceSize_;
-
- long n_;
- int currLvl_;
- int numItems_;
- int offset_; //bytes for Direct, doubles for heap
-
- DoublesSketchAccessor(
- final QuantilesDoublesSketch ds,
- final boolean forceSize,
- final int level) {
- checkLvl(level);
- ds_ = ds;
- forceSize_ = forceSize;
- setLevel(level);
- }
-
- private static final void checkLvl(final int level) {
- if ((level != BB_LVL_IDX) && (level < 0)) {
- throw new SketchesArgumentException("Parameter level is < 0.");
- }
- }
-
- /**
- * Wrap the given QuantilesDoublesSketch
- * @param ds the given QuantilesDoublesSketch
- * @param forceSize Generally, this must be true if modeling an updatable sketch, i.e., not compact.
- * See {@link #setLevel(int) setLevel(int)} below.
- *
- * @return this
- */
- static DoublesSketchAccessor wrap(final QuantilesDoublesSketch ds, final boolean forceSize) {
- if (ds.hasMemorySegment()) {
- return new DirectDoublesSketchAccessor(ds, forceSize, BB_LVL_IDX);
- }
- return new HeapDoublesSketchAccessor(ds, forceSize, BB_LVL_IDX);
- }
-
- abstract DoublesSketchAccessor copyAndSetLevel(final int level);
-
- /*
- * This initializes the following internal member variables:
- * If the constructor parameter forceSize is true and accessing the BaseBuffer,
- * it forces the numItems_ to be 2K independent of the actual getBaseBufferCount(). If forceSize is false and accessing the BaseBuffer, numItems_ is set to getBaseBufferCount().
- *
- * If forceSize is true and accessing a higher level, the numItems_ is always K. If forceSize is false and accessing a higher level, numItems_ is K
- * only if the level is valid (i.e., a corresponding bit in bitPattern is one), otherwise it is zero. Prior to this method being called, any extra space for the combined buffer required
- * by either the update or merge operations must already be allocated. Update Version: The base buffer is initially full, and after it has been sorted and
- * zipped, will be used as a size2KBuf scratch buffer for the remaining recursive carries.
- * The lowest non-valid level, determined by the bit-pattern, will used internally as a
- * size K scratch buffer and the ultimate target.
- * Thus no additional buffer storage is required outside the combined buffer. Merge Version: During merging, each level from the source sketch that must be
- * merged is entered into this method and is assigned to the optional source size K buffer
- * (optSrcKBuf). Because the base buffer may have data, a separate size2K
- * scratch buffer must be provided. The next-lowest. non-valid level, determined by the
- * bit-pattern, will used as a sizeKBuf scratch buffer. Downsample Merge Version: This is a variant of the above Merge Version, except at
- * each level the downsampling is performed and the target level is computed for the target merge.
- * In this case the optSrcKBuf is the result of the downsample process and needs to be allocated
- * for that purpose.
- *
- * Recursive carry: This starts with a given sorted, size 2K buffer, which is zipped
- * into a size K buffer. If the next level is not valid, the size K buffer is already in position,
- * the bit pattern is updated and returned. If the next level is valid, it is merged with the size K buffer into the size 2K buffer.
- * Continue the recursion until a non-valid level becomes filled by the size K buffer,
- * the bit pattern is updated and returned. This class contains a highly specialized sort called blockyTandemMergeSort().
- * It also contains methods that are used while building histograms and other common
- * functions. Count = N % (2*K)
- */
- private int baseBufferCount_;
-
- /**
- * Active levels expressed as a bit pattern.
- *
- * Pattern = N / (2 * K)
- */
- private long bitPattern_;
-
- /**
- * This single array contains the base buffer plus all used levels.
- * A level is of size K and is either full and sorted.
- * Whether a level buffer is present is indicated by the bitPattern_.
- * The base buffer is sorted and has max length 2*K but uses only baseBufferCount_ items.
- * The base buffer precedes the level buffers. This buffer does not include the min, max items.
- *
- * The levels arrays require quite a bit of explanation, which we defer until later. Count = N % (2*K) Pattern = N / (2 * K) The levels arrays require quite a bit of explanation, which we defer until later. Note: It is easy to prove that the following simplified code which launches multiple waves of
- * carry propagation does exactly the same amount of merging work (including the work of
- * allocating fresh buffers) as the more complicated and seemingly more efficient approach that
- * tracks a single carry propagation wave through both sketches.
- *
- * This simplified code probably does do slightly more "outer loop" work, but I am pretty
- * sure that even that is within a constant factor of the more complicated code, plus the
- * total amount of "outer loop" work is at least a factor of K smaller than the total amount of
- * merging work, which is identical in the two approaches.
- *
- * Note: a two-way merge that doesn't modify either of its two inputs could be implemented
- * by making a deep copy of the larger sketch and then merging the smaller one into it.
- * However, it was decided not to do this.
- *
- * @param The intent of the design of this class was to isolate the detailed knowledge of the bit and
- * byte layout of the serialized form of the sketches derived from the base sketch classes into one place.
- * This allows the possibility of the introduction of different serialization
- * schemes with minimal impact on the rest of the library.
- * LAYOUT: The low significance bytes of this long based data structure are on the right.
- * The multi-byte primitives are stored in native byte order.
- * The single byte fields are treated as unsigned. An empty QuantilesItemsSketch, on-heap QuantilesDoublesSketch or compact off-heap QuantilesDoublesSketch only require 8
- * bytes. An off-heap UpdatableQuantilesDoublesSketch and all non-empty sketches require at least 16 bytes of
- * preamble. A k of 128 produces a normalized, rank error of about 1.7%.
- * For example, the median returned from getQuantile(0.5) will be between the actual quantiles
- * from the hypothetically sorted array of input quantiles at normalized ranks of 0.483 and 0.517, with
- * a confidence of about 99%. The given MemorySegment must be writable and it must contain a UpdatableQuantilesDoublesSketch.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. The given MemorySegment must be writable and it must contain a UpdatableQuantilesDoublesSketch.
- * The sketch will be updated and managed totally within the MemorySegment. If the given source
- * MemorySegment is created off-heap, then all the management of the sketch's internal data will be off-heap as well. NOTE:If during updating of the sketch the sketch requires more capacity than the given size of the MemorySegment, the sketch
- * will request more capacity using the {@link MemorySegmentRequest MemorySegmentRequest} interface. The default of this interface will
- * return a new MemorySegment on the heap. It is up to the user to optionally extend this interface if more flexible
- * handling of requests for more capacity is required. It is required that the results of the union operation, which can be obtained at any time,
- * is obtained from {@link #getResult() }.
- *
- * @param sketchIn the sketch to be merged into this one.
- */
- public abstract void union(QuantilesDoublesSketch sketchIn);
-
- /**
- * Iterative union operation, which means this method can be repeatedly called.
- * Merges the given MemorySegment image of a QuantilesDoublesSketch into this union object.
- * The given MemorySegment object is not modified and a link to it is not retained.
- * It is required that the ratio of the two K's be a power of 2.
- * This is easily satisfied if each of the K's are already a power of 2.
- * If the given sketch is null or empty it is ignored.
- *
- * It is required that the results of the union operation, which can be obtained at any time,
- * is obtained from {@link #getResult() }.
- *
- * @param seg MemorySegment image of sketch to be merged
- */
- public abstract void union(MemorySegment seg);
-
- /**
- * Update this union with the given double (or float) data Item.
- *
- * @param dataItem The given double datum.
- */
- public abstract void update(double dataItem);
-
- /**
- * Gets the result of this Union as an UpdatableQuantilesDoublesSketch, which enables further update
- * operations on the resulting sketch. The Union state has not been changed, which allows
- * further union operations.
- *
- * @return the result of this Union operation
- */
- public abstract UpdatableQuantilesDoublesSketch getResult();
-
- /**
- * Places the result of this Union into the provided MemorySegment as an UpdatableQuantilesDoublesSketch,
- * which enables further update operations on the resulting sketch. The Union state has not
- * been changed, which allows further union operations.
- *
- * @param dstSeg the destination MemorySegment for the result
- * @param mSegReq the MemorySegmentRequest used if the given MemorySegment needs to expand.
- * Otherwise, it can be null and the default MemorySegmentRequest will be used.
- * @return the result of this Union operation
- */
- public abstract UpdatableQuantilesDoublesSketch getResult(MemorySegment dstSeg, MemorySegmentRequest mSegReq);
-
- /**
- * Gets the result of this Union as an UpdatableQuantilesDoublesSketch, which enables further update
- * operations on the resulting sketch. The Union is reset to the virgin state.
- *
- * @return the result of this Union operation and reset.
- */
- public abstract UpdatableQuantilesDoublesSketch getResultAndReset();
-
- /**
- * Resets this Union to a virgin state.
- */
- public abstract void reset();
-
- /**
- * Serialize this union to a byte array. Result is an UpdatableQuantilesDoublesSketch, serialized in an
- * unordered, non-compact form. The resulting byte[] can be heapified or wrapped as either a
- * sketch or a union.
- *
- * @return byte array of this union
- */
- public abstract byte[] toByteArray();
-
- /**
- * Returns summary information about the backing sketch.
- */
- @Override
- public abstract String toString();
-
- /**
- * Returns summary information about the backing sketch. Used for debugging.
- * @param sketchSummary if true includes sketch summary
- * @param dataDetail if true includes data detail
- * @return summary information about the sketch.
- */
- public abstract String toString(boolean sketchSummary, boolean dataDetail);
-
-}
diff --git a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnionBuilder.java b/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnionBuilder.java
deleted file mode 100644
index 555b2660a..000000000
--- a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnionBuilder.java
+++ /dev/null
@@ -1,85 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.quantiles;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.MemorySegmentRequest;
-
-/**
- * For building a new QuantilesDoublesSketch Union operation.
- *
- * @author Lee Rhodes
- */
-public class QuantilesDoublesUnionBuilder {
- private int bMaxK = PreambleUtil.DEFAULT_K;
-
- /**
- * Constructor for a new QuantilesDoublesUnionBuilder. The default configuration is
- * A k of 128 produces a normalized, rank error of about 1.7%.
- * For example, the median returned from getQuantile(0.5) will be between the actual quantiles
- * from the hypothetically sorted array of input quantiles at normalized ranks of 0.483 and 0.517, with
- * a confidence of about 99%. The size of an QuantilesItemsSketch is very dependent on the size of the generic Items input into the sketch,
- * so there is no comparable size table as there is for the QuantilesDoublesSketch. Count = N % (2 * K) Pattern = N / (2 * K) The levels arrays require quite a bit of explanation, which we defer until later.
- *
- *
- *
- * The approximateLowerBoundOnP is defined with respect to the right tail of the binomial
- * distribution.
- *
- *
- * @param n is the number of trials. Must be non-negative.
- * @param k is the number of successes. Must be non-negative, and cannot exceed n.
- * @param numStdDevs the number of standard deviations defining the confidence interval
- * @return the lower bound of the approximate Clopper-Pearson confidence interval for the
- * unknown success probability.
- */
- public static double approximateLowerBoundOnP(final long n, final long k, final double numStdDevs) {
- checkInputs(n, k);
- if (n == 0) { return 0.0; } // the coin was never flipped, so we know nothing
- else if (k == 0) { return 0.0; }
- else if (k == 1) { return (exactLowerBoundOnPForKequalsOne(n, deltaOfNumStdevs(numStdDevs))); }
- else if (k == n) { return (exactLowerBoundOnPForKequalsN(n, deltaOfNumStdevs(numStdDevs))); }
- else {
- final double x = abramowitzStegunFormula26p5p22((n - k) + 1, k, (-1.0 * numStdDevs));
- return (1.0 - x); // which is p
- }
- }
-
- /**
- * Computes upper bound of approximate Clopper-Pearson confidence interval for a binomial
- * proportion.
- *
- *
- * The approximateUpperBoundOnP is defined with respect to the left tail of the binomial
- * distribution.
- *
- * @param n is the number of trials. Must be non-negative.
- * @param k is the number of successes. Must be non-negative, and cannot exceed n.
- * @param numStdDevs the number of standard deviations defining the confidence interval
- * @return the upper bound of the approximate Clopper-Pearson confidence interval for the
- * unknown success probability.
- */
- public static double approximateUpperBoundOnP(final long n, final long k, final double numStdDevs) {
- checkInputs(n, k);
- if (n == 0) { return 1.0; } // the coin was never flipped, so we know nothing
- else if (k == n) { return 1.0; }
- else if (k == (n - 1)) {
- return (exactUpperBoundOnPForKequalsNminusOne(n, deltaOfNumStdevs(numStdDevs)));
- }
- else if (k == 0) {
- return (exactUpperBoundOnPForKequalsZero(n, deltaOfNumStdevs(numStdDevs)));
- }
- else {
- final double x = abramowitzStegunFormula26p5p22(n - k, k + 1, numStdDevs);
- return (1.0 - x); // which is p
- }
- }
-
- /**
- * Computes an estimate of an unknown binomial proportion.
- * @param n is the number of trials. Must be non-negative.
- * @param k is the number of successes. Must be non-negative, and cannot exceed n.
- * @return the estimate of the unknown binomial proportion.
- */
- public static double estimateUnknownP(final long n, final long k) {
- checkInputs(n, k);
- if (n == 0) { return 0.5; } // the coin was never flipped, so we know nothing
- else { return ((double) k / (double) n); }
- }
-
- private static void checkInputs(final long n, final long k) {
- if (n < 0) { throw new SketchesArgumentException("N must be non-negative"); }
- if (k < 0) { throw new SketchesArgumentException("K must be non-negative"); }
- if (k > n) { throw new SketchesArgumentException("K cannot exceed N"); }
- }
-
- /**
- * Computes an approximation to the erf() function.
- * @param x is the input to the erf function
- * @return returns erf(x), accurate to roughly 7 decimal digits.
- */
- public static double erf(final double x) {
- if (x < 0.0) { return (-1.0 * (erf_of_nonneg(-1.0 * x))); }
- else { return (erf_of_nonneg(x)); }
- }
-
- /**
- * Computes an approximation to normalCDF(x).
- * @param x is the input to the normalCDF function
- * @return returns the approximation to normalCDF(x).
- */
- public static double normalCDF(final double x) {
- return (0.5 * (1.0 + (erf(x / (Math.sqrt(2.0))))));
- }
-
- //@formatter:off
- // Abramowitz and Stegun formula 7.1.28, p. 88; Claims accuracy of about 7 decimal digits
- private static double erf_of_nonneg(final double x) {
- // The constants from the book
- final double a1 = 0.07052_30784;
- final double a3 = 0.00927_05272;
- final double a5 = 0.00027_65672;
- final double a2 = 0.04228_20123;
- final double a4 = 0.00015_20143;
- final double a6 = 0.00004_30638;
- final double x2 = x * x; // x squared, x cubed, etc.
- final double x3 = x2 * x;
- final double x4 = x2 * x2;
- final double x5 = x2 * x3;
- final double x6 = x3 * x3;
- final double sum = ( 1.0
- + (a1 * x)
- + (a2 * x2)
- + (a3 * x3)
- + (a4 * x4)
- + (a5 * x5)
- + (a6 * x6) );
- final double sum2 = sum * sum; // raise the sum to the 16th power
- final double sum4 = sum2 * sum2;
- final double sum8 = sum4 * sum4;
- final double sum16 = sum8 * sum8;
- return (1.0 - (1.0 / sum16));
- }
- //@formatter:on
-
- private static double deltaOfNumStdevs(final double kappa) {
- return (normalCDF(-1.0 * kappa));
- }
-
- // Formula 26.5.22 on page 945 of Abramowitz & Stegun, which is an approximation
- // of the inverse of the incomplete beta function I_x(a,b) = delta
- // viewed as a scalar function of x.
- // In other words, we specify delta, and it gives us x (with a and b held constant).
- // However, delta is specified in an indirect way through yp which
- // is the number of stdDevs that leaves delta probability in the right
- // tail of a standard gaussian distribution.
-
- // We point out that the variable names correspond to those in the book,
- // and it is worth keeping it that way so that it will always be easy to verify
- // that the formula was typed in correctly.
-
- private static double abramowitzStegunFormula26p5p22(final double a, final double b,
- final double yp) {
- final double b2m1 = (2.0 * b) - 1.0;
- final double a2m1 = (2.0 * a) - 1.0;
- final double lambda = ((yp * yp) - 3.0) / 6.0;
- final double htmp = (1.0 / a2m1) + (1.0 / b2m1);
- final double h = 2.0 / htmp;
- final double term1 = (yp * (Math.sqrt(h + lambda))) / h;
- final double term2 = (1.0 / b2m1) - (1.0 / a2m1);
- final double term3 = (lambda + (5.0 / 6.0)) - (2.0 / (3.0 * h));
- final double w = term1 - (term2 * term3);
- final double xp = a / (a + (b * (Math.exp(2.0 * w))));
- return xp;
- }
-
- // Formulas for some special cases.
-
- private static double exactUpperBoundOnPForKequalsZero(final double n, final double delta) {
- return (1.0 - Math.pow(delta, (1.0 / n)));
- }
-
- private static double exactLowerBoundOnPForKequalsN(final double n, final double delta) {
- return (Math.pow(delta, (1.0 / n)));
- }
-
- private static double exactLowerBoundOnPForKequalsOne(final double n, final double delta) {
- return (1.0 - Math.pow((1.0 - delta), (1.0 / n)));
- }
-
- private static double exactUpperBoundOnPForKequalsNminusOne(final double n, final double delta) {
- return (Math.pow((1.0 - delta), (1.0 / n)));
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/BoundsOnRatiosInSampledSets.java b/src/main/java/org/apache/datasketches/common/BoundsOnRatiosInSampledSets.java
deleted file mode 100644
index b65afe43a..000000000
--- a/src/main/java/org/apache/datasketches/common/BoundsOnRatiosInSampledSets.java
+++ /dev/null
@@ -1,130 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-import static org.apache.datasketches.common.BoundsOnBinomialProportions.approximateLowerBoundOnP;
-import static org.apache.datasketches.common.BoundsOnBinomialProportions.approximateUpperBoundOnP;
-
-/**
- * This class is used to compute the bounds on the estimate of the ratio |B| / |A|, where:
- *
- *
- *
- * @author Kevin Lang
- */
-public final class BoundsOnRatiosInSampledSets {
- private static final double NUM_STD_DEVS = 2.0; //made a constant to simplify interface.
-
- private BoundsOnRatiosInSampledSets() {}
-
- /**
- * Return the approximate lower bound based on a 95% confidence interval
- * @param a See class javadoc
- * @param b See class javadoc
- * @param f the inclusion probability used to produce the set with size a and should
- * generally be less than 0.5. Above this value, the results not be reliable.
- * When f = 1.0 this returns the estimate.
- * @return the approximate upper bound
- */
- public static double getLowerBoundForBoverA(final long a, final long b, final double f) {
- checkInputs(a, b, f);
- if (a == 0) { return 0.0; }
- if (f == 1.0) { return (double) b / a; }
- return approximateLowerBoundOnP(a, b, NUM_STD_DEVS * hackyAdjuster(f));
- }
-
- /**
- * Return the approximate upper bound based on a 95% confidence interval
- * @param a See class javadoc
- * @param b See class javadoc
- * @param f the inclusion probability used to produce the set with size a.
- * @return the approximate lower bound
- */
- public static double getUpperBoundForBoverA(final long a, final long b, final double f) {
- checkInputs(a, b, f);
- if (a == 0) { return 1.0; }
- if (f == 1.0) { return (double) b / a; }
- return approximateUpperBoundOnP(a, b, NUM_STD_DEVS * hackyAdjuster(f));
- }
-
- /**
- * Return the estimate of b over a
- * @param a See class javadoc
- * @param b See class javadoc
- * @return the estimate of b over a
- */
- public static double getEstimateOfBoverA(final long a, final long b) {
- checkInputs(a, b, 0.3);
- if (a == 0) { return 0.5; }
- return (double) b / a;
- }
-
- /**
- * Return the estimate of A. See class javadoc.
- * @param a See class javadoc
- * @param f the inclusion probability used to produce the set with size a.
- * @return the approximate lower bound
- */
- public static double getEstimateOfA(final long a, final double f) {
- checkInputs(a, 1, f);
- return a / f;
- }
-
- /**
- * Return the estimate of B. See class javadoc.
- * @param b See class javadoc
- * @param f the inclusion probability used to produce the set with size b.
- * @return the approximate lower bound
- */
- public static double getEstimateOfB(final long b, final double f) {
- checkInputs(b + 1, b, f);
- return b / f;
- }
-
- /**
- * This hackyAdjuster is tightly coupled with the width of the confidence interval normally
- * specified with number of standard deviations. To simplify this interface the number of
- * standard deviations has been fixed to 2.0, which corresponds to a confidence interval of
- * 95%.
- * @param f the inclusion probability used to produce the set with size a.
- * @return the hacky Adjuster
- */
- private static double hackyAdjuster(final double f) {
- final double tmp = Math.sqrt(1.0 - f);
- return (f <= 0.5) ? tmp : tmp + (0.01 * (f - 0.5));
- }
-
- static void checkInputs(final long a, final long b, final double f) {
- if ( ( (a - b) | (a) | (b) ) < 0) { //if any group goes negative
- throw new SketchesArgumentException(
- "a must be >= b and neither a nor b can be < 0: a = " + a + ", b = " + b);
- }
- if ((f > 1.0) || (f <= 0.0)) {
- throw new SketchesArgumentException("Required: ((f <= 1.0) && (f > 0.0)): " + f);
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/ByteArrayUtil.java b/src/main/java/org/apache/datasketches/common/ByteArrayUtil.java
deleted file mode 100644
index abf16eede..000000000
--- a/src/main/java/org/apache/datasketches/common/ByteArrayUtil.java
+++ /dev/null
@@ -1,334 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-/**
- * Useful methods for byte arrays.
- * @author Lee Rhodes
- */
-public final class ByteArrayUtil {
-
- /**
- * No argument constructor.
- */
- public ByteArrayUtil() { }
-
- /**
- * Copies bytes from source to target with offsets on both the source and target.
- * @param source the given source
- * @param srcStart the source starting index
- * @param target the give target
- * @param tgtStart the target starting index
- * @param numBytes the number of bytes to be transferred.
- */
- public static void copyBytes(final byte[] source, final int srcStart,
- final byte[] target, final int tgtStart, final int numBytes) {
- Util.checkBounds(srcStart, numBytes, source.length);
- Util.checkBounds(tgtStart, numBytes, target.length);
- for (int i = 0, j = srcStart, k = tgtStart; i < numBytes; i++) {
- target[k++] = source[j++];
- }
- }
-
- /**
- * Get a short from the given byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the short
- */
- public static short getShortLE(final byte[] array, final int offset) {
- return (short) ((array[offset ] & 0XFF )
- | ((array[offset + 1] & 0XFF) << 8));
- }
-
- /**
- * Put the source short into the destination byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source short
- */
- public static void putShortLE(final byte[] array, final int offset, final short value) {
- array[offset ] = (byte) (value );
- array[offset + 1] = (byte) (value >>> 8);
- }
-
- /**
- * Get a short from the given byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the short
- */
- public static short getShortBE(final byte[] array, final int offset) {
- return (short) ((array[offset + 1] & 0XFF )
- | ((array[offset ] & 0XFF) << 8));
- }
-
- /**
- * Put the source short into the destination byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source short
- */
- public static void putShortBE(final byte[] array, final int offset, final short value) {
- array[offset + 1] = (byte) (value );
- array[offset ] = (byte) (value >>> 8);
- }
-
- /**
- * Get a int from the given byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the int
- */
- public static int getIntLE(final byte[] array, final int offset) {
- return ( array[offset ] & 0XFF )
- | ((array[offset + 1] & 0XFF) << 8)
- | ((array[offset + 2] & 0XFF) << 16)
- | ((array[offset + 3] & 0XFF) << 24);
- }
-
- /**
- * Put the source int into the destination byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source int
- */
- public static void putIntLE(final byte[] array, final int offset, final int value) {
- array[offset ] = (byte) (value );
- array[offset + 1] = (byte) (value >>> 8);
- array[offset + 2] = (byte) (value >>> 16);
- array[offset + 3] = (byte) (value >>> 24);
- }
-
- /**
- * Get a int from the given byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the int
- */
- public static int getIntBE(final byte[] array, final int offset) {
- return ( array[offset + 3] & 0XFF )
- | ((array[offset + 2] & 0XFF) << 8)
- | ((array[offset + 1] & 0XFF) << 16)
- | ((array[offset ] & 0XFF) << 24);
- }
-
- /**
- * Put the source int into the destination byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source int
- */
- public static void putIntBE(final byte[] array, final int offset, final int value) {
- array[offset + 3] = (byte) (value );
- array[offset + 2] = (byte) (value >>> 8);
- array[offset + 1] = (byte) (value >>> 16);
- array[offset ] = (byte) (value >>> 24);
- }
-
- /**
- * Get a long from the given byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the long
- */
- public static long getLongLE(final byte[] array, final int offset) {
- return ( array[offset ] & 0XFFL )
- | ((array[offset + 1] & 0XFFL) << 8)
- | ((array[offset + 2] & 0XFFL) << 16)
- | ((array[offset + 3] & 0XFFL) << 24)
- | ((array[offset + 4] & 0XFFL) << 32)
- | ((array[offset + 5] & 0XFFL) << 40)
- | ((array[offset + 6] & 0XFFL) << 48)
- | ((array[offset + 7] & 0XFFL) << 56);
- }
-
- /**
- * Put the source long into the destination byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source long
- */
- public static void putLongLE(final byte[] array, final int offset, final long value) {
- array[offset ] = (byte) (value );
- array[offset + 1] = (byte) (value >>> 8);
- array[offset + 2] = (byte) (value >>> 16);
- array[offset + 3] = (byte) (value >>> 24);
- array[offset + 4] = (byte) (value >>> 32);
- array[offset + 5] = (byte) (value >>> 40);
- array[offset + 6] = (byte) (value >>> 48);
- array[offset + 7] = (byte) (value >>> 56);
- }
-
- /**
- * Get a long from the source byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source starting point
- * @return the long
- */
- public static long getLongBE(final byte[] array, final int offset) {
- return ( array[offset + 7] & 0XFFL )
- | ((array[offset + 6] & 0XFFL) << 8)
- | ((array[offset + 5] & 0XFFL) << 16)
- | ((array[offset + 4] & 0XFFL) << 24)
- | ((array[offset + 3] & 0XFFL) << 32)
- | ((array[offset + 2] & 0XFFL) << 40)
- | ((array[offset + 1] & 0XFFL) << 48)
- | ((array[offset ] & 0XFFL) << 56);
- }
-
- /**
- * Put the source long into the destination byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination starting point
- * @param value source long
- */
- public static void putLongBE(final byte[] array, final int offset, final long value) {
- array[offset + 7] = (byte) (value );
- array[offset + 6] = (byte) (value >>> 8);
- array[offset + 5] = (byte) (value >>> 16);
- array[offset + 4] = (byte) (value >>> 24);
- array[offset + 3] = (byte) (value >>> 32);
- array[offset + 2] = (byte) (value >>> 40);
- array[offset + 1] = (byte) (value >>> 48);
- array[offset ] = (byte) (value >>> 56);
- }
-
- /**
- * Get a float from the given byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the float
- */
- public static float getFloatLE(final byte[] array, final int offset) {
- return Float.intBitsToFloat(getIntLE(array, offset));
- }
-
- /**
- * Put the source float into the destination byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source float
- */
- public static void putFloatLE(final byte[] array, final int offset, final float value) {
- putIntLE(array, offset, Float.floatToRawIntBits(value));
- }
-
- /**
- * Get a float from the given byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the float
- */
- public static float getFloatBE(final byte[] array, final int offset) {
- return Float.intBitsToFloat(getIntBE(array, offset));
- }
-
- /**
- * Put the source float into the destination byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source float
- */
- public static void putFloatBE(final byte[] array, final int offset, final float value) {
- putIntBE(array, offset, Float.floatToRawIntBits(value));
- }
-
- /**
- * Get a double from the given byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the double
- */
- public static double getDoubleLE(final byte[] array, final int offset) {
- return Double.longBitsToDouble(getLongLE(array, offset));
- }
-
- /**
- * Put the source double into the destination byte array starting at the given offset
- * in little endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source double
- */
- public static void putDoubleLE(final byte[] array, final int offset, final double value) {
- putLongLE(array, offset, Double.doubleToRawLongBits(value));
- }
-
- /**
- * Get a double from the given byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array source byte array
- * @param offset source offset
- * @return the double
- */
- public static double getDoubleBE(final byte[] array, final int offset) {
- return Double.longBitsToDouble(getLongBE(array, offset));
- }
-
- /**
- * Put the source double into the destination byte array starting at the given offset
- * in big endian order.
- * There is no bounds checking.
- * @param array destination byte array
- * @param offset destination offset
- * @param value source double
- */
- public static void putDoubleBE(final byte[] array, final int offset, final double value) {
- putLongBE(array, offset, Double.doubleToRawLongBits(value));
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/Family.java b/src/main/java/org/apache/datasketches/common/Family.java
deleted file mode 100644
index 032373685..000000000
--- a/src/main/java/org/apache/datasketches/common/Family.java
+++ /dev/null
@@ -1,267 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-import java.nio.ByteOrder;
-import java.util.HashMap;
-import java.util.Locale;
-import java.util.Map;
-
-/**
- * Defines the various families of sketch and set operation classes. A family defines a set of
- * classes that share fundamental algorithms and behaviors. The classes within a family may
- * still differ by how they are stored and accessed. For example, internally there are separate
- * classes for the QuickSelect sketch algorithm that operate on the Java heap and off-heap.
- * Not all of these families have parallel forms on and off-heap but are included for completeness.
- *
- * @author Lee Rhodes
- */
-public enum Family {
- /**
- * The Alpha Sketch family is a member of the Theta Sketch Framework of sketches and is best
- * suited for real-time processes where both the updating of the sketch and getting the estimate
- * is performed directly on the sketch. In this situation the AlphaSketch has roughly a
- * 30% improvement (~1/sqrt(2*k)) in its error distribution as compared to the QuickSelect
- * (or similar KMV-derived) sketches.
- *
- *
- * Also see:
- * blog.teamleadnet.com/2012/07/quick-select-algorithm-find-kth-element.html
- * See QuickSelectTest for examples and testNG tests.
- *
- * @author Lee Rhodes
- */
-public final class QuickSelect {
-
- private QuickSelect() {}
-
- /**
- * Gets the 0-based kth order statistic from the array. Warning! This changes the ordering
- * of elements in the given array!
- *
- * @param arr The array to be re-arranged.
- * @param lo The lowest 0-based index to be considered.
- * @param hi The highest 0-based index to be considered.
- * @param pivot The 0-based index of the value to pivot on.
- * @return The value of the smallest (n)th element where n is 0-based.
- */
- public static long select(final long[] arr, int lo, int hi, final int pivot) {
- while (hi > lo) {
- final int j = partition(arr, lo, hi);
- if (j == pivot) {
- return arr[pivot];
- }
- if (j > pivot) {
- hi = j - 1;
- }
- else {
- lo = j + 1;
- }
- }
- return arr[pivot];
- }
-
- /**
- * Gets the 1-based kth order statistic from the array including any zero values in the
- * array. Warning! This changes the ordering of elements in the given array!
- *
- * @param arr The hash array.
- * @param pivot The 1-based index of the value that is chosen as the pivot for the array.
- * After the operation all values below this 1-based index will be less than this value
- * and all values above this index will be greater. The 0-based index of the pivot will be
- * pivot-1.
- * @return The value of the smallest (N)th element including zeros, where N is 1-based.
- */
- public static long selectIncludingZeros(final long[] arr, final int pivot) {
- final int arrSize = arr.length;
- final int adj = pivot - 1;
- return select(arr, 0, arrSize - 1, adj);
- }
-
- /**
- * Gets the 1-based kth order statistic from the array excluding any zero values in the
- * array. Warning! This changes the ordering of elements in the given array!
- *
- * @param arr The hash array.
- * @param nonZeros The number of non-zero values in the array.
- * @param pivot The 1-based index of the value that is chosen as the pivot for the array.
- * After the operation all values below this 1-based index will be less than this value
- * and all values above this index will be greater. The 0-based index of the pivot will be
- * pivot+arr.length-nonZeros-1.
- * @return The value of the smallest (N)th element excluding zeros, where N is 1-based.
- */
- public static long selectExcludingZeros(final long[] arr, final int nonZeros, final int pivot) {
- if (pivot > nonZeros) {
- return 0L;
- }
- final int arrSize = arr.length;
- final int zeros = arrSize - nonZeros;
- final int adjK = (pivot + zeros) - 1;
- return select(arr, 0, arrSize - 1, adjK);
- }
-
- /**
- * Partition arr[] into arr[lo .. i-1], arr[i], arr[i+1,hi]
- *
- * @param arr The given array to partition
- * @param lo the low index
- * @param hi the high index
- * @return the next partition value. Ultimately, the desired pivot.
- */
- private static int partition(final long[] arr, final int lo, final int hi) {
- int i = lo, j = hi + 1; //left and right scan indices
- final long v = arr[lo]; //partitioning item value
- while (true) {
- //Scan right, scan left, check for scan complete, and exchange
- while (arr[ ++i] < v) {
- if (i == hi) {
- break;
- }
- }
- while (v < arr[ --j]) {
- if (j == lo) {
- break;
- }
- }
- if (i >= j) {
- break;
- }
- final long x = arr[i];
- arr[i] = arr[j];
- arr[j] = x;
- }
- //put v=arr[j] into position with a[lo .. j-1] <= a[j] <= a[j+1 .. hi]
- final long x = arr[lo];
- arr[lo] = arr[j];
- arr[j] = x;
- return j;
- }
-
- //For double arrays
-
- /**
- * Gets the 0-based kth order statistic from the array. Warning! This changes the ordering
- * of elements in the given array!
- *
- * @param arr The array to be re-arranged.
- * @param lo The lowest 0-based index to be considered.
- * @param hi The highest 0-based index to be considered.
- * @param pivot The 0-based smallest value to pivot on.
- * @return The value of the smallest (n)th element where n is 0-based.
- */
- public static double select(final double[] arr, int lo, int hi, final int pivot) {
- while (hi > lo) {
- final int j = partition(arr, lo, hi);
- if (j == pivot) {
- return arr[pivot];
- }
- if (j > pivot) {
- hi = j - 1;
- }
- else {
- lo = j + 1;
- }
- }
- return arr[pivot];
- }
-
- /**
- * Gets the 1-based kth order statistic from the array including any zero values in the
- * array. Warning! This changes the ordering of elements in the given array!
- *
- * @param arr The hash array.
- * @param pivot The 1-based index of the value that is chosen as the pivot for the array.
- * After the operation all values below this 1-based index will be less than this value
- * and all values above this index will be greater. The 0-based index of the pivot will be
- * pivot-1.
- * @return The value of the smallest (N)th element including zeros, where N is 1-based.
- */
- public static double selectIncludingZeros(final double[] arr, final int pivot) {
- final int arrSize = arr.length;
- final int adj = pivot - 1;
- return select(arr, 0, arrSize - 1, adj);
- }
-
- /**
- * Gets the 1-based kth order statistic from the array excluding any zero values in the
- * array. Warning! This changes the ordering of elements in the given array!
- *
- * @param arr The hash array.
- * @param nonZeros The number of non-zero values in the array.
- * @param pivot The 1-based index of the value that is chosen as the pivot for the array.
- * After the operation all values below this 1-based index will be less than this value
- * and all values above this index will be greater. The 0-based index of the pivot will be
- * pivot+arr.length-nonZeros-1.
- * @return The value of the smallest (N)th element excluding zeros, where N is 1-based.
- */
- public static double selectExcludingZeros(final double[] arr, final int nonZeros, final int pivot) {
- if (pivot > nonZeros) {
- return 0L;
- }
- final int arrSize = arr.length;
- final int zeros = arrSize - nonZeros;
- final int adjK = (pivot + zeros) - 1;
- return select(arr, 0, arrSize - 1, adjK);
- }
-
- /**
- * Partition arr[] into arr[lo .. i-1], arr[i], arr[i+1,hi]
- *
- * @param arr The given array to partition
- * @param lo the low index
- * @param hi the high index
- * @return the next partition value. Ultimately, the desired pivot.
- */
- private static int partition(final double[] arr, final int lo, final int hi) {
- int i = lo, j = hi + 1; //left and right scan indices
- final double v = arr[lo]; //partitioning item value
- while (true) {
- //Scan right, scan left, check for scan complete, and exchange
- while (arr[ ++i] < v) {
- if (i == hi) {
- break;
- }
- }
- while (v < arr[ --j]) {
- if (j == lo) {
- break;
- }
- }
- if (i >= j) {
- break;
- }
- final double x = arr[i];
- arr[i] = arr[j];
- arr[j] = x;
- }
- //put v=arr[j] into position with a[lo .. j-1] <= a[j] <= a[j+1 .. hi]
- final double x = arr[lo];
- arr[lo] = arr[j];
- arr[j] = x;
- return j;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/ResizeFactor.java b/src/main/java/org/apache/datasketches/common/ResizeFactor.java
deleted file mode 100644
index 5dfc8addc..000000000
--- a/src/main/java/org/apache/datasketches/common/ResizeFactor.java
+++ /dev/null
@@ -1,80 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-/**
- * For the Families that accept this configuration parameter, it controls the size multiple that
- * affects how fast the internal cache grows, when more space is required.
- * See Resize Factor
- *
- * @author Lee Rhodes
- */
-public enum ResizeFactor {
- /**
- * Do not resize. Sketch will be configured to full size.
- */
- X1(0),
- /**
- * Resize factor is 2.
- */
- X2(1),
- /**
- * Resize factor is 4.
- */
- X4(2),
- /**
- * Resize factor is 8.
- */
- X8(3);
-
- private int lg_;
-
- ResizeFactor(final int lg) {
- this.lg_ = lg;
- }
-
- /**
- * Returns the Log-base 2 of the Resize Factor
- * @return the Log-base 2 of the Resize Factor
- */
- public int lg() {
- return lg_;
- }
-
- /**
- * Returns the Resize Factor given the Log-base 2 of the Resize Factor
- * @param lg a value between zero and 3, inclusive.
- * @return the Resize Factor given the Log-base 2 of the Resize Factor
- */
- public static ResizeFactor getRF(final int lg) {
- if (X1.lg() == lg) { return X1; }
- if (X2.lg() == lg) { return X2; }
- if (X4.lg() == lg) { return X4; }
- return X8;
- }
-
- /**
- * Returns the Resize Factor
- * @return the Resize Factor
- */
- public int getValue() {
- return 1 << lg_;
- }
-}
diff --git a/src/main/java/org/apache/datasketches/common/SketchesArgumentException.java b/src/main/java/org/apache/datasketches/common/SketchesArgumentException.java
deleted file mode 100644
index 4644f595b..000000000
--- a/src/main/java/org/apache/datasketches/common/SketchesArgumentException.java
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-/**
- * Illegal Arguments Exception class for the library
- *
- * @author Lee Rhodes
- */
-public class SketchesArgumentException extends SketchesException {
- private static final long serialVersionUID = 1L;
-
- //other constructors to be added as needed.
-
- /**
- * Constructs a new runtime exception with the specified detail message. The cause is not
- * initialized, and may subsequently be initialized by a call to
- * Throwable.initCause(java.lang.Throwable).
- *
- * @param message the detail message. The detail message is saved for later retrieval by the
- * Throwable.getMessage() method.
- */
- public SketchesArgumentException(final String message) {
- super(message);
- }
-}
diff --git a/src/main/java/org/apache/datasketches/common/SketchesException.java b/src/main/java/org/apache/datasketches/common/SketchesException.java
deleted file mode 100644
index 83f427cb0..000000000
--- a/src/main/java/org/apache/datasketches/common/SketchesException.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common;
-
-/**
- * Exception class for the library
- *
- * @author Lee Rhodes
- */
-public class SketchesException extends RuntimeException {
- private static final long serialVersionUID = 1L;
-
- //other constructors to be added as needed.
-
- /**
- * Constructs a new runtime exception with the specified detail message. The cause is not
- * initialized, and may subsequently be initialized by a call to
- * Throwable.initCause(java.lang.Throwable).
- *
- * @param message the detail message. The detail message is saved for later retrieval by the
- * Throwable.getMessage() method.
- */
- public SketchesException(final String message) {
- super(message);
- }
-
- /**
- * Constructs a new runtime exception with the specified detail message and cause.
- *
- *
- * "This operation or mode is not supported: ".
- *
- *
- *
- *
- * @param n The input int argument.
- * @return the ceiling power of 2.
- */
- public static int ceilingPowerOf2(final int n) {
- if (n <= 1) { return 1; }
- final int topIntPwrOf2 = 1 << 30;
- return n >= topIntPwrOf2 ? topIntPwrOf2 : Integer.highestOneBit((n - 1) << 1);
- }
-
- /**
- * Computes the long ceiling power of 2 within the range [1, 2^62]. This is the smallest positive power
- * of 2 that is equal to or greater than the given n and a positive long.
- *
- *
- *
- *
- * @param n The input long argument.
- * @return the ceiling power of 2.
- */
- public static long ceilingPowerOf2(final long n) {
- if (n <= 1L) { return 1L; }
- final long topIntPwrOf2 = 1L << 62;
- return n >= topIntPwrOf2 ? topIntPwrOf2 : Long.highestOneBit((n - 1L) << 1);
- }
-
- /**
- * Computes the floor power of 2 given n is in the range [1, 2^31-1].
- * This is the largest positive power of 2 that equal to or less than the given n and equal
- * to a positive integer.
- *
- *
- *
- *
- * @param n The given int argument.
- * @return the floor power of 2 as an int.
- */
- public static int floorPowerOf2(final int n) {
- if (n <= 1) { return 1; }
- return Integer.highestOneBit(n);
- }
-
- /**
- * Computes the floor power of 2 given n is in the range [1, 2^63-1].
- * This is the largest positive power of 2 that is equal to or less than the given n and
- * equal to a positive integer.
- *
- *
- *
- *
- * @param n The given long argument.
- * @return the floor power of 2 as a long
- */
- public static long floorPowerOf2(final long n) {
- if (n <= 1) { return 1; }
- return Long.highestOneBit(n);
- }
-
- /**
- * This is a long integer equivalent to Math.ceil(n / (double)(1 << k))
- * where: 0 < k ≤ 6 and n is a non-negative long.
- * These limits are not checked for speed reasons.
- * @param n the input dividend as a positive long greater than zero.
- * @param k the input divisor exponent of 2 as a positive integer where 0 < k ≤ 6.
- * @return the long integer equivalent to Math.ceil(n / 2^k).
- */
- public static long ceilingMultiple2expK(final long n, final int k) {
- final long mask = (1L << k) - 1L;
- return (n & mask) > 0 ? (n >>> k) + 1 : n >>> k;
- }
-
- /**
- * Computes the inverse integer power of 2: 1/(2^exp) = 2^(-exp).
- * @param exp a positive value between 0 and 1023 inclusive
- * @return the inverse integer power of 2: 1/(2^exp) = 2^(-exp)
- */
- public static double invPow2(final int exp) {
- if ((exp | (1024 - exp - 1)) < 0) {
- throw new SketchesArgumentException("exp cannot be negative or greater than 1023: " + exp);
- }
- return Double.longBitsToDouble((1023L - exp) << 52);
- }
-
- /**
- * Computes the next larger integer point in the power series
- * point = 2( i / ppo ) given the current point in the series.
- * For illustration, this can be used in a loop as follows:
- *
- * {@code
- * int maxP = 1024;
- * int minP = 1;
- * int ppo = 2;
- *
- * for (int p = minP; p <= maxP; p = pwr2SeriesNext(ppo, p)) {
- * System.out.print(p + " ");
- * }
- * //generates the following series:
- * //1 2 3 4 6 8 11 16 23 32 45 64 91 128 181 256 362 512 724 1024
- * }
- *
- * @param ppo Points-Per-Octave, or the number of points per integer powers of 2 in the series.
- * @param curPoint the current point of the series. Must be ≥ 1.
- * @return the next point in the power series.
- */
- public static long pwr2SeriesNext(final int ppo, final long curPoint) {
- final long cur = curPoint < 1L ? 1L : curPoint;
- int gi = (int)round(log2(cur) * ppo); //current generating index
- long next;
- do {
- next = round(pow(2.0, (double) ++gi / ppo));
- } while ( next <= curPoint);
- return next;
- }
-
- /**
- * Computes the previous, smaller integer point in the power series
- * point = 2( i / ppo ) given the current point in the series.
- * For illustration, this can be used in a loop as follows:
- *
- * {@code
- * int maxP = 1024;
- * int minP = 1;
- * int ppo = 2;
- *
- * for (int p = maxP; p >= minP; p = pwr2SeriesPrev(ppo, p)) {
- * System.out.print(p + " ");
- * }
- * //generates the following series:
- * //1024 724 512 362 256 181 128 91 64 45 32 23 16 11 8 6 4 3 2 1
- * }
- *
- * @param ppo Points-Per-Octave, or the number of points per integer powers of 2 in the series.
- * @param curPoint the current point of the series. Must be ≥ 1.
- * @return the previous, smaller point in the power series.
- * A returned value of zero terminates the series.
- */
- public static int pwr2SeriesPrev(final int ppo, final int curPoint) {
- if (curPoint <= 1) { return 0; }
- int gi = (int)round(log2(curPoint) * ppo); //current generating index
- int prev;
- do {
- prev = (int)round(pow(2.0, (double) --gi / ppo));
- } while (prev >= curPoint);
- return prev;
- }
-
- /**
- * Computes the next larger double in the power series
- * point = logBase( i / ppb ) given the current point in the series.
- * For illustration, this can be used in a loop as follows:
- *
- * {@code
- * double maxP = 1024.0;
- * double minP = 1.0;
- * int ppb = 2;
- * double logBase = 2.0;
- *
- * for (double p = minP; p <= maxP; p = powerSeriesNextDouble(ppb, p, true, logBase)) {
- * System.out.print(p + " ");
- * }
- * //generates the following series:
- * //1 2 3 4 6 8 11 16 23 32 45 64 91 128 181 256 362 512 724 1024
- * }
- *
- * @param ppb Points-Per-Base, or the number of points per integer powers of base in the series.
- * @param curPoint the current point of the series. Must be ≥ 1.0.
- * @param roundToLong if true the output will be rounded to the nearest long.
- * @param logBase the desired base of the logarithms
- * @return the next point in the power series.
- */
- public static double powerSeriesNextDouble(final int ppb, final double curPoint,
- final boolean roundToLong, final double logBase) {
- final double cur = curPoint < 1.0 ? 1.0 : curPoint;
- double gi = round(logBaseOfX(logBase, cur) * ppb ); //current generating index
- double next;
- do {
- final double n = pow(logBase, ++gi / ppb);
- next = roundToLong ? round(n) : n;
- } while (next <= cur);
- return next;
- }
-
- /**
- * Returns the ceiling of a given n given a base, where the ceiling is an integral power of the base.
- * This is the smallest positive power of base that is equal to or greater than the given n
- * and equal to a mathematical integer.
- * The result of this function is consistent with {@link #ceilingPowerOf2(int)} for values
- * less than one. I.e., if n < 1, the result is 1.
- *
- *
- *
- * @author Lee Rhodes
- */
-public interface Positional {
-
- /**
- * Gets an instance of this Positional.
- * @param capacity the upper limit of positional range.
- * @return an instance of Positional.
- */
- static Positional getInstance(final long capacity) {
- return new PositionalImpl(capacity);
- }
-
- /**
- * Increments the current position by the given increment.
- * @param increment the given increment
- * @return this Positional
- */
- Positional incrementPosition(long increment);
-
- /**
- * Gets the end position
- * @return the end position
- */
- long getEnd();
-
- /**
- * Gets the current position
- * @return the current position
- */
- long getPosition();
-
- /**
- * Gets start position
- * @return start position
- */
- long getStart();
-
- /**
- * The number of elements remaining between the current position and the end position
- * @return {@code (end - position)}
- */
- long getRemaining();
-
- /**
- * Returns true if there are elements remaining between the current position and the end position
- * @return {@code (end - position) > 0}
- */
- boolean hasRemaining();
-
- /**
- * Resets the current position to the start position,
- * This does not modify any data.
- * @return this Positional
- */
- Positional resetPosition();
-
- /**
- * Sets the current position.
- * Checks that the positional invariants are not violated.
- * @param position the given current position.
- * @return this Positional
- * @throws PositionInvariantsException if positional invariants have been violated.
- */
- Positional setPosition(long position);
-
- /**
- * Sets start position, current position, and end position.
- * Checks that the positional invariants are not violated.
- * @param start the start position in the buffer
- * @param position the current position between the start and end
- * @param end the end position in the buffer
- * @return this Positional
- * @throws PositionInvariantsException if positional invariants have been violated.
- */
- Positional setStartPositionEnd(
- long start,
- long position,
- long end);
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/positional/PositionalImpl.java b/src/main/java/org/apache/datasketches/common/positional/PositionalImpl.java
deleted file mode 100644
index 01e89eb23..000000000
--- a/src/main/java/org/apache/datasketches/common/positional/PositionalImpl.java
+++ /dev/null
@@ -1,133 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common.positional;
-
-/**
- * This implements the positional API.
- * This is different from and simpler than Java BufferImpl positional approach.
- *
- *
- * @author Lee Rhodes
- */
-class PositionalImpl implements Positional {
- private final long capacity;
- private long start = 0;
- private long pos = 0;
- private long end;
-
- /**
- * Construct with total capacity.
- * @param capacity the upper limit of positional range.
- */
- PositionalImpl(
- final long capacity) {
- this.capacity = end = capacity;
- }
-
- @Override
- public final PositionalImpl incrementPosition(final long increment) {
- pos += increment;
- return this;
- }
-
- @Override
- public final long getEnd() {
- return end;
- }
-
- @Override
- public final long getPosition() {
- return pos;
- }
-
- @Override
- public final long getStart() {
- return start;
- }
-
- @Override
- public final long getRemaining() {
- return end - pos;
- }
-
- @Override
- public final boolean hasRemaining() {
- return (end - pos) > 0;
- }
-
- @Override
- public final PositionalImpl resetPosition() {
- pos = start;
- return this;
- }
-
- @Override
- public final PositionalImpl setPosition(final long position) {
- checkInvariants(start, position, end, capacity);
- pos = position;
- return this;
- }
-
- @Override
- public final PositionalImpl setStartPositionEnd(final long start, final long position, final long end) {
- checkInvariants(start, position, end, capacity);
- this.start = start;
- this.end = end;
- pos = position;
- return this;
- }
-
- //RESTRICTED
-
- /**
- * The invariants equation is: {@code 0 <= start <= position <= end <= capacity}.
- * If this equation is violated an PositionInvariantsException will be thrown.
- * @param start the lowest start position
- * @param pos the current position
- * @param end the highest position
- * @param cap the capacity of the backing resource.
- */
- private static final void checkInvariants(final long start, final long pos, final long end,
- final long cap) {
- if ((start | pos | end | cap | (pos - start) | (end - pos) | (cap - end) ) < 0L) {
- throw new PositionInvariantsException(
- "Violation of Invariants: "
- + "start: " + start
- + " <= pos: " + pos
- + " <= end: " + end
- + " <= cap: " + cap
- + "; (pos - start): " + (pos - start)
- + ", (end - pos): " + (end - pos)
- + ", (cap - end): " + (cap - end)
- );
- }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/positional/PositionalSegment.java b/src/main/java/org/apache/datasketches/common/positional/PositionalSegment.java
deleted file mode 100644
index 192b56271..000000000
--- a/src/main/java/org/apache/datasketches/common/positional/PositionalSegment.java
+++ /dev/null
@@ -1,486 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common.positional;
-
-import java.lang.foreign.MemorySegment;
-
-/**
- * Defines the API for relative positional access to a MemorySegment.
- *
- * @author Lee Rhodes
- */
-public interface PositionalSegment extends Positional {
-
- /**
- * Gets an instance of this PositionalSegment.
- * @param seg the given MemorySegment to create the PositionalSegment from.
- * @return a new PositionalSegment.
- */
- static PositionalSegment wrap(final MemorySegment seg) {
- return new PositionalSegmentImpl(seg);
- }
-
- /**
- * Returns a slice of this PositionalSegment at the current position.
- * The end of the slice is the end of the underlying segment.
- * @return a slice of this PositionalSegment at the current position.
- */
- PositionalSegment asSlice();
-
- /**
- * Returns the underlying MemorySegment.
- * The current start, position and end are ignored.
- * @return the underlying MemorySegment
- */
- MemorySegment getMemorySegment();
-
- //PRIMITIVE getX() and getXArray()
-
- /**
- * Gets the boolean value at the current position.
- * Increments the position by Byte.BYTES.
- * @return the boolean at the current position
- */
- boolean getBoolean();
-
- /**
- * Gets the boolean value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the boolean at the given offset
- */
- boolean getBoolean(long offsetBytes);
-
- //intentionally removed getBooleanArray(...)
-
- /**
- * Gets the byte value at the current position.
- * Increments the position by Byte.BYTES.
- * @return the byte at the current position
- */
- byte getByte();
-
- /**
- * Gets the byte value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the byte at the given offset
- */
- byte getByte(long offsetBytes);
-
- /**
- * Gets the byte array at the current position.
- * Increments the position by Byte.BYTES * (lengthBytes - dstOffsetBytes).
- * @param dstArray The preallocated destination array.
- * @param dstOffsetBytes offset in array units
- * @param lengthBytes number of array units to transfer
- */
- void getByteArray(
- byte[] dstArray,
- int dstOffsetBytes,
- int lengthBytes);
-
- /**
- * Gets the char value at the current position.
- * Increments the position by Character.BYTES.
- * @return the char at the current position
- */
- char getChar();
-
- /**
- * Gets the char value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the char at the given offset
- */
- char getChar(long offsetBytes);
-
- /**
- * Gets the char array at the current position.
- * Increments the position by Character.BYTES * (lengthChars - dstOffsetChars).
- * @param dstArray The preallocated destination array.
- * @param dstOffsetChars offset in array units
- * @param lengthChars number of array units to transfer
- */
- void getCharArray(
- char[] dstArray,
- int dstOffsetChars,
- int lengthChars);
-
- /**
- * Gets the double value at the current position.
- * Increments the position by Double.BYTES.
- * @return the double at the current position
- */
- double getDouble();
-
- /**
- * Gets the double value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the double at the given offset
- */
- double getDouble(long offsetBytes);
-
- /**
- * Gets the double array at the current position.
- * Increments the position by Double.BYTES * (lengthDoubles - dstOffsetDoubles).
- * @param dstArray The preallocated destination array.
- * @param dstOffsetDoubles offset in array units
- * @param lengthDoubles number of array units to transfer
- */
- void getDoubleArray(
- double[] dstArray,
- int dstOffsetDoubles,
- int lengthDoubles);
-
- /**
- * Gets the float value at the current position.
- * Increments the position by Float.BYTES.
- * @return the float at the current position
- */
- float getFloat();
-
- /**
- * Gets the float value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the float at the given offset
- */
- float getFloat(long offsetBytes);
-
- /**
- * Gets the float array at the current position.
- * Increments the position by Float.BYTES * (lengthFloats - dstOffsetFloats).
- * @param dstArray The preallocated destination array.
- * @param dstOffsetFloats offset in array units
- * @param lengthFloats number of array units to transfer
- */
- void getFloatArray(
- float[] dstArray,
- int dstOffsetFloats,
- int lengthFloats);
-
- /**
- * Gets the int value at the current position.
- * Increments the position by Integer.BYTES.
- * @return the int at the current position
- */
- int getInt();
-
- /**
- * Gets the int value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the int at the given offset
- */
- int getInt(long offsetBytes);
-
- /**
- * Gets the int array at the current position.
- * Increments the position by Integer.BYTES * (lengthInts - dstOffsetInts).
- * @param dstArray The preallocated destination array.
- * @param dstOffsetInts offset in array units
- * @param lengthInts number of array units to transfer
- */
- void getIntArray(
- int[] dstArray,
- int dstOffsetInts,
- int lengthInts);
-
- /**
- * Gets the long value at the current position.
- * Increments the position by Long.BYTES.
- * @return the long at the current position
- */
- long getLong();
-
- /**
- * Gets the long value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the long at the given offset
- */
- long getLong(long offsetBytes);
-
- /**
- * Gets the long array at the current position.
- * Increments the position by Long.BYTES * (lengthLongs - dstOffsetLongs).
- * @param dstArray The preallocated destination array.
- * @param dstOffsetLongs offset in array units
- * @param lengthLongs number of array units to transfer
- */
- void getLongArray(
- long[] dstArray,
- int dstOffsetLongs,
- int lengthLongs);
-
- /**
- * Gets the short value at the current position.
- * Increments the position by Short.BYTES.
- * @return the short at the current position
- */
- short getShort();
-
- /**
- * Gets the short value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @return the short at the given offset
- */
- short getShort(long offsetBytes);
-
- /**
- * Gets the short array at the current position.
- * Increments the position by Short.BYTES * (lengthShorts - dstOffsetShorts).
- * @param dstArray The preallocated destination array.
- * @param dstOffsetShorts offset in array units
- * @param lengthShorts number of array units to transfer
- */
- void getShortArray(
- short[] dstArray,
- int dstOffsetShorts,
- int lengthShorts);
-
- //PRIMITIVE setX() and setXArray()
-
- /**
- * Sets the boolean value at the current position.
- * Increments the position by Byte.BYTES.
- * @param value the value to put
- */
- void setBoolean(boolean value);
-
- /**
- * Sets the boolean value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start.
- * @param value the value to put
- */
- void setBoolean(
- long offsetBytes,
- boolean value);
-
- //intentionally removed putBooleanArray(...)
-
- /**
- * Sets the byte value at the current position.
- * Increments the position by Byte.BYTES.
- * @param value the value to put
- */
- void setByte(byte value);
-
- /**
- * Sets the byte value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param value the value to put
- */
- void setByte(
- long offsetBytes,
- byte value);
-
- /**
- * Sets the byte array at the current position.
- * Increments the position by Byte.BYTES * (lengthBytes - srcOffsetBytes).
- * @param srcArray The source array.
- * @param srcOffsetBytes offset in array units
- * @param lengthBytes number of array units to transfer
- */
- void setByteArray(
- byte[] srcArray,
- int srcOffsetBytes,
- int lengthBytes);
-
- /**
- * Sets the char value at the current position.
- * Increments the position by Character.BYTES.
- * @param value the value to put
- */
- void setChar(char value);
-
- /**
- * Sets the char value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param value the value to put
- */
- void setChar(
- long offsetBytes,
- char value);
-
- /**
- * Sets the char array at the current position.
- * Increments the position by Character.BYTES * (lengthChars - srcOffsetChars).
- * @param srcArray The source array.
- * @param srcOffsetChars offset in array units
- * @param lengthChars number of array units to transfer
- */
- void setCharArray(
- char[] srcArray,
- int srcOffsetChars,
- int lengthChars);
-
- /**
- * Sets the double value at the current position.
- * Increments the position by Double.BYTES.
- * @param value the value to put
- */
- void setDouble(double value);
-
- /**
- * Sets the double value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param value the value to put
- */
- void setDouble(
- long offsetBytes,
- double value);
-
- /**
- * Sets the double array at the current position.
- * Increments the position by Double.BYTES * (lengthDoubles - srcOffsetDoubles).
- * @param srcArray The source array.
- * @param srcOffsetDoubles offset in array units
- * @param lengthDoubles number of array units to transfer
- */
- void setDoubleArray(
- double[] srcArray,
- int srcOffsetDoubles,
- int lengthDoubles);
-
- /**
- * Sets the float value at the current position.
- * Increments the position by Float.BYTES.
- * @param value the value to put
- */
- void setFloat(float value);
-
- /**
- * Sets the float value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param value the value to put
- */
- void setFloat(
- long offsetBytes,
- float value);
-
- /**
- * Sets the float array at the current position.
- * Increments the position by Float.BYTES * (lengthFloats - srcOffsetFloats).
- * @param srcArray The source array.
- * @param srcOffsetFloats offset in array units
- * @param lengthFloats number of array units to transfer
- */
- void setFloatArray(
- float[] srcArray,
- int srcOffsetFloats,
- int lengthFloats);
-
- /**
- * Sets the int value at the current position.
- * Increments the position by Integer.BYTES.
- * @param value the value to put
- */
- void setInt(int value);
-
- /**
- * Sets the int value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param value the value to put
- */
- void setInt(
- long offsetBytes,
- int value);
-
- /**
- * Sets the int array at the current position.
- * Increments the position by Integer.BYTES * (lengthInts - srcOffsetInts).
- * @param srcArray The source array.
- * @param srcOffsetInts offset in array units
- * @param lengthInts number of array units to transfer
- */
- void setIntArray(
- int[] srcArray,
- int srcOffsetInts,
- int lengthInts);
-
- /**
- * Sets the long value at the current position.
- * Increments the position by Long.BYTES.
- * @param value the value to put
- */
- void setLong(long value);
-
- /**
- * Sets the long value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param value the value to put
- */
- void setLong(
- long offsetBytes,
- long value);
-
- /**
- * Sets the long array at the current position.
- * Increments the position by Long.BYTES * (lengthLongs - srcOffsetLongs).
- * @param srcArray The source array.
- * @param srcOffsetLongs offset in array units
- * @param lengthLongs number of array units to transfer
- */
- void setLongArray(
- long[] srcArray,
- int srcOffsetLongs,
- int lengthLongs);
-
- /**
- * Sets the short value at the current position.
- * Increments the position by Short.BYTES.
- * @param value the value to put
- */
- void setShort(short value);
-
- /**
- * Sets the short value at the given offset.
- * This does not change the position.
- * @param offsetBytes offset bytes relative to this MemorySegment start
- * @param value the value to put
- */
- void setShort(
- long offsetBytes,
- short value);
-
- /**
- * Sets the short array at the current position.
- * Increments the position by Short.BYTES * (lengthShorts - srcOffsetShorts).
- * @param srcArray The source array.
- * @param srcOffsetShorts offset in array units
- * @param lengthShorts number of array units to transfer
- */
- void setShortArray(
- short[] srcArray,
- int srcOffsetShorts,
- int lengthShorts);
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/positional/PositionalSegmentImpl.java b/src/main/java/org/apache/datasketches/common/positional/PositionalSegmentImpl.java
deleted file mode 100644
index 380f0bb20..000000000
--- a/src/main/java/org/apache/datasketches/common/positional/PositionalSegmentImpl.java
+++ /dev/null
@@ -1,336 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.common.positional;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_CHAR_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_DOUBLE_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_FLOAT_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_LONG_UNALIGNED;
-import static java.lang.foreign.ValueLayout.JAVA_SHORT_UNALIGNED;
-
-import java.lang.foreign.MemorySegment;
-
-/**
- * Implementation of PositionalSegment
- *
- * @author Lee Rhodes
- */
-final class PositionalSegmentImpl extends PositionalImpl implements PositionalSegment {
- private static final byte SHORT_SHIFT = 1;
- private static final byte CHAR_SHIFT = 1;
- private static final byte INT_SHIFT = 2;
- private static final byte LONG_SHIFT = 3;
- private static final byte FLOAT_SHIFT = 2;
- private static final byte DOUBLE_SHIFT = 3;
-
- private final MemorySegment seg;
-
- /**
- * Constructs with the given MemorySegment
- * @param seg the given MemorySegment
- */
- PositionalSegmentImpl(final MemorySegment seg) {
- super(seg.byteSize());
- this.seg = seg;
- }
-
- @Override
- public PositionalSegment asSlice() {
- final MemorySegment slice = seg.asSlice(getPosition());
- return new PositionalSegmentImpl(slice);
- }
-
- @Override
- public MemorySegment getMemorySegment() {
- return seg;
- }
-
- //PRIMITIVE getX() and getXArray()
-
- @Override
- public boolean getBoolean() {
- return getByte() != 0;
- }
-
- @Override
- public boolean getBoolean(final long offsetBytes) {
- return getByte(offsetBytes) != 0;
- }
-
- @Override
- public byte getByte() {
- final byte aByte = seg.get(JAVA_BYTE, getPosition());
- incrementPosition(Byte.BYTES);
- return aByte;
- }
-
- @Override
- public byte getByte(final long offsetBytes) {
- return seg.get(JAVA_BYTE, offsetBytes);
- }
-
- @Override
- public void getByteArray(final byte[] dstArray, final int dstOffsetBytes, final int lengthBytes) {
- MemorySegment.copy(seg, JAVA_BYTE, getPosition(), dstArray, dstOffsetBytes, lengthBytes);
- incrementPosition(lengthBytes);
- }
-
- @Override
- public char getChar() {
- final char achar = seg.get(JAVA_CHAR_UNALIGNED, getPosition());
- incrementPosition(Character.BYTES);
- return achar;
- }
-
- @Override
- public char getChar(final long offsetBytes) {
- return seg.get(JAVA_CHAR_UNALIGNED, offsetBytes);
- }
-
- @Override
- public void getCharArray(final char[] dstArray, final int dstOffsetChars, final int lengthChars) {
- MemorySegment.copy(seg, JAVA_CHAR_UNALIGNED, getPosition(), dstArray, dstOffsetChars, lengthChars);
- incrementPosition(lengthChars << CHAR_SHIFT);
- }
-
- @Override
- public double getDouble() {
- final double dbl = seg.get(JAVA_DOUBLE_UNALIGNED, getPosition());
- incrementPosition(Double.BYTES);
- return dbl;
- }
-
- @Override
- public double getDouble(final long offsetBytes) {
- return seg.get(JAVA_DOUBLE_UNALIGNED, offsetBytes);
- }
-
- @Override
- public void getDoubleArray(final double[] dstArray, final int dstOffsetDoubles, final int lengthDoubles) {
- MemorySegment.copy(seg, JAVA_DOUBLE_UNALIGNED, getPosition(), dstArray, dstOffsetDoubles, lengthDoubles);
- incrementPosition(lengthDoubles << DOUBLE_SHIFT);
- }
-
- @Override
- public float getFloat() {
- final float flt = seg.get(JAVA_FLOAT_UNALIGNED, getPosition());
- incrementPosition(Float.BYTES);
- return flt;
- }
-
- @Override
- public float getFloat(final long offsetBytes) {
- return seg.get(JAVA_FLOAT_UNALIGNED, offsetBytes);
- }
-
- @Override
- public void getFloatArray(final float[] dstArray, final int dstOffsetFloats, final int lengthFloats) {
- MemorySegment.copy(seg, JAVA_FLOAT_UNALIGNED, getPosition(), dstArray, dstOffsetFloats, lengthFloats);
- incrementPosition(lengthFloats << FLOAT_SHIFT);
- }
-
- @Override
- public int getInt() {
- final int i = seg.get(JAVA_INT_UNALIGNED, getPosition());
- incrementPosition(Integer.BYTES);
- return i;
- }
-
- @Override
- public int getInt(final long offsetBytes) {
- return seg.get(JAVA_INT_UNALIGNED, offsetBytes);
- }
-
- @Override
- public void getIntArray(final int[] dstArray, final int dstOffsetInts, final int lengthInts) {
- MemorySegment.copy(seg, JAVA_INT_UNALIGNED, getPosition(), dstArray, dstOffsetInts, lengthInts);
- incrementPosition(lengthInts << INT_SHIFT);
- }
-
- @Override
- public long getLong() {
- final long along = seg.get(JAVA_LONG_UNALIGNED, getPosition());
- incrementPosition(Long.BYTES);
- return along;
- }
-
- @Override
- public long getLong(final long offsetBytes) {
- return seg.get(JAVA_LONG_UNALIGNED, offsetBytes);
- }
-
- @Override
- public void getLongArray(final long[] dstArray, final int dstOffsetLongs, final int lengthLongs) {
- MemorySegment.copy(seg, JAVA_LONG_UNALIGNED, getPosition(), dstArray, dstOffsetLongs, lengthLongs);
- incrementPosition(lengthLongs << LONG_SHIFT);
- }
-
- @Override
- public short getShort() {
- final short ashort = seg.get(JAVA_SHORT_UNALIGNED, getPosition());
- incrementPosition(Short.BYTES);
- return ashort;
- }
-
- @Override
- public short getShort(final long offsetBytes) {
- return seg.get(JAVA_SHORT_UNALIGNED, offsetBytes);
- }
-
- @Override
- public void getShortArray(final short[] dstArray, final int dstOffsetShorts, final int lengthShorts) {
- MemorySegment.copy(seg, JAVA_SHORT_UNALIGNED, getPosition(), dstArray, dstOffsetShorts, lengthShorts);
- incrementPosition(lengthShorts << SHORT_SHIFT);
- }
-
- //PRIMITIVE setX() and setXArray() implementations
-
- @Override
- public void setBoolean(final boolean value) {
- setByte(value ? (byte)1 : 0);
- }
-
- @Override
- public void setBoolean(final long offsetBytes, final boolean value) {
- setByte(offsetBytes, value ? (byte)1 : 0);
- }
-
- @Override
- public void setByte(final byte value) {
- seg.set(JAVA_BYTE, getPosition(), value);
- incrementPosition(Byte.BYTES);
- }
-
- @Override
- public void setByte(final long offsetBytes, final byte value) {
- seg.set(JAVA_BYTE, offsetBytes, value);
- }
-
- @Override
- public void setByteArray(final byte[] srcArray, final int srcOffsetBytes, final int lengthBytes) {
- MemorySegment.copy(srcArray, srcOffsetBytes, seg, JAVA_BYTE, getPosition(), lengthBytes);
- incrementPosition(lengthBytes);
- }
-
- @Override
- public void setChar(final char value) {
- seg.set(JAVA_CHAR_UNALIGNED, getPosition(), value);
- incrementPosition(Character.BYTES);
- }
-
- @Override
- public void setChar(final long offsetBytes, final char value) {
- seg.set(JAVA_CHAR_UNALIGNED, offsetBytes, value);
- }
-
- @Override
- public void setCharArray(final char[] srcArray, final int srcOffsetChars, final int lengthChars) {
- MemorySegment.copy(srcArray, srcOffsetChars, seg, JAVA_CHAR_UNALIGNED, getPosition(), lengthChars);
- incrementPosition(lengthChars << CHAR_SHIFT);
- }
-
- @Override
- public void setDouble(final double value) {
- seg.set(JAVA_DOUBLE_UNALIGNED, getPosition(), value);
- incrementPosition(Double.BYTES);
- }
-
- @Override
- public void setDouble(final long offsetBytes, final double value) {
- seg.set(JAVA_DOUBLE_UNALIGNED, offsetBytes, value);
- }
-
- @Override
- public void setDoubleArray(final double[] srcArray, final int srcOffsetDoubles, final int lengthDoubles) {
- MemorySegment.copy(srcArray, srcOffsetDoubles, seg, JAVA_DOUBLE_UNALIGNED, getPosition(), lengthDoubles);
- incrementPosition(lengthDoubles << DOUBLE_SHIFT);
- }
-
- @Override
- public void setFloat(final float value) {
- seg.set(JAVA_FLOAT_UNALIGNED, getPosition(), value);
- incrementPosition(Float.BYTES);
- }
-
- @Override
- public void setFloat(final long offsetBytes, final float value) {
- seg.set(JAVA_FLOAT_UNALIGNED, offsetBytes, value);
- }
-
- @Override
- public void setFloatArray(final float[] srcArray, final int srcOffsetFloats, final int lengthFloats) {
- MemorySegment.copy(srcArray, srcOffsetFloats, seg, JAVA_FLOAT_UNALIGNED, getPosition(), lengthFloats);
- incrementPosition(lengthFloats << FLOAT_SHIFT);
- }
-
- @Override
- public void setInt(final int value) {
- seg.set(JAVA_INT_UNALIGNED, getPosition(), value);
- incrementPosition(Integer.BYTES);
- }
-
- @Override
- public void setInt(final long offsetBytes, final int value) {
- seg.set(JAVA_INT_UNALIGNED, offsetBytes, value);
- }
-
- @Override
- public void setIntArray(final int[] srcArray, final int srcOffsetInts, final int lengthInts) {
- MemorySegment.copy(srcArray, srcOffsetInts, seg, JAVA_INT_UNALIGNED, getPosition(), lengthInts);
- incrementPosition(lengthInts << INT_SHIFT);
- }
-
- @Override
- public void setLong(final long value) {
- seg.set(JAVA_LONG_UNALIGNED, getPosition(), value);
- incrementPosition(Long.BYTES);
- }
-
- @Override
- public void setLong(final long offsetBytes, final long value) {
- seg.set(JAVA_LONG_UNALIGNED, offsetBytes, value);
- }
-
- @Override
- public void setLongArray(final long[] srcArray, final int srcOffsetLongs, final int lengthLongs) {
- MemorySegment.copy(srcArray, srcOffsetLongs, seg, JAVA_LONG_UNALIGNED, getPosition(), lengthLongs);
- incrementPosition(lengthLongs << LONG_SHIFT);
- }
-
- @Override
- public void setShort(final short value) {
- seg.set(JAVA_SHORT_UNALIGNED, getPosition(), value);
- incrementPosition(Short.BYTES);
- }
-
- @Override
- public void setShort(final long offsetBytes, final short value) {
- seg.set(JAVA_SHORT_UNALIGNED, offsetBytes, value);
- }
-
- @Override
- public void setShortArray(final short[] srcArray, final int srcOffsetShorts, final int lengthShorts) {
- MemorySegment.copy(srcArray, srcOffsetShorts, seg, JAVA_SHORT_UNALIGNED, getPosition(), lengthShorts);
- incrementPosition(lengthShorts << SHORT_SHIFT);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/common/positional/package-info.java b/src/main/java/org/apache/datasketches/common/positional/package-info.java
deleted file mode 100644
index 9049e3573..000000000
--- a/src/main/java/org/apache/datasketches/common/positional/package-info.java
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * This package provides the classes that define and implement a relative positional API.
- *
- * @see org.apache.datasketches.common.positional.Positional
- *
- * @author Lee Rhodes
- */
-package org.apache.datasketches.common.positional;
diff --git a/src/main/java/org/apache/datasketches/count/CountMinSketch.java b/src/main/java/org/apache/datasketches/count/CountMinSketch.java
deleted file mode 100644
index 9bf576e13..000000000
--- a/src/main/java/org/apache/datasketches/count/CountMinSketch.java
+++ /dev/null
@@ -1,493 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.count;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_LONG_UNALIGNED;
-
-import java.lang.foreign.MemorySegment;
-import java.nio.charset.StandardCharsets;
-import java.util.Random;
-
-import org.apache.datasketches.common.Family;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesException;
-import org.apache.datasketches.common.Util;
-import org.apache.datasketches.common.positional.PositionalSegment;
-import org.apache.datasketches.hash.MurmurHash3;
-
-/**
- * Java implementation of the CountMin sketch data structure of Cormode and Muthukrishnan.
- * This implementation is inspired by and compatible with the datasketches-cpp version by Charlie Dickens.
- *
- * The CountMin sketch is a probabilistic data structure that provides frequency estimates for items
- * in a data stream. It uses multiple hash functions to distribute items across a two-dimensional array,
- * providing approximate counts with configurable error bounds.
- *
- * Reference: http://dimacs.rutgers.edu/~graham/pubs/papers/cm-full.pdf
- */
-public class CountMinSketch {
- private final byte numHashes_;
- private final int numBuckets_;
- private final long seed_;
- private final long[] hashSeeds_;
- private final long[] sketchArray_;
- private long totalWeight_;
-
- // Thread-local MemorySegment to avoid allocations in hot paths with explicit endianness control
- private static final ThreadLocal
- * Format = EMPTY_MERGED/EMPTY_HIP: NoWindow, NoSV, HIP or NoHIP = 00X.
- * The first 8 bytes are common for all Formats.
- * PI = 2, FIcol = 0
- * Long adr ||
- * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
- * 0 ||---SEED HASH-----|-Flags--|-FIcol--|---lgK--|-FamID--|-SerVer-|---PI---|
- *
- *
- * Format = SPARSE_HYBRID_MERGED: {NoWindow, SV, NoHIP} = 2 = 010.
- * PI = 4, FIcol = 0
- * Long adr ||
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||-------------SV Length Ints--------|--------numCoupons = numSV---------|
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 || |----------Start SV bit stream------|
- *
- *
- * Format = SPARSE_HYBRID_HIP: {NoWindow, SV, HIP} = 3 = 011
- * PI = 8, FIcol = 0
- * Long adr ||
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||-------------SV Length Ints--------|--------numCoupons = numSV---------|
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 ||----------------------------------KxP----------------------------------|
- *
- * || 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 |
- * 3 ||-------------------------------HIP Accum-------------------------------|
- *
- * || 39 | 38 | 37 | 36 | 35 | 34 | 33 | 32 |
- * 4 || |----------Start of SV stream-------|
- *
- *
- * Format = PINNED_SLIDING_MERGED_NOSV: {Window, No SV, NoHIP} = 4 = 100
- * PI = 4, FIcol = valid
- * Long adr ||
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||----------Window Length Ints-------|------------numCoupons-------------|
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 || |--------Start of Window stream-----|
- *
- *
- * Format = PINNED_SLIDING_HIP_NOSV: {Window, No SV, HIP} = 5 = 101
- * PI = 8, FIcol = valid
- * Long adr ||
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||----------Window Length Ints-------|------------numCoupons-------------|
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 ||----------------------------------KxP----------------------------------|
- *
- * || 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 |
- * 3 ||-------------------------------HIP Accum-------------------------------|
- *
- * || 39 | 38 | 37 | 36 | 35 | 34 | 33 | 32 |
- * 4 || |--------Start of Window stream-----|
- *
- *
- * Format = PINNED_SLIDING_MERGED: {Window, SV, NoHIP} = 6 = 110
- * PI = 6, FIcol = valid
- * Long adr ||
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||---------------numSV---------------|------------numCoupons-------------|
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 ||----------Window Length Ints-------|-------------SV Length Ints--------|
- *
- * || XX | XX | XX | XX | 27 | 26 | 25 | 24 |
- * 3 ||--------Start of SV stream---------|--------Start of Window stream-----|
- *
- *
- * Format = PINNED_SLIDING_HIP: {Window, SV, HIP} = 7 = 111
- * PI = 10, FIcol = valid
- * Long adr ||
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||---------------numSV---------------|------------numCoupons-------------|
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 ||----------------------------------KxP----------------------------------|
- *
- * || 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 |
- * 3 ||-------------------------------HIP Accum-------------------------------|
- *
- * || 39 | 38 | 37 | 36 | 35 | 34 | 33 | 32 |
- * 4 ||----------Window Length Ints-------|-------------SV Length Ints--------|
- *
- * || XX | XX | XX | XX | 43 | 42 | 41 | 40 |
- * 5 ||--------Start of SV stream---------|--------Start of Window stream-----|
- *
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class PreambleUtil {
-
- private PreambleUtil() {}
-
- private static final String fmt = "%10d%10x";
-
- /**
- * The serialization version for the set of serialization formats defined in this class
- */
- static final byte SER_VER = 1;
-
- //Flag bit masks, Byte 5
- static final int RESERVED_FLAG_MASK = 1; //Reserved.
- static final int COMPRESSED_FLAG_MASK = 2;
- static final int HIP_FLAG_MASK = 4;
- static final int SUP_VAL_FLAG_MASK = 8; //num Surprising Values > 0
- static final int WINDOW_FLAG_MASK = 16;//window length > 0
-
- //PREAMBLE SIZE
-
- /**
- * This defines the preamble space required by each of the formats in units of 4-byte integers.
- */
- private static final byte[] preIntDefs = { 2, 2, 4, 8, 4, 8, 6, 10 };
-
- /**
- * Returns the defined size of the preamble in units of integers (4 bytes) given the
- * Format.
- * @param format the given Format.
- * @return the defined size of the preamble in units of integers (4 bytes) given the Format.
- */
- static byte getDefinedPreInts(final Format format) {
- return preIntDefs[format.ordinal()];
- }
-
- //PREAMBLE LO_FIELD DEFINITIONS, OFFSETS, AND GETS
-
- /**
- * This defines the seven fields of the first eight bytes of the preamble.
- * The ordinal of these values defines the byte offset.
- * Do not change the order.
- */
- enum LoField { PRE_INTS, SER_VERSION, FAMILY, LG_K, FI_COL, FLAGS, SEED_HASH }
-
- /**
- * Returns the defined byte offset from the start of the preamble given a LoField.
- * This only applies to the first 8 bytes of the preamble.
- * @param loField the given LoField.
- * @return the defined byte offset from the start of the preamble given a LoField.
- */
- static int getLoFieldOffset(final LoField loField) {
- return loField.ordinal();
- }
-
- static int getPreInts(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, getLoFieldOffset(LoField.PRE_INTS)) & 0XFF;
- }
-
- static int getSerVer(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, getLoFieldOffset(LoField.SER_VERSION)) & 0XFF;
- }
-
- static Family getFamily(final MemorySegment seg) {
- final int fam = seg.get(JAVA_BYTE, getLoFieldOffset(LoField.FAMILY)) & 0XFF;
- return Family.idToFamily(fam);
- }
-
- static int getLgK(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, getLoFieldOffset(LoField.LG_K)) & 0XFF;
- }
-
- static int getFiCol(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, getLoFieldOffset(LoField.FI_COL)) & 0XFF;
- }
-
- static int getFlags(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, getLoFieldOffset(LoField.FLAGS)) & 0XFF;
- }
-
- static short getSeedHash(final MemorySegment seg) {
- return seg.get(JAVA_SHORT_UNALIGNED, getLoFieldOffset(LoField.SEED_HASH));
- }
-
- static int getFormatOrdinal(final MemorySegment seg) {
- final int flags = getFlags(seg);
- return (flags >>> 2) & 0x7;
- }
-
- static Format getFormat(final MemorySegment seg) {
- final int ordinal = getFormatOrdinal(seg);
- return Format.ordinalToFormat(ordinal);
- }
-
- static boolean hasHip(final MemorySegment seg) {
- return (getFlags(seg) & HIP_FLAG_MASK) > 0;
- }
-
- static boolean hasSv(final MemorySegment seg) {
- return (getFlags(seg) & SUP_VAL_FLAG_MASK) > 0;
- }
-
- static boolean hasWindow(final MemorySegment seg) {
- return (getFlags(seg) & WINDOW_FLAG_MASK) > 0;
- }
-
- static boolean isCompressed(final MemorySegment seg) {
- return (getFlags(seg) & COMPRESSED_FLAG_MASK) > 0;
- }
-
- //PREAMBLE HI_FIELD DEFINITIONS
-
- /**
- * This defines the eight additional preamble fields located after the LoField.
- * Do not change the order.
- *
- *
- * Long || Start Byte Adr:
- * Adr:
- * || 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
- * 0 || Preamble_Longs | SerVer | FamID | Flags |----Num Hashes---|-----Unused------|
- *
- * || 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
- * 1 ||---------------------------------Hash Seed-------------------------------------|
- *
- * || 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 |
- * 2 ||-------BitArray Length (in longs)----------|-----------Unused------------------|
- *
- * || 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 |
- * 3 ||---------------------------------NumBitsSet------------------------------------|
- *
- *
- * The raw BitArray bits, if non-empty, start at byte 32.
- */
-
- /**
- * Serializes the current BloomFilter to an array of bytes.
- *
- * Integer.MAX_VALUE.
- *
- *
- *
- *
- * 1 For speed we do employ some randomization that introduces a small probability that
- * our proof of the worst-case bound might not apply to a given run. However, we have ensured
- * that this probability is extremely small. For example, if the stream causes one table purge
- * (rebuild), our proof of the worst case bound applies with probability at least 1 - 1E-14.
- * If the stream causes 1E9 purges, our proof applies with probability at least 1 - 1E-5.
- *
- * @param
- * If (x.equals(y)) implies: x.hashCode() == y.hashCode().
- * If (!x.equals(y)) does NOT imply: x.hashCode() != y.hashCode().
- * @return the hashCode computed from getEstimate().
- */
- @Override
- public int hashCode() {
- final int prime = 31;
- final int result = 1;
- return (prime * result) + (int) (est ^ (est >>> 32));
- }
-
- /**
- * This equals is computed only from the Row.getEstimate() value and does not imply equality
- * of the other items within the row: item and upper and lower bounds.
- * Defined this way, this equals will be consistent with compareTo(Row).
- * @param obj the other row to determine equality with.
- * @return true if this.getEstimate() equals ((Row<T>)obj).getEstimate().
- */
- @SuppressWarnings("unchecked")
- @Override
- public boolean equals(final Object obj) {
- if (this == obj) { return true; }
- if ( (obj == null) || !(obj instanceof Row)) { return false; }
- final Row
- *
- *
- *
- *
- * 1 For speed we do employ some randomization that introduces a small probability that
- * our proof of the worst-case bound might not apply to a given run. However, we have ensured
- * that this probability is extremely small. For example, if the stream causes one table purge
- * (rebuild), our proof of the worst case bound applies with probability at least 1 - 1E-14.
- * If the stream causes 1E9 purges, our proof applies with probability at least 1 - 1E-5.
- *
- * @author Justin Thaler
- * @author Lee Rhodes
- */
-@SuppressFBWarnings(value = "SIC_INNER_SHOULD_BE_STATIC_ANON", justification = "Harmless, fix later")
-public class FrequentLongsSketch {
-
- private static final int STR_PREAMBLE_TOKENS = 6;
-
- /**
- * Log2 Maximum length of the arrays internal to the hash map supported by the data
- * structure.
- */
- private final int lgMaxMapSize;
-
- /**
- * The current number of counters supported by the hash map.
- */
- private int curMapCap; //the threshold to purge
-
- /**
- * Tracks the total of decremented counts.
- */
- private long offset;
-
- /**
- * The sum of all frequencies of the stream so far.
- */
- private long streamWeight = 0;
-
- /**
- * The maximum number of samples used to compute approximate median of counters when doing
- * decrement
- */
- private final int sampleSize;
-
- /**
- * Hash map mapping stored items to approximate counts
- */
- private ReversePurgeLongHashMap hashMap;
-
- /**
- * Construct this sketch with the parameter maxMapSize and the default initialMapSize (8).
- *
- * @param maxMapSize Determines the physical size of the internal hash map managed by this
- * sketch and must be a power of 2. The maximum capacity of this internal hash map is
- * 0.75 times * maxMapSize. Both the ultimate accuracy and size of this sketch are a
- * function of maxMapSize.
- */
- public FrequentLongsSketch(final int maxMapSize) {
- this(exactLog2OfInt(maxMapSize, "maxMapSize"), LG_MIN_MAP_SIZE);
- }
-
- /**
- * Construct this sketch with parameter lgMapMapSize and lgCurMapSize. This internal
- * constructor is used when deserializing the sketch.
- *
- * @param lgMaxMapSize Log2 of the physical size of the internal hash map managed by this
- * sketch. The maximum capacity of this internal hash map is 0.75 times 2^lgMaxMapSize.
- * Both the ultimate accuracy and size of this sketch are a function of lgMaxMapSize.
- *
- * @param lgCurMapSize Log2 of the starting (current) physical size of the internal hash
- * map managed by this sketch.
- */
- FrequentLongsSketch(final int lgMaxMapSize, final int lgCurMapSize) {
- //set initial size of hash map
- this.lgMaxMapSize = Math.max(lgMaxMapSize, LG_MIN_MAP_SIZE);
- final int lgCurMapSz = Math.max(lgCurMapSize, LG_MIN_MAP_SIZE);
- hashMap = new ReversePurgeLongHashMap(1 << lgCurMapSz);
- curMapCap = hashMap.getCapacity();
- final int maxMapCap =
- (int) ((1 << lgMaxMapSize) * ReversePurgeLongHashMap.getLoadFactor());
- offset = 0;
- sampleSize = Math.min(SAMPLE_SIZE, maxMapCap);
- }
-
- /**
- * Returns a sketch instance of this class from the given srcSeg,
- * which must be a MemorySegment representation of this sketch class.
- *
- * @param srcSeg a MemorySegment representation of a sketch of this class.
- * @return a sketch instance of this class.
- */
- public static FrequentLongsSketch getInstance(final MemorySegment srcSeg) {
- Objects.requireNonNull(srcSeg, "Source MemorySegment must not be null.");
- final long pre0 = PreambleUtil.checkPreambleSize(srcSeg); //check MemorySegment capacity
- final int maxPreLongs = Family.FREQUENCY.getMaxPreLongs();
-
- final int preLongs = extractPreLongs(pre0); //Byte 0
- final int serVer = extractSerVer(pre0); //Byte 1
- final int familyID = extractFamilyID(pre0); //Byte 2
- final int lgMaxMapSize = extractLgMaxMapSize(pre0); //Byte 3
- final int lgCurMapSize = extractLgCurMapSize(pre0); //Byte 4
- final boolean empty = (extractFlags(pre0) & EMPTY_FLAG_MASK) != 0; //Byte 5
-
- // Checks
- final boolean preLongsEq1 = (preLongs == 1); //Byte 0
- final boolean preLongsEqMax = (preLongs == maxPreLongs);
- if (!preLongsEq1 && !preLongsEqMax) {
- throw new SketchesArgumentException(
- "Possible Corruption: PreLongs must be 1 or " + maxPreLongs + ": " + preLongs);
- }
- if (serVer != SER_VER) { //Byte 1
- throw new SketchesArgumentException(
- "Possible Corruption: Ser Ver must be " + SER_VER + ": " + serVer);
- }
- final int actFamID = Family.FREQUENCY.getID(); //Byte 2
- if (familyID != actFamID) {
- throw new SketchesArgumentException(
- "Possible Corruption: FamilyID must be " + actFamID + ": " + familyID);
- }
- if (empty ^ preLongsEq1) { //Byte 5 and Byte 0
- throw new SketchesArgumentException(
- "Possible Corruption: (PreLongs == 1) ^ Empty == True.");
- }
-
- if (empty) {
- return new FrequentLongsSketch(lgMaxMapSize, LG_MIN_MAP_SIZE);
- }
- //get full preamble
- final long[] preArr = new long[preLongs];
- MemorySegment.copy(srcSeg, JAVA_LONG_UNALIGNED, 0, preArr, 0, preLongs);
-
- final FrequentLongsSketch fls = new FrequentLongsSketch(lgMaxMapSize, lgCurMapSize);
- fls.streamWeight = 0; //update after
- fls.offset = preArr[3];
-
- final int preBytes = preLongs << 3;
- final int activeItems = extractActiveItems(preArr[1]);
-
- //Get countArray
- final long[] countArray = new long[activeItems];
- final int reqBytes = preBytes + (2 * activeItems * Long.BYTES); //count Arr + Items Arr
- checkBounds(0, reqBytes, srcSeg.byteSize()); //check MemorySegment capacity
- MemorySegment.copy(srcSeg, JAVA_LONG_UNALIGNED, preBytes, countArray, 0, activeItems);
-
- //Get itemArray
- final int itemsOffset = preBytes + (Long.BYTES * activeItems);
- final long[] itemArray = new long[activeItems];
- MemorySegment.copy(srcSeg, JAVA_LONG_UNALIGNED, itemsOffset, itemArray, 0, activeItems);
- //update the sketch
- for (int i = 0; i < activeItems; i++) {
- fls.update(itemArray[i], countArray[i]);
- }
- fls.streamWeight = preArr[2]; //override streamWeight due to updating
- return fls;
- }
-
- /**
- * Returns a sketch instance of this class from the given String,
- * which must be a String representation of this sketch class.
- *
- * @param string a String representation of a sketch of this class.
- * @return a sketch instance of this class.
- */
- public static FrequentLongsSketch getInstance(final String string) {
- Objects.requireNonNull(string, "string must not be null.");
- final String[] tokens = string.split(",");
- if (tokens.length < (STR_PREAMBLE_TOKENS + 2)) {
- throw new SketchesArgumentException(
- "String not long enough: " + tokens.length);
- }
- final int serVer = Integer.parseInt(tokens[0]);
- final int famID = Integer.parseInt(tokens[1]);
- final int lgMax = Integer.parseInt(tokens[2]);
- final int flags = Integer.parseInt(tokens[3]);
- final long streamWt = Long.parseLong(tokens[4]);
- final long offset = Long.parseLong(tokens[5]); //error offset
- //should always get at least the next 2 from the map
- final int numActive = Integer.parseInt(tokens[6]);
- final int lgCur = Integer.numberOfTrailingZeros(Integer.parseInt(tokens[7]));
-
- //checks
- if (serVer != SER_VER) {
- throw new SketchesArgumentException("Possible Corruption: Bad SerVer: " + serVer);
- }
- Family.FREQUENCY.checkFamilyID(famID);
- final boolean empty = flags > 0;
- if (!empty && (numActive == 0)) {
- throw new SketchesArgumentException(
- "Possible Corruption: !Empty && NumActive=0; strLen: " + numActive);
- }
- final int numTokens = tokens.length;
- if ((2 * numActive) != (numTokens - STR_PREAMBLE_TOKENS - 2)) {
- throw new SketchesArgumentException(
- "Possible Corruption: Incorrect # of tokens: " + numTokens
- + ", numActive: " + numActive);
- }
-
- final FrequentLongsSketch sketch = new FrequentLongsSketch(lgMax, lgCur);
- sketch.streamWeight = streamWt;
- sketch.offset = offset;
- sketch.hashMap = deserializeFromStringArray(tokens);
- return sketch;
- }
-
- /**
- * Returns the estimated a priori error given the maxMapSize for the sketch and the
- * estimatedTotalStreamWeight.
- * @param maxMapSize the planned map size to be used when constructing this sketch.
- * @param estimatedTotalStreamWeight the estimated total stream weight.
- * @return the estimated a priori error.
- */
- public static double getAprioriError(final int maxMapSize, final long estimatedTotalStreamWeight) {
- return getEpsilon(maxMapSize) * estimatedTotalStreamWeight;
- }
-
- /**
- * Returns the current number of counters the sketch is configured to support.
- *
- * @return the current number of counters the sketch is configured to support.
- */
- public int getCurrentMapCapacity() {
- return curMapCap;
- }
-
- /**
- * Returns epsilon used to compute a priori error.
- * This is just the value 3.5 / maxMapSize.
- * @param maxMapSize the planned map size to be used when constructing this sketch.
- * @return epsilon used to compute a priori error.
- */
- public static double getEpsilon(final int maxMapSize) {
- if (!isPowerOf2(maxMapSize)) {
- throw new SketchesArgumentException("maxMapSize is not a power of 2.");
- }
- return 3.5 / maxMapSize;
- }
-
- /**
- * Gets the estimate of the frequency of the given item.
- * Note: The true frequency of a item would be the sum of the counts as a result of the
- * two update functions.
- *
- * @param item the given item
- * @return the estimate of the frequency of the given item
- */
- public long getEstimate(final long item) {
- // If item is tracked:
- // Estimate = itemCount + offset; Otherwise it is 0.
- final long itemCount = hashMap.get(item);
- return (itemCount > 0) ? itemCount + offset : 0;
- }
-
- /**
- * Gets the guaranteed lower bound frequency of the given item, which can never be
- * negative.
- *
- * @param item the given item.
- * @return the guaranteed lower bound frequency of the given item. That is, a number which
- * is guaranteed to be no larger than the real frequency.
- */
- public long getLowerBound(final long item) {
- //LB = itemCount or 0
- return hashMap.get(item);
- }
-
- /**
- * Returns an array of Rows that include frequent items, estimates, upper and lower bounds
- * given a threshold and an ErrorCondition. If the threshold is lower than getMaximumError(),
- * then getMaximumError() will be used instead.
- *
- *
- * If (x.equals(y)) implies: x.hashCode() == y.hashCode().
- * If (!x.equals(y)) does NOT imply: x.hashCode() != y.hashCode().
- * @return the hashCode computed from getEstimate().
- */
- @Override
- public int hashCode() {
- final int prime = 31;
- final int result = 1;
- return (prime * result) + (int) (est ^ (est >>> 32));
- }
-
- /**
- * This equals is computed only from the Row.getEstimate() value and does not imply equality
- * of the other items within the row: item and upper and lower bounds.
- * Defined this way, this equals will be consistent with compareTo(Row).
- * @param obj the other row to determine equality with.
- * @return true if this.getEstimate() equals ((Row)obj).getEstimate().
- */
- @Override
- public boolean equals(final Object obj) {
- if (this == obj) { return true; }
- if ( (obj == null) || !(obj instanceof Row)) { return false; }
- final Row that = (Row) obj;
- if (est != that.est) { return false; }
- return true;
- }
-
- } // End of class Row
-
- Row[] sortItems(final long threshold, final ErrorType errorType) {
- final ArrayList
- * * Long || Start Byte Adr:
- * Adr:
- * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
- * 0 ||------ unused -----|-Flags--|-LgCur--| LgMax | FamID | SerVer | PreambleLongs |
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||------------(unused)-----------------|--------ActiveItems------------------------|
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 ||-----------------------------------streamLength----------------------------------|
- * || 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 |
- * 3 ||---------------------------------offset------------------------------------------|
- * || 39 | 38 | 37 | 36 | 35 | 34 | 33 | 32 |
- * 5 ||----------start of values buffer, followed by keys buffer------------------------|
- *
- *
- * @author Lee Rhodes
- */
-final class PreambleUtil {
-
- private PreambleUtil() {}
-
- // ###### DO NOT MESS WITH THIS FROM HERE ...
- // Preamble byte Addresses
- static final int PREAMBLE_LONGS_BYTE = 0; // either 1 or 4
- static final int SER_VER_BYTE = 1;
- static final int FAMILY_BYTE = 2;
- static final int LG_MAX_MAP_SIZE_BYTE = 3;
- static final int LG_CUR_MAP_SIZE_BYTE = 4;
- static final int FLAGS_BYTE = 5;
- static final int SER_DE_ID_SHORT = 6; // to 7
- static final int ACTIVE_ITEMS_INT = 8; // to 11 : 0 to 4 in pre1
- static final int STREAMLENGTH_LONG = 16; // to 23 : pre2
- static final int OFFSET_LONG = 24; // to 31 : pre3
-
- // flag bit masks
- // due to a mistake different bits were used in C++ and Java to indicate empty sketch
- // therefore both are set and checked for compatibility with historical binary format
- static final int EMPTY_FLAG_MASK = 5;
-
- // Specific values for this implementation
- static final int SER_VER = 1;
-
- /**
- * Returns a human readable string summary of the preamble state of the given MemorySegment.
- * Note: other than making sure that the given MemorySegment size is large
- * enough for just the preamble, this does not do much value checking of the contents of the
- * preamble as this is primarily a tool for debugging the preamble visually.
- *
- * @param srcSeg the given MemorySegment
- * @return the summary preamble string.
- */
- public static String preambleToString(final MemorySegment srcSeg) {
- final long pre0 = checkPreambleSize(srcSeg); //make sure we can get the assumed preamble
- final int preLongs = extractPreLongs(pre0); //byte 0
- final int serVer = extractSerVer(pre0); //byte 1
- final Family family = Family.idToFamily(extractFamilyID(pre0)); //byte 2
- final int lgMaxMapSize = extractLgMaxMapSize(pre0); //byte 3
- final int lgCurMapSize = extractLgCurMapSize(pre0); //byte 4
- final int flags = extractFlags(pre0); //byte 5
-
- final String flagsStr = zeroPad(Integer.toBinaryString(flags), 8) + ", " + (flags);
- final boolean empty = (flags & EMPTY_FLAG_MASK) > 0;
- final int maxMapSize = 1 << lgMaxMapSize;
- final int curMapSize = 1 << lgCurMapSize;
- final int maxPreLongs = Family.FREQUENCY.getMaxPreLongs();
-
- //Assumed if preLongs == 1
- int activeItems = 0;
- long streamLength = 0;
- long offset = 0;
-
- //Assumed if preLongs == maxPreLongs
-
- if (preLongs == maxPreLongs) {
- //get full preamble
- final long[] preArr = new long[preLongs];
- MemorySegment.copy(srcSeg, JAVA_LONG_UNALIGNED, 0, preArr, 0, preLongs);
- activeItems = extractActiveItems(preArr[1]);
- streamLength = preArr[2];
- offset = preArr[3];
- }
-
- final StringBuilder sb = new StringBuilder();
- sb.append(LS)
- .append("### FREQUENCY SKETCH PREAMBLE SUMMARY:").append(LS)
- .append("Byte 0: Preamble Longs : ").append(preLongs).append(LS)
- .append("Byte 1: Serialization Version: ").append(serVer).append(LS)
- .append("Byte 2: Family : ").append(family.toString()).append(LS)
- .append("Byte 3: MaxMapSize : ").append(maxMapSize).append(LS)
- .append("Byte 4: CurMapSize : ").append(curMapSize).append(LS)
- .append("Byte 5: Flags Field : ").append(flagsStr).append(LS)
- .append(" EMPTY : ").append(empty).append(LS);
-
- if (preLongs == 1) {
- sb.append(" --ABSENT, ASSUMED:").append(LS);
- } else { //preLongs == maxPreLongs
- sb.append("Bytes 8-11 : ActiveItems : ").append(activeItems).append(LS);
- sb.append("Bytes 16-23: StreamLength : ").append(streamLength).append(LS)
- .append("Bytes 24-31: Offset : ").append(offset).append(LS);
- }
-
- sb.append( "Preamble Bytes : ").append(preLongs * 8).append(LS);
- sb.append( "TOTAL Sketch Bytes : ").append((preLongs + (activeItems * 2)) << 3)
- .append(LS)
- .append("### END FREQUENCY SKETCH PREAMBLE SUMMARY").append(LS);
- return sb.toString();
- }
-
- // @formatter:on
-
- static int extractPreLongs(final long pre0) { //Byte 0
- final long mask = 0X3FL; //Lower 6 bits
- return (int) (pre0 & mask);
- }
-
- static int extractSerVer(final long pre0) { //Byte 1
- final int shift = SER_VER_BYTE << 3;
- final long mask = 0XFFL;
- return (int) ((pre0 >>> shift) & mask);
- }
-
- static int extractFamilyID(final long pre0) { //Byte 2
- final int shift = FAMILY_BYTE << 3;
- final long mask = 0XFFL;
- return (int) ((pre0 >>> shift) & mask);
- }
-
- static int extractLgMaxMapSize(final long pre0) { //Byte 3
- final int shift = LG_MAX_MAP_SIZE_BYTE << 3;
- final long mask = 0XFFL;
- return (int) ((pre0 >>> shift) & mask);
- }
-
- static int extractLgCurMapSize(final long pre0) { //Byte 4
- final int shift = LG_CUR_MAP_SIZE_BYTE << 3;
- final long mask = 0XFFL;
- return (int) ((pre0 >>> shift) & mask);
- }
-
- static int extractFlags(final long pre0) { //Byte 5
- final int shift = FLAGS_BYTE << 3;
- final long mask = 0XFFL;
- return (int) ((pre0 >>> shift) & mask);
- }
-
- static int extractActiveItems(final long pre1) { //Bytes 8 to 11
- final long mask = 0XFFFFFFFFL;
- return (int) (pre1 & mask) ;
- }
-
- static long insertPreLongs(final int preLongs, final long pre0) { //Byte 0
- final long mask = 0X3FL; //Lower 6 bits
- return (preLongs & mask) | (~mask & pre0);
- }
-
- static long insertSerVer(final int serVer, final long pre0) { //Byte 1
- final int shift = SER_VER_BYTE << 3;
- final long mask = 0XFFL;
- return ((serVer & mask) << shift) | (~(mask << shift) & pre0);
- }
-
- static long insertFamilyID(final int familyID, final long pre0) { //Byte 2
- final int shift = FAMILY_BYTE << 3;
- final long mask = 0XFFL;
- return ((familyID & mask) << shift) | (~(mask << shift) & pre0);
- }
-
- static long insertLgMaxMapSize(final int lgMaxMapSize, final long pre0) { //Byte 3
- final int shift = LG_MAX_MAP_SIZE_BYTE << 3;
- final long mask = 0XFFL;
- return ((lgMaxMapSize & mask) << shift) | (~(mask << shift) & pre0);
- }
-
- static long insertLgCurMapSize(final int lgCurMapSize, final long pre0) { //Byte 4
- final int shift = LG_CUR_MAP_SIZE_BYTE << 3;
- final long mask = 0XFFL;
- return ((lgCurMapSize & mask) << shift) | (~(mask << shift) & pre0);
- }
-
- static long insertFlags(final int flags, final long pre0) { //Byte 5
- final int shift = FLAGS_BYTE << 3;
- final long mask = 0XFFL;
- return ((flags & mask) << shift) | (~(mask << shift) & pre0);
- }
-
- static long insertActiveItems(final int activeItems, final long pre1) { //Bytes 8 to 11
- final long mask = 0XFFFFFFFFL;
- return (activeItems & mask) | (~mask & pre1);
- }
-
- /**
- * Checks MemorySegment for capacity to hold the preamble and returns the first 8 bytes.
- * @param seg the given MemorySegment
- * @return the first 8 bytes of preamble as a long.
- */
- static long checkPreambleSize(final MemorySegment seg) {
- final long cap = seg.byteSize();
- if (cap < 8) { throwNotBigEnough(cap, 8); }
- final long pre0 = seg.get(JAVA_LONG_UNALIGNED, 0);
- final int preLongs = (int) (pre0 & 0X3FL); //lower 6 bits
- final int required = Math.max(preLongs << 3, 8);
- if (cap < required) { throwNotBigEnough(cap, required); }
- return pre0;
- }
-
- private static void throwNotBigEnough(final long cap, final int required) {
- throw new SketchesArgumentException(
- "Possible Corruption: "
- + "Size of byte array or MemorySegment not large enough for Preamble: Size: " + cap
- + ", Required: " + required);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/frequencies/ReversePurgeItemHashMap.java b/src/main/java/org/apache/datasketches/frequencies/ReversePurgeItemHashMap.java
deleted file mode 100644
index 7f998f505..000000000
--- a/src/main/java/org/apache/datasketches/frequencies/ReversePurgeItemHashMap.java
+++ /dev/null
@@ -1,388 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.frequencies;
-
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.common.Util.exactLog2OfInt;
-import static org.apache.datasketches.frequencies.Util.hash;
-
-import java.lang.reflect.Array;
-
-import org.apache.datasketches.common.QuickSelect;
-
-/**
- * Implements a linear-probing based hash map of (key, value) pairs and is distinguished by a
- * "reverse" purge operation that removes all keys in the map whose associated values are ≤ 0
- * and is performed in reverse, starting at the "back" of the array and moving toward the front.
- *
- * @param Z := ( sum[j=1,m](2^(-M[j])) )^(-1).
- * But the HIP estimator requires a computation of the probability defined above.
- * We accomplish both by redefining Z as
- * Z := ( m + sum[j=1,m](2^(-M[j] - 1)) )^(-1).
- * They are mathematically equivalent since:
- * m + sum[j=1,m](2^(-M[j] - 1)) == m + sum[j=1,m](2^(-M[j])) - m == sum[j=1,m](2^(-M[j])).
- *
- * @return KxQ0
- */
- abstract double getKxQ0();
-
- /**
- * This second KxQ register is shifted by 32 bits to give us more than 90 bits of mantissa
- * precision, which produces more accurate results for very large counts.
- * @return KxQ1
- */
- abstract double getKxQ1();
-
- @Override
- double getLowerBound(final int numStdDev) {
- HllUtil.checkNumStdDev(numStdDev);
- return HllEstimators.hllLowerBound(this, numStdDev);
- }
-
- @Override
- int getSegDataStart() {
- return HLL_BYTE_ARR_START;
- }
-
- abstract AuxHashMap getNewAuxHashMap();
-
- /**
- * Returns the number of slots that have the value CurMin.
- * If CurMin is 0, then it returns the number of zeros in the array.
- * @return the number of slots that have the value CurMin.
- */
- abstract int getNumAtCurMin();
-
- @Override
- int getPreInts() {
- return HLL_PREINTS;
- }
-
- //overridden by Hll4Array and DirectHll4Array
- abstract int getNibble(int slotNo);
-
- abstract int getSlotValue(int slotNo);
-
- @Override //used by HLL6 and HLL8, Overridden by HLL4
- int getUpdatableSerializationBytes() {
- return HLL_BYTE_ARR_START + getHllByteArrBytes();
- }
-
- @Override
- double getUpperBound(final int numStdDev) {
- HllUtil.checkNumStdDev(numStdDev);
- return HllEstimators.hllUpperBound(this, numStdDev);
- }
-
- @Override
- abstract PairIterator iterator();
-
- @Override
- void mergeTo(final HllSketch that) {
- throw new SketchesStateException("Possible Corruption, improper access.");
- }
-
- abstract void putAuxHashMap(AuxHashMap auxHashMap, boolean compact);
-
- abstract void putCurMin(int curMin);
-
- abstract void putHipAccum(double hipAccum);
-
- abstract void putKxQ0(double kxq0);
-
- abstract void putKxQ1(double kxq1);
-
- //overridden by Hll4Array and DirectHll4Array
- abstract void putNibble(int slotNo, int nibValue);
-
- abstract void putNumAtCurMin(int numAtCurMin);
-
- abstract void updateSlotWithKxQ(final int slotNo, final int value);
-
- abstract void updateSlotNoKxQ(final int slotNo, final int value);
-
- //Compute HLL byte array lengths, used by both heap and direct.
-
- static final int hll4ArrBytes(final int lgConfigK) {
- return 1 << (lgConfigK - 1);
- }
-
- static final int hll6ArrBytes(final int lgConfigK) {
- final int numSlots = 1 << lgConfigK;
- return ((numSlots * 3) >>> 2) + 1;
- }
-
- static final int hll8ArrBytes(final int lgConfigK) {
- return 1 << lgConfigK;
- }
-
- /**
- * Common HIP and KxQ incremental update for all heap and direct Hll.
- * This is used when incrementally updating an existing array with non-zero values.
- * @param host the origin implementation
- * @param oldValue old value
- * @param newValue new value
- */
- //Called here and by Heap and Direct 6 and 8 bit implementations
- //In C: again-two-registers.c Lines 851 to 871
- static final void hipAndKxQIncrementalUpdate(final AbstractHllArray host,
- final int oldValue, final int newValue) {
- assert newValue > oldValue;
- final double kxq0 = host.getKxQ0();
- final double kxq1 = host.getKxQ1();
- //update hipAccum BEFORE updating kxq0 and kxq1
- host.addToHipAccum((1 << host.getLgConfigK()) / (kxq0 + kxq1));
- incrementalUpdateKxQ(host, oldValue, newValue, kxq0, kxq1);
- }
-
- //separate KxQ updates
- static final void incrementalUpdateKxQ(final AbstractHllArray host,
- final int oldValue, final int newValue, double kxq0, double kxq1) {
- //update kxq0 and kxq1; subtract first, then add.
- if (oldValue < 32) { host.putKxQ0(kxq0 -= invPow2(oldValue)); }
- else { host.putKxQ1(kxq1 -= invPow2(oldValue)); }
- if (newValue < 32) { host.putKxQ0(kxq0 + invPow2(newValue)); }
- else { host.putKxQ1(kxq1 + invPow2(newValue)); }
- }
-}
diff --git a/src/main/java/org/apache/datasketches/hll/AuxHashMap.java b/src/main/java/org/apache/datasketches/hll/AuxHashMap.java
deleted file mode 100644
index f255b0fd6..000000000
--- a/src/main/java/org/apache/datasketches/hll/AuxHashMap.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-import org.apache.datasketches.common.MemorySegmentStatus;
-
-/**
- * @author Lee Rhodes
- */
-interface AuxHashMap extends MemorySegmentStatus {
-
- AuxHashMap copy();
-
- int getAuxCount();
-
- int[] getAuxIntArr();
-
- int getCompactSizeBytes();
-
- PairIterator getIterator();
-
- int getLgAuxArrInts();
-
- int getUpdatableSizeBytes();
-
- @Override
- boolean hasMemorySegment();
-
- @Override
- boolean isOffHeap();
-
- @Override
- boolean isSameResource(MemorySegment seg);
-
- /**
- * Adds the slotNo and value to the aux array.
- * @param slotNo the index from the HLL array
- * @param value the HLL value at the slotNo.
- * @throws SketchesStateException if this slotNo already exists in the aux array.
- */
- void mustAdd(int slotNo, int value);
-
- /**
- * Returns value given slotNo. If this fails an exception is thrown.
- * @param slotNo the index from the HLL array
- * @return value the HLL value at the slotNo
- * @throws SketchesStateException if valid slotNo and value is not found.
- */
- int mustFindValueFor(int slotNo);
-
- /**
- * Replaces the entry at slotNo with the given value.
- * @param slotNo the index from the HLL array
- * @param value the HLL value at the slotNo
- * @throws SketchesStateException if a valid slotNo, value is not found.
- */
- void mustReplace(int slotNo, int value);
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/BaseHllSketch.java b/src/main/java/org/apache/datasketches/hll/BaseHllSketch.java
deleted file mode 100644
index 99107220e..000000000
--- a/src/main/java/org/apache/datasketches/hll/BaseHllSketch.java
+++ /dev/null
@@ -1,416 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.apache.datasketches.hash.MurmurHash3.hash;
-import static org.apache.datasketches.hll.HllUtil.HLL_HIP_RSE_FACTOR;
-import static org.apache.datasketches.hll.HllUtil.HLL_NON_HIP_RSE_FACTOR;
-import static org.apache.datasketches.hll.HllUtil.KEY_BITS_26;
-import static org.apache.datasketches.hll.HllUtil.KEY_MASK_26;
-
-import java.lang.foreign.MemorySegment;
-import java.nio.ByteBuffer;
-
-import org.apache.datasketches.common.MemorySegmentStatus;
-import org.apache.datasketches.common.Util;
-
-/**
- * Although this class is package-private, it provides a single place to define and document
- * the common public API for both HllSketch and HllUnion.
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-abstract class BaseHllSketch implements MemorySegmentStatus {
-
- abstract void couponUpdate(int coupon);
-
- /**
- * Gets the size in bytes of the current sketch when serialized using
- * toCompactByteArray().
- * @return the size in bytes of the current sketch when serialized using
- * toCompactByteArray().
- */
- public abstract int getCompactSerializationBytes();
-
- /**
- * This is less accurate than the getEstimate() method and is automatically used
- * when the sketch has gone through union operations where the more accurate HIP estimator
- * cannot be used.
- * This is made public only for error characterization software that exists in separate
- * packages and is not intended for normal use.
- * @return the composite estimate
- */
- public abstract double getCompositeEstimate();
-
- /**
- * Returns the current mode of the sketch: LIST, SET, HLL
- * @return the current mode of the sketch: LIST, SET, HLL
- */
- abstract CurMode getCurMode();
-
- /**
- * Return the cardinality estimate
- * @return the cardinality estimate
- */
- public abstract double getEstimate();
-
- /**
- * Gets the {@link TgtHllType}
- * @return the TgtHllType enum value
- */
- public abstract TgtHllType getTgtHllType();
-
- /**
- * Gets the lgConfigK.
- * @return the lgConfigK.
- */
- public abstract int getLgConfigK();
-
- /**
- * Gets the approximate lower error bound given the specified number of Standard Deviations.
- *
- * @param numStdDev This must be an integer between 1 and 3, inclusive.
- * See Number of Standard Deviations
- * @return the lower bound.
- */
- public abstract double getLowerBound(int numStdDev);
-
- /**
- * Returns the current serialization version.
- * @return the current serialization version.
- */
- public static final int getSerializationVersion() {
- return PreambleUtil.SER_VER;
- }
-
- /**
- * Returns the current serialization version of the given MemorySegment.
- * @param seg the given MemorySegment containing a serialized HllSketch image.
- * @return the current serialization version.
- */
- public static final int getSerializationVersion(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, PreambleUtil.SER_VER_BYTE) & 0XFF;
- }
-
- /**
- * Gets the current (approximate) Relative Error (RE) asymptotic values given several
- * parameters. This is used primarily for testing.
- * @param upperBound return the RE for the Upper Bound, otherwise for the Lower Bound.
- * @param oooFlag set true if the sketch is the result of a non qualifying HllUnion operation.
- * @param lgConfigK the configured value for the sketch.
- * @param numStdDev the given number of Standard Deviations. This must be an integer between
- * 1 and 3, inclusive.
- * Number of Standard Deviations
- * @return the current (approximate) RelativeError
- */
- public static double getRelErr(final boolean upperBound, final boolean oooFlag,
- final int lgConfigK, final int numStdDev) {
- HllUtil.checkLgK(lgConfigK);
- if (lgConfigK > 12) {
- final double rseFactor = oooFlag ? HLL_NON_HIP_RSE_FACTOR : HLL_HIP_RSE_FACTOR;
- final int configK = 1 << lgConfigK;
- return (numStdDev * rseFactor) / Math.sqrt(configK);
- }
- return Math.abs(RelativeErrorTables.getRelErr(upperBound, oooFlag, lgConfigK, numStdDev));
- }
-
- /**
- * Gets the size in bytes of the current sketch when serialized using
- * toUpdatableByteArray().
- * @return the size in bytes of the current sketch when serialized using
- * toUpdatableByteArray().
- */
- public abstract int getUpdatableSerializationBytes();
-
- /**
- * Gets the approximate upper error bound given the specified number of Standard Deviations.
- *
- * @param numStdDev This must be an integer between 1 and 3, inclusive.
- * Number of Standard Deviations
- * @return the upper bound.
- */
- public abstract double getUpperBound(int numStdDev);
-
- /**
- * Returns true if empty
- * @return true if empty
- */
- public abstract boolean isEmpty();
-
- /**
- * Returns true if the backing MemorySegment of this sketch is in compact form.
- * @return true if the backing MemorySegment of this sketch is in compact form.
- */
- public abstract boolean isCompact();
-
- /**
- * This HLL family of sketches and operators is always estimating, even for very small values.
- * @return true
- */
- public boolean isEstimationMode() {
- return true;
- }
-
- /**
- * Returns true if this sketch was created using MemorySegment.
- * @return true if this sketch was created using MemorySegment.
- */
- @Override
- public abstract boolean hasMemorySegment();
-
- /**
- * Returns true if the backing MemorySegment for this sketch is off-heap.
- * @return true if the backing MemorySegment for this sketch is off-heap.
- */
- @Override
- public abstract boolean isOffHeap();
-
- /**
- * Gets the Out-of-order flag.
- * @return true if the current estimator is the non-HIP estimator, which is slightly less
- * accurate than the HIP estimator.
- */
- abstract boolean isOutOfOrder();
-
- /**
- * Returns true if the given MemorySegment refers to the same underlying resource as this sketch.
- * The capacities must be the same. If this is a region,
- * the region offset must also be the same.
- *
- * {@code
- * HllUnion union; HllSketch sk, sk2;
- * int lgK = 12;
- * sk = new HllSketch(lgK, TgtHllType.HLL_4); //can be 4, 6, or 8
- * for (int i = 0; i < (2 << lgK); i++) { sk.update(i); }
- * byte[] arr = HllSketch.toCompactByteArray();
- * //...
- * union = HllUnion.heapify(arr); //initializes the HllUnion using data from the array.
- * //OR, if used in an off-heap environment:
- * union = HllUnion.heapify(MemorySegment.ofArray(arr)); //same as above, except from MemorySegment object.
- *
- * //To recover an updatable heap sketch:
- * sk2 = HllSketch.heapify(arr);
- * //OR, if used in an off-heap environment:
- * sk2 = HllSketch.heapify(MemorySegment.ofArray(arr));
- * }
- *
- * {@code
- * HllUnion union; HllSketch sk;
- * int lgK = 12;
- * sk = new HllSketch(lgK, TgtHllType.HLL_8) //must be 8
- * for (int i = 0; i < (2 << lgK); i++) { sk.update(i); }
- * byte[] arr = sk.toUpdatableByteArray();
- * MemorySegment wseg = MemorySegment.wrap(arr);
- * //...
- * union = HllUnion.writableWrap(wseg); //no deserialization!
- * }
- * @return this sketch as an updatable byte array.
- */
- public abstract byte[] toUpdatableByteArray();
-
- /**
- * Human readable summary as a string.
- * @return Human readable summary as a string.
- */
- @Override
- public String toString() {
- return toString(true, false, false, false);
- }
-
- /**
- * Human readable summary with optional detail. Does not list empty entries.
- * @param summary if true, output the sketch summary
- * @param detail if true, output the internal data array
- * @param auxDetail if true, output the internal Aux array, if it exists.
- * @return human readable string with optional detail.
- */
- public String toString(final boolean summary, final boolean detail, final boolean auxDetail) {
- return toString(summary, detail, auxDetail, false);
- }
-
- /**
- * Human readable summary with optional detail
- * @param summary if true, output the sketch summary
- * @param detail if true, output the internal data array
- * @param auxDetail if true, output the internal Aux array, if it exists.
- * @param all if true, outputs all entries including empty ones
- * @return human readable string with optional detail.
- */
- public abstract String toString(boolean summary, boolean detail, boolean auxDetail,
- boolean all);
-
- /**
- * Present the given long as a potential unique item.
- *
- * @param datum The given long datum.
- */
- public void update(final long datum) {
- final long[] data = { datum };
- couponUpdate(coupon(hash(data, Util.DEFAULT_UPDATE_SEED)));
- }
-
- /**
- * Present the given double (or float) datum as a potential unique item.
- * The double will be converted to a long using Double.doubleToLongBits(datum),
- * which normalizes all NaN values to a single NaN representation.
- * Plus and minus zero will be normalized to plus zero.
- * The special floating-point values NaN and +/- Infinity are treated as distinct.
- *
- * @param datum The given double datum.
- */
- public void update(final double datum) {
- final double d = (datum == 0.0) ? 0.0 : datum; // canonicalize -0.0, 0.0
- final long[] data = { Double.doubleToLongBits(d) };// canonicalize all NaN & +/- infinity forms
- couponUpdate(coupon(hash(data, Util.DEFAULT_UPDATE_SEED)));
- }
-
- /**
- * Present the given String as a potential unique item.
- * The string is converted to a byte array using UTF8 encoding.
- * If the string is null or empty no update attempt is made and the method returns.
- *
- * {@code
- * int lgK = 12; //This is log-base2 of k, so k = 4096. lgK can be from 4 to 21
- * HllSketch sketch = new HllSketch(lgK); //TgtHllType.HLL_4 is the default
- * //OR
- * HllSketch sketch = new HllSketch(lgK, TgtHllType.HLL_6);
- * //OR
- * HllSketch sketch = new HllSketch(lgK, TgtHllType.HLL_8);
- * }
- *
- * {@code
- * long n = 1000000;
- * for (int i = 0; i < n; i++) {
- * sketch.update(i);
- * }
- * }
- *
- * {@code
- * double estimate = sketch.getEstimate();
- * double estUB = sketch.getUpperBound(1.0); //the upper bound at 1 standard deviation.
- * double estLB = sketch.getLowerBound(1.0); //the lower bound at 1 standard deviation.
- * //OR
- * System.out.println(sketch.toString()); //will output a summary of the sketch.
- * }
- *
- * {@code
- * ### HLL SKETCH SUMMARY:
- * Log Config K : 12
- * Hll Target : HLL_4
- * Current Mode : HLL
- * LB : 977348.7024560181
- * Estimate : 990116.6007366662
- * UB : 1003222.5095308956
- * OutOfOrder Flag: false
- * CurMin : 5
- * NumAtCurMin : 1
- * HipAccum : 990116.6007366662
- * }
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-public class HllSketch extends BaseHllSketch {
-
- /**
- * The default Log_base2 of K
- */
- public static final int DEFAULT_LG_K = 12;
-
- /**
- * The default HLL-TYPE is HLL_4
- */
- public static final TgtHllType DEFAULT_HLL_TYPE = TgtHllType.HLL_4;
-
- HllSketchImpl hllSketchImpl = null;
-
- /**
- * Constructs a new on-heap sketch with the default lgConfigK and tgtHllType.
- */
- public HllSketch() {
- this(DEFAULT_LG_K, DEFAULT_HLL_TYPE);
- }
-
- /**
- * Constructs a new on-heap sketch with the default tgtHllType.
- * @param lgConfigK The Log2 of K for the target HLL sketch. This value must be
- * between 4 and 21 inclusively.
- */
- public HllSketch(final int lgConfigK) {
- this(lgConfigK, DEFAULT_HLL_TYPE);
- }
-
- /**
- * Constructs a new on-heap sketch with the type of HLL sketch to configure.
- * @param lgConfigK The Log2 of K for the target HLL sketch. This value must be
- * between 4 and 21 inclusively.
- * @param tgtHllType the desired HLL type.
- */
- public HllSketch(final int lgConfigK, final TgtHllType tgtHllType) {
- hllSketchImpl = new CouponList(HllUtil.checkLgK(lgConfigK), tgtHllType, CurMode.LIST);
- }
-
- /**
- * Constructs a new sketch with the type of HLL sketch to configure and the given
- * MemorySegment as the destination for the sketch. This MemorySegment is usually configured
- * for off-heap MemorySegment. What remains on the java heap is a thin wrapper object that reads and
- * writes to the given MemorySegment.
- *
- * 
- * CouponList Layout
- * Long || Start Byte Adr, Big Endian Illustration
- * Adr:
- * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
- * 0 || Mode | ListCnt| Flags | LgArr | lgK | FamID | SerVer | PI=2 |
- *
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 || |------Coupon Int List Start--------|
- *
- *
- *
- * CouponHashSet Layout
- * Long || Start Byte Adr, Big Endian Illustration
- * Adr:
- * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
- * 0 || Mode | | Flags | LgArr | lgK | FamID | SerVer | PI=3 |
- *
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||-----Coupon Int Hash Set Start-----|---------Hash Set Count------------|
- *
- *
- *
- * HllArray Layout
- * Long || Start Byte Adr, Big Endian Illustration
- * Adr:
- * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
- * 0 || Mode | CurMin | Flags | LgArr | lgK | FamID | SerVer | PI=10 |
- *
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||-------------------------------HIP Accum-------------------------------|
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 ||----------------------------------KxQ0---------------------------------|
- *
- * || 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 |
- * 3 ||----------------------------------KxQ1---------------------------------|
- *
- * || 39 | 38 | 37 | 36 | 35 | 34 | 33 | 32 |
- * 4 ||-------------Aux Count-------------|----------Num At Cur Min-----------|
- *
- * || 47 | 46 | 45 | 44 | 43 | 42 | 41 | 40 |
- * 5 ||...................................|------Start of HLL_X Byte Array----|
- *
- * N ||----End of Byte Array for HLL_4----|...................................|
- * N+1 ||...................................|-----Start of Aux Array for HLL_4--|
- *
- * If in compact form exceptions array will be compacted.
- *
- * @author Lee Rhodes
- */
-final class PreambleUtil {
-
- private PreambleUtil() {}
-
- // ###### DO NOT MESS WITH THIS ...
- // Preamble byte start addresses
- // First 8 Bytes:
- static int PREAMBLE_INTS_BYTE = 0;
- static int SER_VER_BYTE = 1;
- static int FAMILY_BYTE = 2;
- static int LG_K_BYTE = 3;
- static int LG_ARR_BYTE = 4; //used for LIST, SET & HLL_4
- static int FLAGS_BYTE = 5;
- static int LIST_COUNT_BYTE = 6;
- static int HLL_CUR_MIN_BYTE = 6;
- static int MODE_BYTE = 7; //lo2bits = curMode, next 2 bits = tgtHllType
- //mode encoding of combined CurMode and TgtHllType:
- // Dec Lo4Bits TgtHllType, CurMode
- // 0 0000 HLL_4, LIST
- // 1 0001 HLL_4, SET
- // 2 0010 HLL_4, HLL
- // 4 0100 HLL_6, LIST
- // 5 0101 HLL_6, SET
- // 6 0110 HLL_6, HLL
- // 8 1000 HLL_8, LIST
- // 9 1001 HLL_8, SET
- // 10 1010 HLL_8, HLL
-
- //Coupon List
- static int LIST_INT_ARR_START = 8;
-
- //Coupon Hash Set
- static int HASH_SET_COUNT_INT = 8;
- static int HASH_SET_INT_ARR_START = 12;
-
- //HLL
- static int HIP_ACCUM_DOUBLE = 8;
- static int KXQ0_DOUBLE = 16;
- static int KXQ1_DOUBLE = 24;
- static int CUR_MIN_COUNT_INT = 32;
- static int AUX_COUNT_INT = 36;
- static int HLL_BYTE_ARR_START = 40;
-
- //Flag bit masks
- static final int RESERVED_FLAG_MASK = 1; //Set to 0 but not read.
- static final int READ_ONLY_FLAG_MASK = 2; //Set but not read. Reserved.
- static final int EMPTY_FLAG_MASK = 4;
- static final int COMPACT_FLAG_MASK = 8;
- static final int OUT_OF_ORDER_FLAG_MASK = 16;
- static final int REBUILD_CURMIN_NUM_KXQ_MASK = 32; //used only by HllUnion
-
- //Mode byte masks
- static final int CUR_MODE_MASK = 3;
- static final int TGT_HLL_TYPE_MASK = 12;
-
- //Other constants
- static final int SER_VER = 1;
- static final int FAMILY_ID = 7;
- static final int LIST_PREINTS = 2;
- static final int HASH_SET_PREINTS = 3;
- static final int HLL_PREINTS = 10;
-
- static String toString(final byte[] byteArr) {
- final MemorySegment seg = MemorySegment.ofArray(byteArr);
- return toString(seg);
- }
-
- static String toString(final MemorySegment seg) {
- //First 8 bytes
- final int preInts = seg.get(JAVA_BYTE, PREAMBLE_INTS_BYTE);
- final int serVer = seg.get(JAVA_BYTE, SER_VER_BYTE);
- final Family family = Family.idToFamily(seg.get(JAVA_BYTE, FAMILY_BYTE));
- final int lgK = seg.get(JAVA_BYTE, LG_K_BYTE);
- final int lgArr = seg.get(JAVA_BYTE, LG_ARR_BYTE);
- final int flags = seg.get(JAVA_BYTE, FLAGS_BYTE);
- //Flags
- final String flagsStr = zeroPad(Integer.toBinaryString(flags), 8) + ", " + (flags);
- final boolean compact = (flags & COMPACT_FLAG_MASK) > 0;
- final boolean oooFlag = (flags & OUT_OF_ORDER_FLAG_MASK) > 0;
- final boolean readOnly = (flags & READ_ONLY_FLAG_MASK) > 0;
- final boolean empty = (flags & EMPTY_FLAG_MASK) > 0;
- final boolean rebuildKxQ = (flags & REBUILD_CURMIN_NUM_KXQ_MASK) > 0;
-
- final int hllCurMin = seg.get(JAVA_BYTE, HLL_CUR_MIN_BYTE);
- final int listCount = hllCurMin;
- final int modeByte = seg.get(JAVA_BYTE, MODE_BYTE);
- final CurMode curMode = CurMode.fromOrdinal(modeByte & 3);
- final TgtHllType tgtHllType = TgtHllType.fromOrdinal((modeByte >>> 2) & 3);
-
- double hipAccum = 0;
- double kxq0 = 0;
- double kxq1 = 0;
- int hashSetCount = 0;
- int curMinCount = 0;
- int exceptionCount = 0;
-
- if (curMode == CurMode.SET) {
- hashSetCount = seg.get(JAVA_INT_UNALIGNED, HASH_SET_COUNT_INT);
- }
- else if (curMode == CurMode.HLL) {
- hipAccum = seg.get(JAVA_DOUBLE_UNALIGNED, HIP_ACCUM_DOUBLE);
- kxq0 = seg.get(JAVA_DOUBLE_UNALIGNED, KXQ0_DOUBLE);
- kxq1 = seg.get(JAVA_DOUBLE_UNALIGNED, KXQ1_DOUBLE);
- curMinCount = seg.get(JAVA_INT_UNALIGNED, CUR_MIN_COUNT_INT);
- exceptionCount = seg.get(JAVA_INT_UNALIGNED, AUX_COUNT_INT);
- }
-
- final StringBuilder sb = new StringBuilder();
- sb.append(LS);
- sb.append("### HLL SKETCH PREAMBLE:").append(LS);
- sb.append("Byte 0: Preamble Ints : ").append(preInts).append(LS);
- sb.append("Byte 1: SerVer : ").append(serVer).append(LS);
- sb.append("Byte 2: Family : ").append(family).append(LS);
- sb.append("Byte 3: lgK : ").append(lgK).append(LS);
- //expand byte 4: LgArr
- if (curMode == CurMode.LIST) {
- sb.append("Byte 4: LgArr: List Arr : ").append(lgArr).append(LS);
- }
- if (curMode == CurMode.SET) {
- sb.append("Byte 4: LgArr: Hash Set Arr : ").append(lgArr).append(LS);
- }
- if (curMode == CurMode.HLL) {
- sb.append("Byte 4: LgArr or Aux LgArr : ").append(lgArr).append(LS);
- }
- //expand byte 5: Flags
- sb.append("Byte 5: Flags: : ").append(flagsStr).append(LS);
- sb.append(" READ_ONLY : ").append(readOnly).append(LS);
- sb.append(" EMPTY : ").append(empty).append(LS);
- sb.append(" COMPACT : ").append(compact).append(LS);
- sb.append(" OUT_OF_ORDER : ").append(oooFlag).append(LS);
- sb.append(" REBUILD_KXQ : ").append(rebuildKxQ).append(LS);
- //expand byte 6: ListCount, CurMin
- if (curMode == CurMode.LIST) {
- sb.append("Byte 6: List Count/CurMin : ").append(listCount).append(LS);
- }
- if (curMode == CurMode.SET) {
- sb.append("Byte 6: (not used) : ").append(LS);
- }
- if (curMode == CurMode.HLL) {
- sb.append("Byte 6: Cur Min : ").append(hllCurMin).append(LS);
- }
- final String modes = curMode.toString() + ", " + tgtHllType.toString();
- sb.append("Byte 7: Mode : ").append(modes).append(LS);
- if (curMode == CurMode.SET) {
- sb.append("Hash Set Count : ").append(hashSetCount).append(LS);
- }
- if (curMode == CurMode.HLL) {
- sb.append("HIP Accum : ").append(hipAccum).append(LS);
- sb.append("KxQ0 : ").append(kxq0).append(LS);
- sb.append("KxQ1 : ").append(kxq1).append(LS);
- sb.append("Num At Cur Min : ").append(curMinCount).append(LS);
- sb.append("Aux Count : ").append(exceptionCount).append(LS);
- }
- sb.append("### END HLL SKETCH PREAMBLE").append(LS);
- return sb.toString();
- }
- //@formatter:on
-
- static int extractPreInts(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, PREAMBLE_INTS_BYTE) & 0X3F;
- }
-
- static void insertPreInts(final MemorySegment wseg, final int preInts) {
- wseg.set(JAVA_BYTE, PREAMBLE_INTS_BYTE, (byte) (preInts & 0X3F));
- }
-
- static int extractSerVer(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, SER_VER_BYTE) & 0XFF;
- }
-
- static void insertSerVer(final MemorySegment wseg) {
- wseg.set(JAVA_BYTE, SER_VER_BYTE, (byte) SER_VER);
- }
-
- static int extractFamilyId(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, FAMILY_BYTE) & 0XFF;
- }
-
- static void insertFamilyId(final MemorySegment wseg) {
- wseg.set(JAVA_BYTE, FAMILY_BYTE, (byte) FAMILY_ID);
- }
-
- static int extractLgK(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, LG_K_BYTE) & 0XFF;
- }
-
- static void insertLgK(final MemorySegment wseg, final int lgK) {
- wseg.set(JAVA_BYTE, LG_K_BYTE, (byte) lgK);
- }
-
- static int extractLgArr(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, LG_ARR_BYTE) & 0XFF;
- }
-
- static void insertLgArr(final MemorySegment wseg, final int lgArr) {
- wseg.set(JAVA_BYTE, LG_ARR_BYTE, (byte) lgArr);
- }
-
- static int extractListCount(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, LIST_COUNT_BYTE) & 0XFF;
- }
-
- static void insertListCount(final MemorySegment wseg, final int listCnt) {
- wseg.set(JAVA_BYTE, LIST_COUNT_BYTE, (byte) listCnt);
- }
-
- static int extractCurMin(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, HLL_CUR_MIN_BYTE) & 0XFF;
- }
-
- static void insertCurMin(final MemorySegment wseg, final int curMin) {
- wseg.set(JAVA_BYTE, HLL_CUR_MIN_BYTE, (byte) curMin);
- }
-
- static double extractHipAccum(final MemorySegment seg) {
- return seg.get(JAVA_DOUBLE_UNALIGNED, HIP_ACCUM_DOUBLE);
- }
-
- static void insertHipAccum(final MemorySegment wseg, final double hipAccum) {
- wseg.set(JAVA_DOUBLE_UNALIGNED, HIP_ACCUM_DOUBLE, hipAccum);
- }
-
- static double extractKxQ0(final MemorySegment seg) {
- return seg.get(JAVA_DOUBLE_UNALIGNED, KXQ0_DOUBLE);
- }
-
- static void insertKxQ0(final MemorySegment wseg, final double kxq0) {
- wseg.set(JAVA_DOUBLE_UNALIGNED, KXQ0_DOUBLE, kxq0);
- }
-
- static double extractKxQ1(final MemorySegment seg) {
- return seg.get(JAVA_DOUBLE_UNALIGNED, KXQ1_DOUBLE);
- }
-
- static void insertKxQ1(final MemorySegment wseg, final double kxq1) {
- wseg.set(JAVA_DOUBLE_UNALIGNED, KXQ1_DOUBLE, kxq1);
- }
-
- static int extractHashSetCount(final MemorySegment seg) {
- return seg.get(JAVA_INT_UNALIGNED, HASH_SET_COUNT_INT);
- }
-
- static void insertHashSetCount(final MemorySegment wseg, final int hashSetCnt) {
- wseg.set(JAVA_INT_UNALIGNED, HASH_SET_COUNT_INT, hashSetCnt);
- }
-
- static int extractNumAtCurMin(final MemorySegment seg) {
- return seg.get(JAVA_INT_UNALIGNED, CUR_MIN_COUNT_INT);
- }
-
- static void insertNumAtCurMin(final MemorySegment wseg, final int numAtCurMin) {
- wseg.set(JAVA_INT_UNALIGNED, CUR_MIN_COUNT_INT, numAtCurMin);
- }
-
- static int extractAuxCount(final MemorySegment seg) {
- return seg.get(JAVA_INT_UNALIGNED, AUX_COUNT_INT);
- }
-
- static void insertAuxCount(final MemorySegment wseg, final int auxCount) {
- wseg.set(JAVA_INT_UNALIGNED, AUX_COUNT_INT, auxCount);
- }
-
- //Mode bits
- static void insertCurMode(final MemorySegment wseg, final CurMode curMode) {
- final int curModeId = curMode.ordinal();
- int mode = wseg.get(JAVA_BYTE, MODE_BYTE) & ~CUR_MODE_MASK; //strip bits 0, 1
- mode |= (curModeId & CUR_MODE_MASK);
- wseg.set(JAVA_BYTE, MODE_BYTE, (byte) mode);
- }
-
- static CurMode extractCurMode(final MemorySegment seg) {
- final int curModeId = seg.get(JAVA_BYTE, MODE_BYTE) & CUR_MODE_MASK;
- return CurMode.fromOrdinal(curModeId);
- }
-
- static void insertTgtHllType(final MemorySegment wseg, final TgtHllType tgtHllType) {
- final int typeId = tgtHllType.ordinal();
- int mode = wseg.get(JAVA_BYTE, MODE_BYTE) & ~TGT_HLL_TYPE_MASK; //strip bits 2, 3
- mode |= (typeId << 2) & TGT_HLL_TYPE_MASK;
- wseg.set(JAVA_BYTE, MODE_BYTE, (byte) mode);
- }
-
- static TgtHllType extractTgtHllType(final MemorySegment seg) {
- final int typeId = seg.get(JAVA_BYTE, MODE_BYTE) & TGT_HLL_TYPE_MASK;
- return TgtHllType.fromOrdinal(typeId >>> 2);
- }
-
- static void insertModes(final MemorySegment wseg, final TgtHllType tgtHllType,
- final CurMode curMode) {
- final int curModeId = curMode.ordinal() & 3;
- final int typeId = (tgtHllType.ordinal() & 3) << 2;
- final int mode = typeId | curModeId;
- wseg.set(JAVA_BYTE, MODE_BYTE, (byte) mode);
- }
-
- //Flags
- static void insertEmptyFlag(final MemorySegment wseg, final boolean empty) {
- int flags = wseg.get(JAVA_BYTE, FLAGS_BYTE);
- if (empty) { flags |= EMPTY_FLAG_MASK; }
- else { flags &= ~EMPTY_FLAG_MASK; }
- wseg.set(JAVA_BYTE, FLAGS_BYTE, (byte) flags);
- }
-
- static boolean extractEmptyFlag(final MemorySegment seg) {
- final int flags = seg.get(JAVA_BYTE, FLAGS_BYTE);
- return (flags & EMPTY_FLAG_MASK) > 0;
- }
-
- static void insertCompactFlag(final MemorySegment wseg, final boolean compact) {
- int flags = wseg.get(JAVA_BYTE, FLAGS_BYTE);
- if (compact) { flags |= COMPACT_FLAG_MASK; }
- else { flags &= ~COMPACT_FLAG_MASK; }
- wseg.set(JAVA_BYTE, FLAGS_BYTE, (byte) flags);
- }
-
- static boolean extractCompactFlag(final MemorySegment seg) {
- final int flags = seg.get(JAVA_BYTE, FLAGS_BYTE);
- return (flags & COMPACT_FLAG_MASK) > 0;
- }
-
- static void insertOooFlag(final MemorySegment wseg, final boolean oooFlag) {
- int flags = wseg.get(JAVA_BYTE, FLAGS_BYTE);
- if (oooFlag) { flags |= OUT_OF_ORDER_FLAG_MASK; }
- else { flags &= ~OUT_OF_ORDER_FLAG_MASK; }
- wseg.set(JAVA_BYTE, FLAGS_BYTE, (byte) flags);
- }
-
- static boolean extractOooFlag(final MemorySegment seg) {
- final int flags = seg.get(JAVA_BYTE, FLAGS_BYTE);
- return (flags & OUT_OF_ORDER_FLAG_MASK) > 0;
- }
-
- static void insertRebuildCurMinNumKxQFlag(final MemorySegment wseg, final boolean rebuild) {
- int flags = wseg.get(JAVA_BYTE, FLAGS_BYTE);
- if (rebuild) { flags |= REBUILD_CURMIN_NUM_KXQ_MASK; }
- else { flags &= ~REBUILD_CURMIN_NUM_KXQ_MASK; }
- wseg.set(JAVA_BYTE, FLAGS_BYTE, (byte) flags);
- }
-
- static boolean extractRebuildCurMinNumKxQFlag(final MemorySegment seg) {
- final int flags = seg.get(JAVA_BYTE, FLAGS_BYTE);
- return (flags & REBUILD_CURMIN_NUM_KXQ_MASK) > 0;
- }
-
- static void insertFlags(final MemorySegment wseg, final int flags) {
- wseg.set(JAVA_BYTE, FLAGS_BYTE, (byte) flags);
- }
-
- static int extractFlags(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, FLAGS_BYTE) & 0XFF;
- }
-
- //Other
- static int extractInt(final MemorySegment seg, final long byteOffset) {
- return seg.get(JAVA_INT_UNALIGNED, byteOffset);
- }
-
- static void insertInt(final MemorySegment wseg, final long byteOffset, final int value) {
- wseg.set(JAVA_INT_UNALIGNED, byteOffset, value);
- }
-
- static int computeLgArr(final MemorySegment seg, final int count, final int lgConfigK) {
- //value is missing, recompute
- final CurMode curMode = extractCurMode(seg);
- if (curMode == CurMode.LIST) { return HllUtil.LG_INIT_LIST_SIZE; }
- int ceilPwr2 = ceilingPowerOf2(count);
- if ((RESIZE_DENOM * count) > (RESIZE_NUMER * ceilPwr2)) { ceilPwr2 <<= 1; }
- if (curMode == CurMode.SET) {
- return Math.max(LG_INIT_SET_SIZE, exactLog2OfLong(ceilPwr2));
- }
- //only used for HLL4
- return Math.max(LG_AUX_ARR_INTS[lgConfigK], exactLog2OfLong(ceilPwr2));
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/RelativeErrorTables.java b/src/main/java/org/apache/datasketches/hll/RelativeErrorTables.java
deleted file mode 100644
index 5a167f996..000000000
--- a/src/main/java/org/apache/datasketches/hll/RelativeErrorTables.java
+++ /dev/null
@@ -1,119 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-/**
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class RelativeErrorTables {
-
- /**
- * Return Relative Error for UB or LB for HIP or Non-HIP as a function of numStdDev.
- * @param upperBound true if for upper bound
- * @param oooFlag true if for Non-HIP
- * @param lgK must be between 4 and 12 inclusive
- * @param stdDev must be between 1 and 3 inclusive
- * @return Relative Error for UB or LB for HIP or Non-HIP as a function of numStdDev.
- */
- static double getRelErr(final boolean upperBound, final boolean oooFlag,
- final int lgK, final int stdDev) {
- final int idx = ((lgK - 4) * 3) + (stdDev - 1);
- final int sw = (oooFlag ? 2 : 0) | (upperBound ? 1 : 0);
- double f = 0;
- switch (sw) {
- case 0 : { //HIP, LB
- f = HIP_LB[idx];
- break;
- }
- case 1 : { //HIP, UB
- f = HIP_UB[idx];
- break;
- }
- case 2 : { //NON_HIP, LB
- f = NON_HIP_LB[idx];
- break;
- }
- case 3 : { //NON_HIP, UB
- f = NON_HIP_UB[idx];
- break;
- }
- default: f = 0;//can't happen
- }
- return f;
- }
-
- //case 0
- private static double[] HIP_LB = //sd 1, 2, 3
- { //Q(.84134), Q(.97725), Q(.99865) respectively
- 0.207316195, 0.502865572, 0.882303765, //4
- 0.146981579, 0.335426881, 0.557052, //5
- 0.104026721, 0.227683872, 0.365888317, //6
- 0.073614601, 0.156781585, 0.245740374, //7
- 0.05205248, 0.108783763, 0.168030442, //8
- 0.036770852, 0.075727545, 0.11593785, //9
- 0.025990219, 0.053145536, 0.080772263, //10
- 0.018373987, 0.037266176, 0.056271814, //11
- 0.012936253, 0.02613829, 0.039387631, //12
- };
-
- //case 1
- private static double[] HIP_UB = //sd 1, 2, 3
- { //Q(.15866), Q(.02275), Q(.00135) respectively
- -0.207805347, -0.355574279, -0.475535095, //4
- -0.146988328, -0.262390832, -0.360864026, //5
- -0.103877775, -0.191503663, -0.269311582, //6
- -0.073452978, -0.138513438, -0.198487447, //7
- -0.051982806, -0.099703123, -0.144128618, //8
- -0.036768609, -0.07138158, -0.104430324, //9
- -0.025991325, -0.050854296, -0.0748143, //10
- -0.01834533, -0.036121138, -0.05327616, //11
- -0.012920332, -0.025572893, -0.037896952, //12
- };
-
- //case 2
- private static double[] NON_HIP_LB = //sd 1, 2, 3
- { //Q(.84134), Q(.97725), Q(.99865) respectively
- 0.254409839, 0.682266712, 1.304022158, //4
- 0.181817353, 0.443389054, 0.778776219, //5
- 0.129432281, 0.295782195, 0.49252279, //6
- 0.091640655, 0.201175925, 0.323664385, //7
- 0.064858051, 0.138523393, 0.218805328, //8
- 0.045851855, 0.095925072, 0.148635751, //9
- 0.032454144, 0.067009668, 0.102660669, //10
- 0.022921382, 0.046868565, 0.071307398, //11
- 0.016155679, 0.032825719, 0.049677541 //12
- };
-
- //case 3
- private static double[] NON_HIP_UB = //sd 1, 2, 3
- { //Q(.15866), Q(.02275), Q(.00135) respectively
- -0.256980172, -0.411905944, -0.52651057, //4
- -0.182332109, -0.310275547, -0.412660505, //5
- -0.129314228, -0.230142294, -0.315636197, //6
- -0.091584836, -0.16834013, -0.236346847, //7
- -0.06487411, -0.122045231, -0.174112107, //8
- -0.04591465, -0.08784505, -0.126917615, //9
- -0.032433119, -0.062897613, -0.091862929, //10
- -0.022960633, -0.044875401, -0.065736049, //11
- -0.016186662, -0.031827816, -0.046973459 //12
- };
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/TgtHllType.java b/src/main/java/org/apache/datasketches/hll/TgtHllType.java
deleted file mode 100644
index 78aaafd1b..000000000
--- a/src/main/java/org/apache/datasketches/hll/TgtHllType.java
+++ /dev/null
@@ -1,77 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-/**
- * Specifies the target type of HLL sketch to be created. It is a target in that the actual
- * allocation of the HLL array is deferred until sufficient number of items have been received by
- * the warm-up phases.
- *
- *
- *
- * @author Lee Rhodes
- */
-public enum TgtHllType {
- /**
- * An HLL sketch with a bin size of 4 bits
- */
- HLL_4,
- /**
- * An HLL sketch with a bin size of 6 bits
- */
- HLL_6,
- /**
- * An HLL Sketch with a bin size of 8 bits
- */
- HLL_8;
-
- private static final TgtHllType values[] = values();
-
- /**
- * Convert the typeId to the enum type
- * @param typeId the given typeId
- * @return the enum type
- */
- public static final TgtHllType fromOrdinal(final int typeId) {
- return values[typeId];
- }
-}
diff --git a/src/main/java/org/apache/datasketches/hll/ToByteArrayImpl.java b/src/main/java/org/apache/datasketches/hll/ToByteArrayImpl.java
deleted file mode 100644
index 4b0c6a96e..000000000
--- a/src/main/java/org/apache/datasketches/hll/ToByteArrayImpl.java
+++ /dev/null
@@ -1,237 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hll;
-
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED;
-import static org.apache.datasketches.hll.AbstractCoupons.find;
-import static org.apache.datasketches.hll.HllUtil.LG_AUX_ARR_INTS;
-import static org.apache.datasketches.hll.PreambleUtil.AUX_COUNT_INT;
-import static org.apache.datasketches.hll.PreambleUtil.HLL_BYTE_ARR_START;
-import static org.apache.datasketches.hll.PreambleUtil.insertAuxCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertCompactFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.insertCurMode;
-import static org.apache.datasketches.hll.PreambleUtil.insertEmptyFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertFamilyId;
-import static org.apache.datasketches.hll.PreambleUtil.insertHashSetCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertHipAccum;
-import static org.apache.datasketches.hll.PreambleUtil.insertInt;
-import static org.apache.datasketches.hll.PreambleUtil.insertKxQ0;
-import static org.apache.datasketches.hll.PreambleUtil.insertKxQ1;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgArr;
-import static org.apache.datasketches.hll.PreambleUtil.insertLgK;
-import static org.apache.datasketches.hll.PreambleUtil.insertListCount;
-import static org.apache.datasketches.hll.PreambleUtil.insertNumAtCurMin;
-import static org.apache.datasketches.hll.PreambleUtil.insertOooFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertPreInts;
-import static org.apache.datasketches.hll.PreambleUtil.insertRebuildCurMinNumKxQFlag;
-import static org.apache.datasketches.hll.PreambleUtil.insertSerVer;
-import static org.apache.datasketches.hll.PreambleUtil.insertTgtHllType;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.SketchesStateException;
-
-/**
- * @author Lee Rhodes
- */
-final class ToByteArrayImpl {
-
- // To byte array used by the heap HLL modes.
- static byte[] toHllByteArray(final AbstractHllArray impl, final boolean compact) {
- int auxBytes = 0;
- if (impl.tgtHllType == TgtHllType.HLL_4) {
- final AuxHashMap auxHashMap = impl.getAuxHashMap();
- if (auxHashMap != null) {
- auxBytes = (compact)
- ? auxHashMap.getCompactSizeBytes()
- : auxHashMap.getUpdatableSizeBytes();
- } else {
- auxBytes = (compact) ? 0 : 4 << LG_AUX_ARR_INTS[impl.lgConfigK];
- }
- }
- final int totBytes = HLL_BYTE_ARR_START + impl.getHllByteArrBytes() + auxBytes;
- final byte[] byteArr = new byte[totBytes];
- final MemorySegment wseg = MemorySegment.ofArray(byteArr);
- insertHll(impl, wseg, compact);
- return byteArr;
- }
-
- private static void insertHll(final AbstractHllArray impl, final MemorySegment wseg,
- final boolean compact) {
- insertCommonHll(impl, wseg, compact);
- final byte[] hllByteArr = ((HllArray)impl).hllByteArr;
- MemorySegment.copy(hllByteArr, 0, wseg, JAVA_BYTE, HLL_BYTE_ARR_START, hllByteArr.length);
- if (impl.getAuxHashMap() != null) {
- insertAux(impl, wseg, compact);
- } else {
- wseg.set(JAVA_INT_UNALIGNED, AUX_COUNT_INT, 0);
- }
- }
-
- private static void insertCommonHll(final AbstractHllArray srcImpl,
- final MemorySegment tgtWseg, final boolean compact) {
- insertPreInts(tgtWseg, srcImpl.getPreInts());
- insertSerVer(tgtWseg);
- insertFamilyId(tgtWseg);
- insertLgK(tgtWseg, srcImpl.getLgConfigK());
- insertEmptyFlag(tgtWseg, srcImpl.isEmpty());
- insertCompactFlag(tgtWseg, compact);
- insertOooFlag(tgtWseg, srcImpl.isOutOfOrder());
- insertCurMin(tgtWseg, srcImpl.getCurMin());
- insertCurMode(tgtWseg, srcImpl.getCurMode());
- insertTgtHllType(tgtWseg, srcImpl.getTgtHllType());
- insertHipAccum(tgtWseg, srcImpl.getHipAccum());
- insertKxQ0(tgtWseg, srcImpl.getKxQ0());
- insertKxQ1(tgtWseg, srcImpl.getKxQ1());
- insertNumAtCurMin(tgtWseg, srcImpl.getNumAtCurMin());
- insertRebuildCurMinNumKxQFlag(tgtWseg, srcImpl.isRebuildCurMinNumKxQFlag());
- }
-
- private static void insertAux(final AbstractHllArray srcImpl, final MemorySegment tgtWseg,
- final boolean tgtCompact) {
- final AuxHashMap auxHashMap = srcImpl.getAuxHashMap();
- final int auxCount = auxHashMap.getAuxCount();
- insertAuxCount(tgtWseg, auxCount);
- insertLgArr(tgtWseg, auxHashMap.getLgAuxArrInts()); //only used for direct HLL
- final long auxStart = srcImpl.auxStart;
- if (tgtCompact) {
- final PairIterator itr = auxHashMap.getIterator();
- int cnt = 0;
- while (itr.nextValid()) { //works whether src has compact MemorySegment or not
- insertInt(tgtWseg, auxStart + (cnt++ << 2), itr.getPair());
- }
- assert cnt == auxCount;
- } else { //updatable
- final int auxInts = 1 << auxHashMap.getLgAuxArrInts();
- final int[] auxArr = auxHashMap.getAuxIntArr();
- MemorySegment.copy(auxArr, 0, tgtWseg, JAVA_INT_UNALIGNED, auxStart, auxInts);
- }
- }
-
- //To byte array for coupons
- static byte[] toCouponByteArray(final AbstractCoupons impl, final boolean dstCompact) {
- final int srcCouponCount = impl.getCouponCount();
- final int srcLgCouponArrInts = impl.getLgCouponArrInts();
- final int srcCouponArrInts = 1 << srcLgCouponArrInts;
- final byte[] byteArrOut;
- final boolean list = impl.getCurMode() == CurMode.LIST;
- //prepare switch
- final int sw = (impl.hasMemorySegment() ? 0 : 4) | (impl.isCompact() ? 0 : 2) | (dstCompact ? 0 : 1);
- switch (sw) {
- case 0: { //Src MemorySegment, Src Compact, Dst Compact
- final MemorySegment srcSeg = impl.getMemorySegment();
- final int bytesOut = impl.getSegDataStart() + (srcCouponCount << 2);
- byteArrOut = new byte[bytesOut];
- MemorySegment.copy(srcSeg, JAVA_BYTE, 0, byteArrOut, 0, bytesOut);
- break;
- }
- case 1: { //Src MemorySegment, Src Compact, Dst Updatable
- final int dataStart = impl.getSegDataStart();
- final int bytesOut = dataStart + (srcCouponArrInts << 2);
- byteArrOut = new byte[bytesOut];
- final MemorySegment segOut = MemorySegment.ofArray(byteArrOut);
- copyCommonListAndSet(impl, segOut);
- insertCompactFlag(segOut, dstCompact);
-
- final int[] tgtCouponIntArr = new int[srcCouponArrInts];
- final PairIterator itr = impl.iterator();
- while (itr.nextValid()) {
- final int pair = itr.getPair();
- final int idx = find(tgtCouponIntArr, srcLgCouponArrInts, pair);
- if (idx < 0) { //found EMPTY
- tgtCouponIntArr[~idx] = pair; //insert
- continue;
- }
- throw new SketchesStateException("Error: found duplicate.");
- }
- MemorySegment.copy(tgtCouponIntArr, 0, segOut, JAVA_INT_UNALIGNED, dataStart, srcCouponArrInts);
-
- if (list) {
- insertListCount(segOut, srcCouponCount);
- } else {
- insertHashSetCount(segOut, srcCouponCount);
- }
- break;
- }
-
- case 6: //Src Heap, Src Updatable, Dst Compact
- case 2: { //Src MemorySegment, Src Updatable, Dst Compact
- final int dataStart = impl.getSegDataStart();
- final int bytesOut = dataStart + (srcCouponCount << 2);
- byteArrOut = new byte[bytesOut];
- final MemorySegment segOut = MemorySegment.ofArray(byteArrOut);
- copyCommonListAndSet(impl, segOut);
- insertCompactFlag(segOut, dstCompact);
-
- final PairIterator itr = impl.iterator();
- int cnt = 0;
- while (itr.nextValid()) {
- insertInt(segOut, dataStart + (cnt++ << 2), itr.getPair());
- }
- if (list) {
- insertListCount(segOut, srcCouponCount);
- } else {
- insertHashSetCount(segOut, srcCouponCount);
- }
- break;
- }
- case 3: { //Src MemorySegment, Src Updatable, Dst Updatable
- final MemorySegment srcSeg = impl.getMemorySegment();
- final int bytesOut = impl.getSegDataStart() + (srcCouponArrInts << 2);
- byteArrOut = new byte[bytesOut];
- MemorySegment.copy(srcSeg, JAVA_BYTE, 0, byteArrOut, 0, bytesOut);
- break;
- }
- case 7: { //Src Heap, Src Updatable, Dst Updatable
- final int dataStart = impl.getSegDataStart();
- final int bytesOut = dataStart + (srcCouponArrInts << 2);
- byteArrOut = new byte[bytesOut];
- final MemorySegment segOut = MemorySegment.ofArray(byteArrOut);
- copyCommonListAndSet(impl, segOut);
-
- MemorySegment.copy(impl.getCouponIntArr(), 0, segOut, JAVA_INT_UNALIGNED, dataStart, srcCouponArrInts);
- if (list) {
- insertListCount(segOut, srcCouponCount);
- } else {
- insertHashSetCount(segOut, srcCouponCount);
- }
- break;
- }
- default: throw new SketchesStateException("Corruption, should not happen: " + sw);
- }
- return byteArrOut;
- }
-
- private static void copyCommonListAndSet(final AbstractCoupons impl,
- final MemorySegment wseg) {
- insertPreInts(wseg, impl.getPreInts());
- insertSerVer(wseg);
- insertFamilyId(wseg);
- insertLgK(wseg, impl.getLgConfigK());
- insertLgArr(wseg, impl.getLgCouponArrInts());
- insertEmptyFlag(wseg, impl.isEmpty());
- insertOooFlag(wseg, impl.isOutOfOrder());
- insertCurMode(wseg, impl.getCurMode());
- insertTgtHllType(wseg, impl.getTgtHllType());
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hll/package-info.java b/src/main/java/org/apache/datasketches/hll/package-info.java
deleted file mode 100644
index ad2f22fa9..000000000
--- a/src/main/java/org/apache/datasketches/hll/package-info.java
+++ /dev/null
@@ -1,159 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * The DataSketches™ HllSketch family package
- * {@link org.apache.datasketches.hll.HllSketch HllSketch} and {@link org.apache.datasketches.hll.HllUnion HllUnion}
- * are the public facing classes of this high performance implementation of Phillipe Flajolet's
- * HyperLogLog algorithm[1] but with significantly improved error behavior and important features that can be
- * essential for large production systems that must handle massive data.
- *
- * Key Features of the DataSketches™ HllSketch and its companion HllUnion
- *
- * Advanced Estimation Algorithms for Optimum Accuracy
- *
- * Zero error at low cardinalities
- * The HllSketch leverages highly compact arrays and hash tables to keep exact counts until the transition to
- * dense mode is required for space reasons. The result is perfect accuracy for very low cardinalities.
- *
- * HIP / Martingale Estimator
- * When obtaining a cardinality estimate, the sketch automatically determines if it was the result of the capture of
- * a single stream, or if was the result of certain qualifying union operations. If this is the case the sketch will
- * take advantage of Edith Cohen's Historical Inverse Probability (HIP) estimation algorithm[2], which was
- * also independently developed by Daniel Ting as the Martingale estimation algorithm[3].
- * This will result in a 20% improvement in accuracy over the standard Flajolet estimator.
- * If it is not a single stream or if the specific union operation did not qualify,
- * the estimator will default to the Composite Estimator.
- *
- * Composite Estimator
- * This advanced estimator is a blend of several algorithms including new algorithms developed by Kevin Lang for his
- * Compressed Probabilistic Counting (CPC) sketch[4]. These algorithms provide near optimal estimation accuracy
- * for cases that don't qualify for HIP / Martingale estimation.
- *
- * 
[6]Three HllSketch Types
- * This HLL implementation offers three different types of HllSketch, each with different
- * trade-offs with accuracy, space and performance. These types are selected with the
- * {@link org.apache.datasketches.hll.TgtHllType TgtHllType} parameter.
- *
- * HLL 8
- * This uses an 8-bit byte per HLL bucket. It is generally the
- * fastest in terms of update time but has the largest storage footprint of about K bytes.
- *
- * HLL 6
- * This uses a 6-bit field per HLL bucket. It is the generally the next fastest
- * in terms of update time with a storage footprint of about 3/4 * K bytes.
- *
- * HLL 4
- * This uses a 4-bit field per HLL bucket and for large counts may require
- * the use of a small internal auxiliary array for storing statistical exceptions, which are rare.
- * For the values of lgConfigK > 13 (K = 8192),
- * this additional array adds about 3% to the overall storage. It is generally the slowest in
- * terms of update time, but has the smallest storage footprint of about K/2 * 1.03 bytes.
- *
- * Off-Heap Operation
- * This HllSketch also offers the capability of operating off-heap. Given a MemorySegment[5] object
- * created by the user, the sketch will perform all of its updates and internal phase transitions
- * in that object, which can actually reside either on-heap or off-heap based on how it was
- * configured. In large systems that must update and union many millions of sketches, having the
- * sketch operate off-heap avoids the serialization and deserialization costs of moving sketches from heap to
- * off-heap and back, and reduces the need for garbage collection.
- *
- * Merging sketches with different configured lgConfigK
- * This enables a user to union an HllSketch that was configured with, say, lgConfigK = 12
- * with another loaded HllSketch that was configured with, say, lgConfigK = 14.
- *
- * Multi-language, multi-platform.
- * The binary structures for our sketch serializations are language and platform independent.
- * This means it is possible to generate an HllSketch on a C++ Windows platform and it can be used on a
- * Java or Python Unix platform.
- *
- *
- */
- private byte[] stateArr_;
-
- private CouponTraverseMap(final int keySizeBytes, final int maxCouponsPerKey) {
- super(keySizeBytes);
- maxCouponsPerKey_ = maxCouponsPerKey;
- }
-
- static CouponTraverseMap getInstance(final int keySizeBytes, final int maxCouponsPerKey) {
- final CouponTraverseMap map = new CouponTraverseMap(keySizeBytes, maxCouponsPerKey);
- map.tableEntries_ = COUPON_MAP_MIN_NUM_ENTRIES;
- map.capacityEntries_ = (int)(map.tableEntries_ * COUPON_MAP_GROW_TRIGGER_FACTOR);
- map.numActiveKeys_ = 0;
- map.numDeletedKeys_ = 0;
- map.entrySizeBytes_ = updateEntrySizeBytes(map.tableEntries_, keySizeBytes, maxCouponsPerKey);
-
- map.keysArr_ = new byte[COUPON_MAP_MIN_NUM_ENTRIES * keySizeBytes];
- map.couponsArr_ = new short[COUPON_MAP_MIN_NUM_ENTRIES * maxCouponsPerKey];
- map.stateArr_ = new byte[COUPON_MAP_MIN_NUM_ENTRIES_ARR_SIZE];
- return map;
- }
-
- @Override //used for test
- double update(final byte[] key, final short coupon) {
- final int entryIndex = findOrInsertKey(key);
- return update(entryIndex, coupon);
- }
-
- @Override
- double update(final int entryIndex, final short value) {
- final int offset = entryIndex * maxCouponsPerKey_;
- boolean wasFound = false;
- for (int i = 0; i < maxCouponsPerKey_; i++) {
- if (couponsArr_[offset + i] == 0) {
- if (wasFound) { return i; }
- couponsArr_[offset + i] = value;
- return i + 1;
- }
- if (couponsArr_[offset + i] == value) {
- wasFound = true;
- }
- }
- if (wasFound) { return maxCouponsPerKey_; }
- return -maxCouponsPerKey_; //signal to promote
- }
-
- @Override
- double getEstimate(final byte[] key) {
- final int entryIndex = findKey(key);
- if (entryIndex < 0) { return 0; }
- return getCouponCount(entryIndex);
- }
-
- @Override
- double getUpperBound(final byte[] key) {
- return getEstimate(key) * (1 + RSE);
- }
-
- @Override
- double getLowerBound(final byte[] key) {
- return getEstimate(key) * (1 - RSE);
- }
-
- /**
- * Returns entryIndex if the given key is found. If not found, returns one's complement entryIndex
- * of an empty slot for insertion, which may be over a deleted key.
- * @param key the given key
- * @return the entryIndex
- */
- @Override
- int findKey(final byte[] key) {
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries_);
- int firstDeletedIndex = -1;
- final int loopIndex = entryIndex;
- do {
- if (isBitClear(stateArr_, entryIndex)) {
- return firstDeletedIndex == -1 ? ~entryIndex : ~firstDeletedIndex; // found empty or deleted
- }
- if (couponsArr_[entryIndex * maxCouponsPerKey_] == 0) { //found deleted
- if (firstDeletedIndex == -1) { firstDeletedIndex = entryIndex; }
- } else if (Map.arraysEqual(keysArr_, entryIndex * keySizeBytes_, key, 0, keySizeBytes_)) {
- return entryIndex; // found key
- }
- entryIndex = (entryIndex + getStride(hash[1], tableEntries_)) % tableEntries_;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- @Override
- int findOrInsertKey(final byte[] key) {
- int entryIndex = findKey(key);
- if (entryIndex < 0) {
- entryIndex = ~entryIndex;
- if (isBitSet(stateArr_, entryIndex)) { // reusing slot from a deleted key
- clearCouponArea(entryIndex);
- numDeletedKeys_--;
- }
- if ((numActiveKeys_ + numDeletedKeys_ + 1) > capacityEntries_) {
- resize();
- entryIndex = ~findKey(key);
- assert entryIndex >= 0;
- }
- System.arraycopy(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_);
- setBit(stateArr_, entryIndex);
- numActiveKeys_++;
- }
- return entryIndex;
- }
-
- @Override
- void deleteKey(final int entryIndex) {
- couponsArr_[entryIndex * maxCouponsPerKey_] = 0;
- numActiveKeys_--;
- numDeletedKeys_++;
- if ((numActiveKeys_ > COUPON_MAP_MIN_NUM_ENTRIES)
- && (numActiveKeys_ < (tableEntries_ * COUPON_MAP_SHRINK_TRIGGER_FACTOR))) {
- resize();
- }
- }
-
- private int getCouponCount(final int entryIndex) {
- final int offset = entryIndex * maxCouponsPerKey_;
- for (int i = 0; i < maxCouponsPerKey_; i++) {
- if (couponsArr_[offset + i] == 0) {
- return i;
- }
- }
- return maxCouponsPerKey_;
- }
-
- @Override
- CouponsIterator getCouponsIterator(final int entryIndex) {
- return new CouponsIterator(couponsArr_, entryIndex * maxCouponsPerKey_, maxCouponsPerKey_);
- }
-
- @Override
- double getEntrySizeBytes() {
- return entrySizeBytes_;
- }
-
- @Override
- int getTableEntries() {
- return tableEntries_;
- }
-
- @Override
- int getCapacityEntries() {
- return capacityEntries_;
- }
-
- @Override
- int getCurrentCountEntries() {
- return numActiveKeys_ + numDeletedKeys_;
- }
-
- @Override
- long getTotalUsageBytes() {
- return keysArr_.length
- + ((long)couponsArr_.length * Short.BYTES)
- + stateArr_.length + (4L * Integer.BYTES);
- }
-
- @Override
- int getActiveEntries() {
- return numActiveKeys_;
- }
-
- @Override
- int getDeletedEntries() {
- return numDeletedKeys_;
- }
-
- @Override
- int getMaxCouponsPerEntry() {
- return maxCouponsPerKey_;
- }
-
- @Override
- int getCapacityCouponsPerEntry() {
- return maxCouponsPerKey_;
- }
-
- private void resize() { //can grow or shrink
- final byte[] oldKeysArr = keysArr_;
- final short[] oldCouponsArr = couponsArr_;
- final byte[] oldStateArr = stateArr_;
- final int oldSizeKeys = tableEntries_;
- tableEntries_ = Math.max(
- nextPrime((int) (numActiveKeys_ / COUPON_MAP_TARGET_FILL_FACTOR)),
- COUPON_MAP_MIN_NUM_ENTRIES
- );
- capacityEntries_ = (int)(tableEntries_ * COUPON_MAP_GROW_TRIGGER_FACTOR);
- numActiveKeys_ = 0;
- numDeletedKeys_ = 0;
- entrySizeBytes_ = updateEntrySizeBytes(tableEntries_, keySizeBytes_, maxCouponsPerKey_);
-
- keysArr_ = new byte[tableEntries_ * keySizeBytes_];
- couponsArr_ = new short[tableEntries_ * maxCouponsPerKey_];
- stateArr_ = new byte[(int) Math.ceil(tableEntries_ / 8.0)];
-
- //move data
- for (int i = 0; i < oldSizeKeys; i++) {
- if (isBitSet(oldStateArr, i) && (oldCouponsArr[i * maxCouponsPerKey_] != 0)) {
- final byte[] key =
- Arrays.copyOfRange(oldKeysArr, i * keySizeBytes_, (i * keySizeBytes_) + keySizeBytes_);
- final int index = insertKey(key);
- System.arraycopy(oldCouponsArr, i * maxCouponsPerKey_, couponsArr_,
- index * maxCouponsPerKey_, maxCouponsPerKey_);
- }
- }
- }
-
- // for internal use during resize, so no resize check here
- private int insertKey(final byte[] key) {
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries_);
- final int loopIndex = entryIndex;
- do {
- if (isBitClear(stateArr_, entryIndex)) {
- System.arraycopy(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_);
- setBit(stateArr_, entryIndex);
- numActiveKeys_++;
- return entryIndex;
- }
- entryIndex = (entryIndex + getStride(hash[1], tableEntries_)) % tableEntries_;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- private void clearCouponArea(final int entryIndex) {
- final int couponAreaIndex = entryIndex * maxCouponsPerKey_;
- for (int i = 0; i < maxCouponsPerKey_; i++) {
- couponsArr_[couponAreaIndex + i] = 0;
- }
- }
-
- private static final double updateEntrySizeBytes(final int tableEntries, final int keySizeBytes,
- final int maxCouponsPerKey) {
- final double byteFraction = Math.ceil(tableEntries / 8.0) / tableEntries;
- return keySizeBytes + ((double) maxCouponsPerKey * Short.BYTES) + byteFraction;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hllmap/CouponsIterator.java b/src/main/java/org/apache/datasketches/hllmap/CouponsIterator.java
deleted file mode 100644
index 81f2d75c7..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/CouponsIterator.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hllmap;
-
-/**
- * Common iterator class for maps that need one.
- *
- * @author Alex Saydakov
- */
-class CouponsIterator {
-
- private final int offset_;
- private final int maxEntries_;
- private final short[] couponsArr_;
- private int index_;
-
- CouponsIterator(final short[] couponsArr, final int offset, final int maxEntries) {
- offset_ = offset;
- maxEntries_ = maxEntries;
- couponsArr_ = couponsArr;
- index_ = -1;
- }
-
- /**
- * next() must be called before the first getValue(). This skips over zero values.
- * @return the next coupon in the array.
- */
- boolean next() {
- index_++;
- while (index_ < maxEntries_) {
- if (couponsArr_[offset_ + index_] != 0) { return true; }
- index_++;
- }
- return false;
- }
-
- /**
- * Returns the value at the current index.
- * @return the value at the current index.
- */
- short getValue() {
- return couponsArr_[offset_ + index_];
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hllmap/HllMap.java b/src/main/java/org/apache/datasketches/hllmap/HllMap.java
deleted file mode 100644
index 36d9b7c1e..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/HllMap.java
+++ /dev/null
@@ -1,344 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hllmap;
-
-import static java.lang.Math.log;
-import static java.lang.Math.sqrt;
-import static org.apache.datasketches.common.Util.invPow2;
-
-import java.util.Arrays;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SuppressFBWarnings;
-import org.apache.datasketches.hash.MurmurHash3;
-
-/**
- * Implements a key-value map where the value is a compact HLL sketch of size k.
- * The HLL bins are compacted into 10 bins per long so that a 1024 bins are compacted into
- * 824 bytes, which is a 20% reduction in space. Higher density compressions are possible
- * (up to 50%), but the required code is much more complex and considerably slower.
- *
- * true if the two specified sub-arrays of bytes are equal to one another.
- * Two arrays are considered equal if all corresponding pairs of elements in the two arrays are
- * equal. In other words, two arrays are equal if and only if they contain the same elements
- * in the same order.
- *
- * @param a one sub-array to be tested for equality
- * @param offsetA the offset in bytes of the start of sub-array a.
- * @param b the other sub-array to be tested for equality
- * @param offsetB the offset in bytes of the start of sub-array b.
- * @param length the length in bytes of the two sub-arrays.
- * @return true if the two sub-arrays are equal
- */
- static final boolean arraysEqual(final byte[] a, final int offsetA, final byte[] b,
- final int offsetB, final int length) {
- for (int i = 0; i < length; i++) {
- if (a[i + offsetA] != b[i + offsetB]) {
- return false;
- }
- }
- return true;
- }
-
- /**
- * Returns the HLL array index and value as a 16-bit coupon given the identifier to be hashed
- * and k.
- * @param identifier the given identifier
- * @return the HLL array index and value
- */
- static final int coupon16(final byte[] identifier) {
- final long[] hash = MurmurHash3.hash(identifier, SEED);
- final int hllIdx = (int) (((hash[0] >>> 1) % 1024) & TEN_BIT_MASK); //hash[0] for 10-bit address
- final int lz = Long.numberOfLeadingZeros(hash[1]);
- final int value = (lz > 62 ? 62 : lz) + 1;
- return (value << 10) | hllIdx;
- }
-
- static final int coupon16Value(final int coupon) {
- return (coupon >>> 10) & SIX_BIT_MASK;
- }
-
- static final int getIndex(final long hash, final int tableEntries) {
- return (int) ((hash >>> 1) % tableEntries);
- }
-
- static final int getStride(final long hash, final int tableEntries) {
- return (int) (((hash >>> 1) % (tableEntries - 2L)) + 1L);
- }
-
- static boolean isBitSet(final byte[] byteArr, final int bitIndex) {
- final int byteIndex = bitIndex / 8;
- final int mask = 1 << (bitIndex % 8);
- return (byteArr[byteIndex] & mask) > 0;
- }
-
- static boolean isBitClear(final byte[] byteArr, final int bitIndex) {
- final int byteIndex = bitIndex / 8;
- final int mask = 1 << (bitIndex % 8);
- return (byteArr[byteIndex] & mask) == 0;
- }
-
- static void clearBit(final byte[] byteArr, final int index) {
- final int byteIndex = index / 8;
- final int mask = 1 << (index % 8);
- byteArr[byteIndex] &= ~mask;
- }
-
- static void setBit(final byte[] bits, final int index) {
- final int byteIndex = index / 8;
- final int mask = 1 << (index % 8);
- bits[byteIndex] |= mask;
- }
-
- /**
- * Returns the next prime number that is greater than the given target. There will be
- * no prime numbers less than the returned prime number that are greater than the given target.
- * @param target the starting value to begin the search for the next prime
- * @return the next prime number that is greater than the given target.
- */
- static int nextPrime(final int target) {
- return BigInteger.valueOf(target).nextProbablePrime().intValueExact();
- }
-
- static String fmtLong(final long value) {
- return String.format("%,d", value);
- }
-
- static String fmtDouble(final double value) {
- return String.format("%,.3f", value);
- }
-
- @Override
- public String toString() {
- final String mcpe = Map.fmtLong(getMaxCouponsPerEntry());
- final String ccpe = Map.fmtLong(getCapacityCouponsPerEntry());
- final String te = Map.fmtLong(getTableEntries());
- final String ce = Map.fmtLong(getCapacityEntries());
- final String cce = Map.fmtLong(getCurrentCountEntries());
- final String ae = Map.fmtLong(getActiveEntries());
- final String de = Map.fmtLong(getDeletedEntries());
- final String esb = Map.fmtDouble(getEntrySizeBytes());
- final String mub = Map.fmtLong(getTotalUsageBytes());
-
- final StringBuilder sb = new StringBuilder();
- final String thisSimpleName = this.getClass().getSimpleName();
- sb.append("### ").append(thisSimpleName).append(" SUMMARY: ").append(LS);
- sb.append(" Max Coupons Per Entry : ").append(mcpe).append(LS);
- sb.append(" Capacity Coupons Per Entry: ").append(ccpe).append(LS);
- sb.append(" Table Entries : ").append(te).append(LS);
- sb.append(" Capacity Entries : ").append(ce).append(LS);
- sb.append(" Current Count Entries : ").append(cce).append(LS);
- sb.append(" Active Entries : ").append(ae).append(LS);
- sb.append(" Deleted Entries : ").append(de).append(LS);
- sb.append(" Entry Size Bytes : ").append(esb).append(LS);
- sb.append(" Memory Usage Bytes : ").append(mub).append(LS);
- sb.append("### END SKETCH SUMMARY").append(LS);
- return sb.toString();
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hllmap/SingleCouponMap.java b/src/main/java/org/apache/datasketches/hllmap/SingleCouponMap.java
deleted file mode 100644
index 4f96b120d..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/SingleCouponMap.java
+++ /dev/null
@@ -1,276 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hllmap;
-
-import java.util.Arrays;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.hash.MurmurHash3;
-
-/**
- * Implements a key-value map where the value is a single coupon or a map reference.
- * This map holds all keys for all levels of the {@link UniqueCountMap}.
- * This map is implemented with a prime sized Open Address, Double Hash, with a 1-bit state array,
- * which indicates the contents of the value.
- *
- * @author Lee Rhodes
- * @author Alexander Saydakov
- * @author Kevin Lang
- */
-final class SingleCouponMap extends Map {
- private static final double RSE = 0.408 / Math.sqrt(1024);
-
- private int tableEntries_;
- private int capacityEntries_;
- private int curCountEntries_;
- private double entrySizeBytes_;
-
- // Arrays
- private byte[] keysArr_;
- private short[] couponsArr_;
-
- /**
- *
- */
- private byte[] stateArr_;
-
- private SingleCouponMap(final int keySizeBytes) {
- super(keySizeBytes);
- }
-
- static SingleCouponMap getInstance(final int initialNumEntries, final int keySizeBytes) {
- final int tableEntries = nextPrime(initialNumEntries);
-
- final SingleCouponMap map = new SingleCouponMap(keySizeBytes);
- map.tableEntries_ = tableEntries;
- map.capacityEntries_ = (int)(tableEntries * COUPON_MAP_GROW_TRIGGER_FACTOR);
- map.curCountEntries_ = 0;
- map.entrySizeBytes_ = updateEntrySizeBytes(tableEntries, keySizeBytes);
-
- map.keysArr_ = new byte[tableEntries * map.keySizeBytes_];
- map.couponsArr_ = new short[tableEntries];
- map.stateArr_ = new byte[(int) Math.ceil(tableEntries / 8.0)];
- return map;
- }
-
- @Override
- double update(final byte[] key, final short coupon) {
- final int entryIndex = findOrInsertKey(key);
- return update(entryIndex, coupon);
- }
-
- @Override
- double update(final int entryIndex, final short coupon) {
- if (couponsArr_[entryIndex] == 0) {
- couponsArr_[entryIndex] = coupon;
- return 1;
- }
- if (isCoupon(entryIndex)) {
- if (couponsArr_[entryIndex] == coupon) { //duplicate
- return 1;
- }
- return 0; // signal to promote
- }
- return -couponsArr_[entryIndex]; // negative level number
- }
-
- @Override
- double getEstimate(final byte[] key) {
- final int entryIndex = findKey(key);
- if (entryIndex < 0) { return 0; }
- if (isCoupon(entryIndex)) { return 1; }
- return -getCoupon(entryIndex); // negative: level #, zero: signal to promote
- }
-
- @Override
- double getUpperBound(final byte[] key) {
- return getEstimate(key) * (1 + RSE);
- }
-
- @Override
- double getLowerBound(final byte[] key) {
- return getEstimate(key) * (1 - RSE);
- }
-
- /**
- * Returns entryIndex if the given key is found. The coupon may be valid or contain a table index.
- * If not found, returns one's complement entryIndex
- * of an empty slot for insertion, which may be over a deleted key.
- * @param key the given key
- * @return the entryIndex
- */
- @Override
- int findKey(final byte[] key) {
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries_);
- final int stride = getStride(hash[1], tableEntries_);
- final int loopIndex = entryIndex;
-
- do {
- if (couponsArr_[entryIndex] == 0) {
- return ~entryIndex; //empty
- }
- if (Map.arraysEqual(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_)) {
- return entryIndex;
- }
- entryIndex = (entryIndex + stride) % tableEntries_;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- @Override
- int findOrInsertKey(final byte[] key) {
- int entryIndex = findKey(key);
- if (entryIndex < 0) {
- if (curCountEntries_ + 1 > capacityEntries_) {
- resize();
- entryIndex = findKey(key);
- assert entryIndex < 0;
- }
- entryIndex = ~entryIndex;
- System.arraycopy(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_);
- curCountEntries_++;
- }
- return entryIndex;
- }
-
- @Override
- CouponsIterator getCouponsIterator(final int entryIndex) {
- return new CouponsIterator(couponsArr_, entryIndex, 1);
- }
-
- @Override
- int getMaxCouponsPerEntry() {
- return 1;
- }
-
- @Override
- int getCapacityCouponsPerEntry() {
- return 1;
- }
-
- @Override
- int getActiveEntries() {
- return curCountEntries_;
- }
-
- @Override
- int getDeletedEntries() {
- return 0;
- }
-
- boolean isCoupon(final int entryIndex) {
- return !isBitSet(stateArr_, entryIndex);
- }
-
- short getCoupon(final int entryIndex) {
- return couponsArr_[entryIndex];
- }
-
- void setCoupon(final int entryIndex, final short coupon, final boolean isLevel) {
- couponsArr_[entryIndex] = coupon;
- if (isLevel) {
- setBit(stateArr_, entryIndex);
- } else {
- clearBit(stateArr_, entryIndex);
- }
- }
-
- void setLevel(final int entryIndex, final int level) {
- couponsArr_[entryIndex] = (short) level;
- setBit(stateArr_, entryIndex);
- }
-
- @Override
- double getEntrySizeBytes() {
- return entrySizeBytes_;
- }
-
- @Override
- int getTableEntries() {
- return tableEntries_;
- }
-
- @Override
- int getCapacityEntries() {
- return capacityEntries_;
- }
-
- @Override
- int getCurrentCountEntries() {
- return curCountEntries_;
- }
-
- @Override
- long getTotalUsageBytes() {
- final long arrays = keysArr_.length
- + (long)couponsArr_.length * Short.BYTES
- + stateArr_.length;
- final long other = 4 * 4 + 8;
- return arrays + other;
- }
-
- private void resize() {
- final byte[] oldKeysArr = keysArr_;
- final short[] oldCouponsArr = couponsArr_;
- final byte[] oldStateArr = stateArr_;
- final int oldTableEntries = tableEntries_;
- tableEntries_ = nextPrime((int) (curCountEntries_ / COUPON_MAP_TARGET_FILL_FACTOR));
- capacityEntries_ = (int)(tableEntries_ * COUPON_MAP_GROW_TRIGGER_FACTOR);
- keysArr_ = new byte[tableEntries_ * keySizeBytes_];
- couponsArr_ = new short[tableEntries_];
- stateArr_ = new byte[(int) Math.ceil(tableEntries_ / 8.0)];
- entrySizeBytes_ = updateEntrySizeBytes(tableEntries_, keySizeBytes_);
- //move the data
- for (int i = 0; i < oldTableEntries; i++) {
- if (oldCouponsArr[i] != 0) {
- final byte[] key =
- Arrays.copyOfRange(oldKeysArr, i * keySizeBytes_, i * keySizeBytes_ + keySizeBytes_);
- insertEntry(key, oldCouponsArr[i], isBitSet(oldStateArr, i));
- }
- }
- }
-
- // for internal use during resize, so no resize check here
- private void insertEntry(final byte[] key, final int coupon, final boolean setStateOne) {
- final long[] hash = MurmurHash3.hash(key, SEED);
- int entryIndex = getIndex(hash[0], tableEntries_);
- final int stride = getStride(hash[1], tableEntries_);
- final int loopIndex = entryIndex;
- do {
- if (couponsArr_[entryIndex] == 0) {
- System.arraycopy(key, 0, keysArr_, entryIndex * keySizeBytes_, keySizeBytes_);
- setCoupon(entryIndex, (short)coupon, setStateOne);
- return;
- }
- entryIndex = (entryIndex + stride) % tableEntries_;
- } while (entryIndex != loopIndex);
- throw new SketchesArgumentException("Key not found and no empty slots!");
- }
-
- private static final double updateEntrySizeBytes(final int tableEntries, final int keySizeBytes) {
- final double byteFraction = Math.ceil(tableEntries / 8.0) / tableEntries;
- return keySizeBytes + Short.BYTES + byteFraction;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/hllmap/UniqueCountMap.java b/src/main/java/org/apache/datasketches/hllmap/UniqueCountMap.java
deleted file mode 100644
index 28274f648..000000000
--- a/src/main/java/org/apache/datasketches/hllmap/UniqueCountMap.java
+++ /dev/null
@@ -1,364 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.hllmap;
-
-import static org.apache.datasketches.common.Util.LS;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-
-/**
- * This is a real-time, key-value HLL mapping sketch that tracks approximate unique counts of
- * identifiers (the values) associated with each key. An example might be tracking the number of
- * unique user identifiers associated with each IP address. This map has been specifically designed
- * for the use-case where the number of keys is quite large (many millions) and the distribution of
- * identifiers per key is very skewed. A typical distribution where this works well is a
- * power-law distribution of identifiers per key of the form y = Cx-α,
- * where α < 0.5, and C is roughly ymax.
- * For example, with 100M keys, over 75% of the keys would have only
- * one identifier, 99% of the keys would have less than 20 identifiers, 99.9% would have less than
- * 200 identifiers, and a very tiny fraction might have identifiers in the thousands.
- *
- *
- * {@link org.apache.datasketches.kll}
- * {@link org.apache.datasketches.kll}
- * {@link org.apache.datasketches.kll}
- * {@link org.apache.datasketches.kll}
- *
- *
- *
- *
- * {@link org.apache.datasketches.kll}
- * {@link org.apache.datasketches.kll}
- * {@link org.apache.datasketches.kll}
- * {@link org.apache.datasketches.kll}
- *
- *
- * @param sketch The current sketch that needs to be expanded.
- * @param newLevelsArrLen the element length of the new Levels array.
- * @param newItemsArrLen the element length of the new Items array.
- * @return the new expanded MemorySegment with preamble.
- */
- static MemorySegment memorySegmentSpaceMgmt(
- final KllSketch sketch,
- final int newLevelsArrLen,
- final int newItemsArrLen) {
- final KllSketch.SketchType sketchType = sketch.sketchType;
- if (sketchType == KLL_ITEMS_SKETCH) { throw new SketchesArgumentException(UNSUPPORTED_MSG); }
- final MemorySegment oldWseg = sketch.getMemorySegment();
- if (oldWseg == null) {
- return null;
- }
- final int typeBytes = sketchType.getBytes();
- final int requiredSketchBytes = DATA_START_ADR
- + (newLevelsArrLen * Integer.BYTES)
- + (2 * typeBytes)
- + (newItemsArrLen * typeBytes);
-
- if (requiredSketchBytes > oldWseg.byteSize()) { //Acquire new larger MemorySegment
- MemorySegmentRequest mSegReq = sketch.getMemorySegmentRequest();
- if (mSegReq == null) {
- mSegReq = MemorySegmentRequest.DEFAULT;
- }
- final MemorySegment newSeg = mSegReq.request(requiredSketchBytes);
- MemorySegment.copy(oldWseg, 0, newSeg, 0, DATA_START_ADR); //copy preamble (first 20 bytes)
- mSegReq.requestClose(oldWseg);
- return newSeg;
- }
- //Expand in current MemorySegment
- return oldWseg;
- }
-
- private static String outputDataDetail(final KllSketch sketch) {
- final int[] levelsArr = sketch.getLevelsArray(SketchStructure.UPDATABLE);
- final int numLevels = sketch.getNumLevels();
- final int k = sketch.getK();
- final int m = sketch.getM();
- final StringBuilder sb = new StringBuilder();
- sb.append(LS + "### KLL ItemsArray & LevelsArray Detail:").append(LS);
- sb.append("Index, Value").append(LS);
- if (levelsArr[0] > 0) {
- final String gbg = " Free Space, Size = " + levelsArr[0];
- for (int i = 0; i < levelsArr[0]; i++) {
- sb.append(" ").append(i + ", ").append(sketch.getItemAsString(i));
- if (i == 0) { sb.append(gbg); }
- sb.append(LS);
- }
- }
- int level = 0;
- while (level < numLevels) {
- final int fromIndex = levelsArr[level];
- final int toIndex = levelsArr[level + 1]; // exclusive
- String lvlData = "";
- if (fromIndex < toIndex) {
- lvlData = " Level[" + level + "]=" + levelsArr[level]
- + ", Cap=" + KllHelper.levelCapacity(k, numLevels, level, m)
- + ", Size=" + KllHelper.currentLevelSizeItems(level, numLevels, levelsArr)
- + ", Wt=" + (1 << level) + LS;
- }
-
- for (int i = fromIndex; i < toIndex; i++) {
- sb.append(" ").append(i + ", ").append(sketch.getItemAsString(i));
- if (i == fromIndex) { sb.append(lvlData); } else { sb.append(LS); }
- }
- level++;
- }
- sb.append(" ----------Level[" + level + "]=" + levelsArr[level] + ": ItemsArray[].length");
- sb.append(LS);
- sb.append("### End ItemsArray & LevelsArray Detail").append(LS);
- return sb.toString();
- }
-
- private static String outputLevels(final int k, final int m, final int numLevels, final int[] levelsArr) {
- final StringBuilder sb = new StringBuilder();
- sb.append(LS + "### KLL Levels Array:").append(LS)
- .append(" Level, Offset: Nominal Capacity, Actual Capacity").append(LS);
- int level = 0;
- for ( ; level < numLevels; level++) {
- sb.append(" ").append(level).append(", ").append(levelsArr[level]).append(": ")
- .append(KllHelper.levelCapacity(k, numLevels, level, m))
- .append(", ").append(KllHelper.currentLevelSizeItems(level, numLevels, levelsArr)).append(LS);
- }
- sb.append(" ").append(level).append(", ").append(levelsArr[level]).append(": ----ItemsArray[].length")
- .append(LS);
- sb.append("### End Levels Array").append(LS);
- return sb.toString();
- }
-
- static long sumTheSampleWeights(final int num_levels, final int[] levels) {
- long total = 0;
- long weight = 1;
- for (int i = 0; i < num_levels; i++) {
- total += weight * (levels[i + 1] - levels[i]);
- weight *= 2;
- }
- return total;
- }
-
- static byte[] toByteArray(final KllSketch srcSk, final boolean updatable) {
- //ITEMS_SKETCH byte array is never updatable
- final boolean myUpdatable = srcSk.sketchType == KLL_ITEMS_SKETCH ? false : updatable;
- final long srcN = srcSk.getN();
- final SketchStructure tgtStructure;
- if (myUpdatable) { tgtStructure = UPDATABLE; }
- else if (srcN == 0) { tgtStructure = COMPACT_EMPTY; }
- else if (srcN == 1) { tgtStructure = COMPACT_SINGLE; }
- else { tgtStructure = COMPACT_FULL; }
- final int totalBytes = srcSk.currentSerializedSizeBytes(myUpdatable);
- final byte[] bytesOut = new byte[totalBytes];
- final PositionalSegment pSeg = PositionalSegment.wrap(MemorySegment.ofArray(bytesOut));
-
- //ints 0,1
- final byte preInts = (byte)tgtStructure.getPreInts();
- final byte serVer = (byte)tgtStructure.getSerVer();
- final byte famId = (byte)(KLL.getID());
- final byte flags = (byte) ((srcSk.isEmpty() ? EMPTY_BIT_MASK : 0)
- | (srcSk.isLevelZeroSorted() ? LEVEL_ZERO_SORTED_BIT_MASK : 0)
- | (srcSk.getN() == 1 ? SINGLE_ITEM_BIT_MASK : 0));
- final short k = (short) srcSk.getK();
- final byte m = (byte) srcSk.getM();
-
- //load first 8 bytes
- pSeg.setByte(preInts); //byte 0
- pSeg.setByte(serVer);
- pSeg.setByte(famId);
- pSeg.setByte(flags);
- pSeg.setShort(k);
- pSeg.setByte(m);
- pSeg.incrementPosition(1); //byte 7 is unused
-
- if (tgtStructure == COMPACT_EMPTY) {
- return bytesOut;
- }
-
- if (tgtStructure == COMPACT_SINGLE) {
- final byte[] siByteArr = srcSk.getSingleItemByteArr();
- final int len = siByteArr.length;
- pSeg.setByteArray(siByteArr, 0, len);
- pSeg.incrementPosition(-len);
- return bytesOut;
- }
-
- // Tgt is either COMPACT_FULL or UPDATABLE
- //ints 2,3
- final long n = srcSk.getN();
- //ints 4
- final short minK = (short) srcSk.getMinK();
- final byte numLevels = (byte) srcSk.getNumLevels();
- //end of full preamble
- final int[] lvlsArr = srcSk.getLevelsArray(tgtStructure);
- final byte[] minMaxByteArr = srcSk.getMinMaxByteArr();
- final byte[] itemsByteArr = tgtStructure == COMPACT_FULL
- ? srcSk.getRetainedItemsByteArr()
- : srcSk.getTotalItemsByteArr();
-
- pSeg.setLong(n);
- pSeg.setShort(minK);
- pSeg.setByte(numLevels);
- pSeg.incrementPosition(1);
- pSeg.setIntArray(lvlsArr, 0, lvlsArr.length);
- pSeg.setByteArray(minMaxByteArr, 0, minMaxByteArr.length);
- pSeg.setByteArray(itemsByteArr, 0, itemsByteArr.length);
- return bytesOut;
- }
-
- static String toStringImpl(final KllSketch sketch, final boolean withLevels, final boolean withLevelsAndItems) {
- final StringBuilder sb = new StringBuilder();
- final int k = sketch.getK();
- final int m = sketch.getM();
- final int numLevels = sketch.getNumLevels();
- final int[] fullLevelsArr = sketch.getLevelsArray(UPDATABLE);
-
- final SketchType sketchType = sketch.sketchType;
- final boolean hasMSeg = sketch.hasMemorySegment();
- final long n = sketch.getN();
- final String epsPct = String.format("%.3f%%", sketch.getNormalizedRankError(false) * 100);
- final String epsPMFPct = String.format("%.3f%%", sketch.getNormalizedRankError(true) * 100);
- final boolean compact = sketch.isCompactMemorySegmentFormat();
-
- final String directStr = hasMSeg ? "Direct" : "";
- final String compactStr = compact ? "Compact" : "";
- final String readOnlyStr = sketch.isReadOnly() ? "true" + ("(" + (compact ? "Format" : "MemorySegment") + ")") : "false";
- final String skTypeStr = sketchType.getName();
- final String className = "Kll" + directStr + compactStr + skTypeStr;
-
- sb.append(LS + "### ").append(className).append(" Summary:").append(LS);
- sb.append(" K : ").append(k).append(LS);
- sb.append(" Dynamic min K : ").append(sketch.getMinK()).append(LS);
- sb.append(" M : ").append(m).append(LS);
- sb.append(" N : ").append(n).append(LS);
- sb.append(" Epsilon : ").append(epsPct).append(LS);
- sb.append(" Epsilon PMF : ").append(epsPMFPct).append(LS);
- sb.append(" Empty : ").append(sketch.isEmpty()).append(LS);
- sb.append(" Estimation Mode : ").append(sketch.isEstimationMode()).append(LS);
- sb.append(" Levels : ").append(numLevels).append(LS);
- sb.append(" Level 0 Sorted : ").append(sketch.isLevelZeroSorted()).append(LS);
- sb.append(" Capacity Items : ").append(fullLevelsArr[numLevels]).append(LS);
- sb.append(" Retained Items : ").append(sketch.getNumRetained()).append(LS);
- sb.append(" Free Space : ").append(sketch.levelsArr[0]).append(LS);
- sb.append(" ReadOnly : ").append(readOnlyStr).append(LS);
- if (sketchType != KLL_ITEMS_SKETCH) {
- sb.append(" Updatable Storage Bytes: ").append(sketch.currentSerializedSizeBytes(true)).append(LS);
- }
- sb.append(" Compact Storage Bytes : ").append(sketch.currentSerializedSizeBytes(false)).append(LS);
-
- final String emptyStr = (sketchType == KLL_ITEMS_SKETCH) ? "Null" : "NaN";
-
- sb.append(" Min Item : ").append(sketch.isEmpty() ? emptyStr : sketch.getMinItemAsString())
- .append(LS);
- sb.append(" Max Item : ").append(sketch.isEmpty() ? emptyStr : sketch.getMaxItemAsString())
- .append(LS);
- sb.append("### End sketch summary").append(LS);
-
- if (withLevels) {
- sb.append(outputLevels(k, m, numLevels, fullLevelsArr));
- }
-
- if (withLevelsAndItems) {
- sb.append(outputDataDetail(sketch));
- }
- return sb.toString();
- }
-
- /**
- * Returns very conservative upper bound of the number of levels based on n.
- * @param n the length of the stream
- * @return floor( log_2(n) )
- */
- static int ubOnNumLevels(final long n) {
- return 1 + Long.numberOfTrailingZeros(floorPowerOf2(n));
- }
-
- /**
- * This grows the levels arr by 1 (if needed) and increases the capacity of the items array
- * at the bottom. Only numLevels, the levels array and the items array are affected.
- * This assumes sketch is writable and UPDATABLE.
- * @param sketch the current sketch
- */
- static void addEmptyTopLevelToCompletelyFullSketch(final KllSketch sketch) {
- final SketchType sketchType = sketch.sketchType;
-
- final int[] myCurLevelsArr = sketch.getLevelsArray(sketch.sketchStructure);
- final int myCurNumLevels = sketch.getNumLevels();
- final int myCurTotalItemsCapacity = myCurLevelsArr[myCurNumLevels];
-
- final int myNewNumLevels;
- final int[] myNewLevelsArr;
- final int myNewTotalItemsCapacity;
-
-
- double[] myCurDoubleItemsArr = null;
- double[] myNewDoubleItemsArr = null;
- double minDouble = Double.NaN;
- double maxDouble = Double.NaN;
-
- float[] myCurFloatItemsArr = null;
- float[] myNewFloatItemsArr = null;
- float minFloat = Float.NaN;
- float maxFloat = Float.NaN;
-
- long[] myCurLongItemsArr = null;
- long[] myNewLongItemsArr = null;
- long minLong = Long.MAX_VALUE;
- long maxLong = Long.MIN_VALUE;
-
- Object[] myCurItemsArr = null;
- Object[] myNewItemsArr = null;
- Object minItem = null;
- Object maxItem = null;
-
- if (sketchType == KLL_DOUBLES_SKETCH) {
- final KllDoublesSketch dblSk = (KllDoublesSketch) sketch;
- myCurDoubleItemsArr = dblSk.getDoubleItemsArray();
- minDouble = dblSk.getMinItem();
- maxDouble = dblSk.getMaxItem();
- //assert we are following a certain growth scheme
- assert myCurDoubleItemsArr.length == myCurTotalItemsCapacity;
- }
- else if (sketchType == KLL_FLOATS_SKETCH) {
- final KllFloatsSketch fltSk = (KllFloatsSketch) sketch;
- myCurFloatItemsArr = fltSk.getFloatItemsArray();
- minFloat = fltSk.getMinItem();
- maxFloat = fltSk.getMaxItem();
- //assert we are following a certain growth scheme
- assert myCurFloatItemsArr.length == myCurTotalItemsCapacity;
- }
- else if (sketchType == KLL_LONGS_SKETCH) {
- final KllLongsSketch lngSk = (KllLongsSketch) sketch;
- myCurLongItemsArr = lngSk.getLongItemsArray();
- minLong = lngSk.getMinItem();
- maxLong = lngSk.getMaxItem();
- //assert we are following a certain growth scheme
- assert myCurLongItemsArr.length == myCurTotalItemsCapacity;
- }
- else { //sketchType == ITEMS_SKETCH
- final KllItemsSketch itmSk = (KllItemsSketch) sketch;
- myCurItemsArr = itmSk.getTotalItemsArray();
- minItem = itmSk.getMinItem();
- maxItem = itmSk.getMaxItem();
- }
- assert myCurLevelsArr[0] == 0; //definition of full is part of the growth scheme
-
- final int deltaItemsCap = levelCapacity(sketch.getK(), myCurNumLevels + 1, 0, sketch.getM());
- myNewTotalItemsCapacity = myCurTotalItemsCapacity + deltaItemsCap;
-
- // Check if growing the levels arr if required.
- // Note that merging MIGHT over-grow levels_, in which case we might not have to grow it
- final boolean growLevelsArr = myCurLevelsArr.length < (myCurNumLevels + 2);
-
- // GROW LEVELS ARRAY
- if (growLevelsArr) {
- //grow levels arr by one and copy the old data to the new array, extra space at the top.
- myNewLevelsArr = Arrays.copyOf(myCurLevelsArr, myCurNumLevels + 2);
- assert myNewLevelsArr.length == (myCurLevelsArr.length + 1);
- myNewNumLevels = myCurNumLevels + 1;
- sketch.incNumLevels(); //increment for off-heap
- } else {
- myNewLevelsArr = myCurLevelsArr;
- myNewNumLevels = myCurNumLevels;
- }
- // This loop updates all level indices EXCLUDING the "extra" index at the top
- for (int level = 0; level <= (myNewNumLevels - 1); level++) {
- myNewLevelsArr[level] += deltaItemsCap;
- }
- myNewLevelsArr[myNewNumLevels] = myNewTotalItemsCapacity; // initialize the new "extra" index at the top
-
- // GROW items ARRAY
- if (sketchType == KLL_DOUBLES_SKETCH) {
- myNewDoubleItemsArr = new double[myNewTotalItemsCapacity];
- // copy and shift the current data into the new array
- System.arraycopy(myCurDoubleItemsArr, 0, myNewDoubleItemsArr, deltaItemsCap, myCurTotalItemsCapacity);
- }
- else if (sketchType == KLL_FLOATS_SKETCH) {
- myNewFloatItemsArr = new float[myNewTotalItemsCapacity];
- // copy and shift the current items data into the new array
- System.arraycopy(myCurFloatItemsArr, 0, myNewFloatItemsArr, deltaItemsCap, myCurTotalItemsCapacity);
- }
- else if (sketchType == KLL_LONGS_SKETCH) {
- myNewLongItemsArr = new long[myNewTotalItemsCapacity];
- // copy and shift the current items data into the new array
- System.arraycopy(myCurLongItemsArr, 0, myNewLongItemsArr, deltaItemsCap, myCurTotalItemsCapacity);
- }
- else { //sketchType == ITEMS_SKETCH
- myNewItemsArr = new Object[myNewTotalItemsCapacity];
- // copy and shift the current items data into the new array
- System.arraycopy(myCurItemsArr, 0, myNewItemsArr, deltaItemsCap, myCurTotalItemsCapacity);
- }
-
- //MemorySegment SPACE MANAGEMENT
- if (sketch.getMemorySegment() != null) {
- final MemorySegment wseg = memorySegmentSpaceMgmt(sketch, myNewLevelsArr.length, myNewTotalItemsCapacity);
- sketch.setMemorySegment(wseg);
- }
-
- //update our sketch with new expanded spaces
- sketch.setNumLevels(myNewNumLevels); //for off-heap only
- sketch.setLevelsArray(myNewLevelsArr); //the KllSketch copy
- if (sketchType == KLL_DOUBLES_SKETCH) {
- final KllDoublesSketch dblSk = (KllDoublesSketch) sketch;
- dblSk.setMinItem(minDouble);
- dblSk.setMaxItem(maxDouble);
- dblSk.setDoubleItemsArray(myNewDoubleItemsArr);
- }
- else if (sketchType == KLL_FLOATS_SKETCH) {
- final KllFloatsSketch fltSk = (KllFloatsSketch) sketch;
- fltSk.setMinItem(minFloat);
- fltSk.setMaxItem(maxFloat);
- fltSk.setFloatItemsArray(myNewFloatItemsArr);
- }
- else if (sketchType == KLL_LONGS_SKETCH) {
- final KllLongsSketch lngSk = (KllLongsSketch) sketch;
- lngSk.setMinItem(minLong);
- lngSk.setMaxItem(maxLong);
- lngSk.setLongItemsArray(myNewLongItemsArr);
- }
- else { //sketchType == ITEMS_SKETCH
- final KllItemsSketch itmSk = (KllItemsSketch) sketch;
- itmSk.setMinItem(minItem);
- itmSk.setMaxItem(maxItem);
- itmSk.setItemsArray(myNewItemsArr);
- }
-
- } //END of addEmptyTopLevelToCompletelyFullSketch(...)
-
- /**
- * Finds the first level starting with level 0 that exceeds its nominal capacity
- * @param k configured size of sketch. Range [m, 2^16]
- * @param m minimum level size. Default is 8.
- * @param numLevels one-based number of current levels
- * @return level to compact
- */
- static int findLevelToCompact(final int k, final int m, final int numLevels, final int[] levels) {
- int level = 0;
- while (true) {
- assert level < numLevels;
- final int pop = levels[level + 1] - levels[level];
- final int cap = KllHelper.levelCapacity(k, numLevels, level, m);
- if (pop >= cap) {
- return level;
- }
- level++;
- }
- }
-
- /**
- * Computes the actual item capacity of a given level given its depth index.
- * If the depth of levels exceeds 30, this uses a folding technique to accurately compute the
- * actual level capacity up to a depth of 60 (or 61 levels).
- * Without folding, the internal calculations would exceed the capacity of a long.
- * This method just decides whether folding is required or not.
- * @param k the configured k of the sketch
- * @param depth the zero-based index of the level being computed.
- * @return the actual capacity of a given level given its depth index.
- */
- static long intCapAux(final int k, final int depth) {
- if (depth <= 30) { return intCapAuxAux(k, depth); }
- final int half = depth / 2;
- final int rest = depth - half;
- final long tmp = intCapAuxAux(k, half);
- return intCapAuxAux(tmp, rest);
- }
-
- /**
- * Performs the integer based calculation of an individual level (or folded level).
- * @param k the configured k of the sketch
- * @param depth the zero-based index of the level being computed. The max depth is 30!
- * @return the actual capacity of a given level given its depth index.
- */
- static long intCapAuxAux(final long k, final int depth) {
- final long twok = k << 1; // for rounding at the end, pre-multiply by 2 here, divide by 2 during rounding.
- final long tmp = ((twok << depth) / powersOfThree[depth]); //2k* (2/3)^depth. 2k also keeps the fraction larger.
- final long result = ((tmp + 1L) >>> 1); // (tmp + 1)/2. If odd, round up. This guarantees an integer.
- assert (result <= k);
- return result;
- }
-
- private final static boolean enablePrinting = false;
-
- /**
- * @param format the format
- * @param args the args
- */
- private static final void printf(final String format, final Object ... args) {
- if (enablePrinting) { System.out.printf(format, args); }
- }
-
- /**
- * @param o the Object to println
- */
- private static void println(final Object o) {
- if (enablePrinting) { System.out.println(o.toString()); }
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/kll/KllItemsHelper.java b/src/main/java/org/apache/datasketches/kll/KllItemsHelper.java
deleted file mode 100644
index a066bf536..000000000
--- a/src/main/java/org/apache/datasketches/kll/KllItemsHelper.java
+++ /dev/null
@@ -1,518 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.kll;
-
-import static java.lang.Math.max;
-import static java.lang.Math.min;
-import static java.lang.reflect.Array.newInstance;
-import static org.apache.datasketches.common.Util.isEven;
-import static org.apache.datasketches.common.Util.isOdd;
-import static org.apache.datasketches.kll.KllHelper.findLevelToCompact;
-import static org.apache.datasketches.kll.KllSketch.DEFAULT_M;
-
-import java.util.Arrays;
-import java.util.Comparator;
-import java.util.Random;
-
-import org.apache.datasketches.common.Util;
-
-/**
- * Static methods to support KllItemsSketch
- * @author Kevin Lang
- * @author Lee Rhodes
- */
-@SuppressWarnings("unchecked")
-final class KllItemsHelper {
-
- /**
- * Create Items Array from given item and weight.
- * Used with weighted update only.
- * @param item the given item
- * @param weight the given weight
- * @return the Items Array.
- */
- static
- *
- *
- *
- * Visual Layout
- * The low significance bytes of the visual data structure below are on the left.
- * The multi-byte primitives are stored in native byte order.
- * The numeric byte and short fields are treated as unsigned.
- * The numeric int and long fields are treated as signed.
- *
- * Preamble Formats
- * The preamble has four formats:
- *
- *
- *
- *
- * Visual Layout
- * The fields in braces are those that can be variable in size.
- *
- * {@code
- * Serialized COMPACT_EMPTY sketch structure, Empty (8 bytes)
- * and COMPACT_SINGLE sketch structure, (single item) (8 + itemSize):
- * Int Adr: Byte Adr ->
- * 0 || 0 | 1 | 2 | 3 |
- * | PreambleInts | SerVer | FamID | Flags |
- *
- * 1 || 4 | 5 | 6 | 7 |
- * ||-----------K-----------| M | unused |
- *
- * 2 || 8 |
- * ||{Single Item} ->
- *
- * Serialized COMPACT_FULL sketch structure, more than one item:
- * Int Adr: Byte Adr ->
- * 0 || 0 | 1 | 2 | 3 |
- * || PreambleInts | SerVer | FamID | Flags |
- *
- * 1 || 4 | 5 | 6 | 7 |
- * ||-----------K-----------| M | unused |
- *
- * 2,3 || 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
- * ||-----------------N_LONG-----------------|
- *
- * 4 || 16 | 17 | 18 | 19 |
- * ||------Min K------------|NumLvls| unused |
- *
- * 5 || 20 |
- * { Levels Array }
- * { Min Item }
- * { Max Item }
- * { Items Array }
- *
- * Serialization Combinations for SerVer and PreambleInts
- * | Sketch Structure | SerVer | PreInts |
- * |------------------|----------------|------------------|
- * | Compact Empty | Empty/Full (1) | Empty/Single (2) | ReadOnly, 8 byte Preamble, nothing else
- * | Compact Single | Single (2) | Empty/Single (2) | ReadOnly, 8 byte Preamble + Single Item
- * | Compact Full | Empty/Full (1) | Full (5) | ReadOnly, 20 Byte Preamble, Short LevelsArr, Retained Items
- * | Updatable | Updatable (3) | Full (5) | Updatable, 20 Byte Preamble, Full LevelsArr, All Items
- * | ERROR | Single (2) | Full (5) |
- * | ERROR | Updatable (3) | Empty/Single (2) |
- * }
- *
- * @author Lee Rhodes
- */
-final class KllPreambleUtil {
-
- private KllPreambleUtil() {}
-
- // Preamble byte addresses
- static final int PREAMBLE_INTS_BYTE_ADR = 0;
- static final int SER_VER_BYTE_ADR = 1;
- static final int FAMILY_BYTE_ADR = 2;
- static final int FLAGS_BYTE_ADR = 3;
- static final int K_SHORT_ADR = 4; // to 5
- static final int M_BYTE_ADR = 6;
- // 7 is reserved for future use
- // SINGLE ITEM ONLY
- static final int DATA_START_ADR_SINGLE_ITEM = 8; //also ok for empty
-
- // MULTI-ITEM
- static final int N_LONG_ADR = 8; // to 15
- static final int MIN_K_SHORT_ADR = 16; // to 17
- static final int NUM_LEVELS_BYTE_ADR = 18;
-
- // 19 is reserved for future use
- static final int DATA_START_ADR = 20; // Full Sketch, not single item
-
- // Other static members
- static final byte SERIAL_VERSION_EMPTY_FULL = 1; // Empty or full preamble, NOT single item format, NOT updatable
- static final byte SERIAL_VERSION_SINGLE = 2; // only single-item format, NOT updatable
- static final byte SERIAL_VERSION_UPDATABLE = 3; // PreInts=5, Full preamble + LevelsArr + min, max + empty space
- static final byte PREAMBLE_INTS_EMPTY_SINGLE = 2; // for empty or single item
- static final byte PREAMBLE_INTS_FULL = 5; // Full preamble, not empty nor single item.
- static final byte KLL_FAMILY = 15;
-
- // Flag bit masks
- static final int EMPTY_BIT_MASK = 1;
- static final int LEVEL_ZERO_SORTED_BIT_MASK = 2;
- static final int SINGLE_ITEM_BIT_MASK = 4;
-
- /**
- * Returns a human readable string summary of the internal state of the given sketch byte array.
- * Used primarily in testing.
- *
- * @param byteArr the given sketch byte array.
- * @param includeData if true, includes detail of retained data.
- * @return the summary string.
- */
- static String toString(final byte[] byteArr, final SketchType sketchType, final boolean includeData) {
- final MemorySegment seg = MemorySegment.ofArray(byteArr);
- return toString(seg, sketchType, includeData, null);
- }
-
- /**
- * Returns a human readable string summary of the internal state of the given sketch byte array.
- * Used primarily in testing.
- *
- * @param byteArr the given sketch byte array.
- * @param includeData if true, includes detail of retained data.
- * @param serDe the serialization/deserialization class, required for KllItemsSketch.
- * @return the summary string.
- */
- static String toString(final byte[] byteArr, final SketchType sketchType, final boolean includeData,
- final ArrayOfItemsSerDe serDe) {
- final MemorySegment seg = MemorySegment.ofArray(byteArr);
- return toString(seg, sketchType, includeData, serDe);
- }
-
- /**
- * Returns a human readable string summary of the internal state of the given MemorySegment.
- * Used primarily in testing.
- *
- * @param seg the given MemorySegment
- * @param includeData if true, includes detail of retained data.
- * @return the summary string.
- */
- static String toString(final MemorySegment seg, final SketchType sketchType, final boolean includeData) {
- return toString(seg, sketchType, includeData, null);
- }
-
- /**
- * Returns a human readable string summary of the internal state of the given MemorySegment.
- * Used primarily in testing.
- *
- * @param seg the given MemorySegment
- * @param sketchType the sketch type: FLOATS_SKETCH, DOUBLES_SKETCH, LONGS_SKETCH, or ITEMS_SKETCH.
- * @param includeData if true, includes detail of retained data.
- * @param serDe must be supplied for KllItemsSketch, otherwise can be null.
- * @return the summary string.
- */
- static {@code
- * SketchIterator itr = sketch.iterator();
- * while (itr.next()) {
- * ...get*();
- * }
- * }
- *
- * @author Lee Rhodes
- */
-public class KllSketchIterator implements QuantilesSketchIteratorAPI {
- protected final int[] levelsArr;
- protected final int numLevels;
- protected int level;
- protected int index;
- protected long weight;
- protected boolean isInitialized_;
-
- KllSketchIterator(final int[] levelsArr, final int numLevels) {
- this.levelsArr = levelsArr;
- this.numLevels = numLevels;
- this.isInitialized_ = false;
- }
-
- @Override
- public long getWeight() {
- return weight;
- }
-
- @Override
- public boolean next() {
- if (!isInitialized_) {
- level = 0;
- index = levelsArr[level];
- weight = 1;
- isInitialized_ = true;
- } else {
- index++;
- }
- if (index < levelsArr[level + 1]) {
- return true;
- }
- // go to the next non-empty level
- do {
- level++;
- if (level == numLevels) {
- return false; // run out of levels
- }
- weight *= 2;
- } while (levelsArr[level] == levelsArr[level + 1]);
- index = levelsArr[level];
- return true;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/kll/package-info.java b/src/main/java/org/apache/datasketches/kll/package-info.java
deleted file mode 100644
index 6da3c26dd..000000000
--- a/src/main/java/org/apache/datasketches/kll/package-info.java
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * This package is for the implementations of the sketch algorithm developed by Zohar Karnin, Kevin Lang,
- * and Edo Liberty that is commonly referred to as the "KLL" sketch after the authors' last names.
- */
-
-package org.apache.datasketches.kll;
diff --git a/src/main/java/org/apache/datasketches/package-info.java b/src/main/java/org/apache/datasketches/package-info.java
deleted file mode 100644
index 3683a3543..000000000
--- a/src/main/java/org/apache/datasketches/package-info.java
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-/**
- * This package is the parent package for all sketch families and common code areas.
- *
- * @author Lee Rhodes
- */
-package org.apache.datasketches;
diff --git a/src/main/java/org/apache/datasketches/partitions/BoundsRule.java b/src/main/java/org/apache/datasketches/partitions/BoundsRule.java
deleted file mode 100644
index 3cd38996b..000000000
--- a/src/main/java/org/apache/datasketches/partitions/BoundsRule.java
+++ /dev/null
@@ -1,48 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.partitions;
-
-/**
- * This instructs the user about which of the upper and lower bounds of a partition definition row
- * should be included with the returned data.
- */
-public enum BoundsRule {
-
- /**
- * Include both the upper and lower bounds
- */
- INCLUDE_BOTH,
-
- /**
- * Include only the upper bound but not the lower bound
- */
- INCLUDE_UPPER,
-
- /**
- * Include only the lower bound but not the upper bound
- */
- INCLUDE_LOWER,
-
- /**
- * Include none
- */
- INCLUDE_NEITHER;
-
-}
diff --git a/src/main/java/org/apache/datasketches/partitions/Partitioner.java b/src/main/java/org/apache/datasketches/partitions/Partitioner.java
deleted file mode 100644
index dd69f1ec2..000000000
--- a/src/main/java/org/apache/datasketches/partitions/Partitioner.java
+++ /dev/null
@@ -1,244 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.partitions;
-
-import static java.lang.Math.ceil;
-import static java.lang.Math.log;
-import static java.lang.Math.max;
-import static java.lang.Math.min;
-import static java.lang.Math.pow;
-import static java.lang.Math.round;
-import static java.util.Collections.unmodifiableList;
-import static org.apache.datasketches.partitions.BoundsRule.INCLUDE_BOTH;
-import static org.apache.datasketches.partitions.BoundsRule.INCLUDE_LOWER;
-import static org.apache.datasketches.partitions.BoundsRule.INCLUDE_NEITHER;
-import static org.apache.datasketches.partitions.BoundsRule.INCLUDE_UPPER;
-import static org.apache.datasketches.quantilescommon.QuantileSearchCriteria.INCLUSIVE;
-import static org.apache.datasketches.quantilescommon.QuantilesAPI.EMPTY_MSG;
-
-import java.util.ArrayDeque;
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.quantilescommon.GenericPartitionBoundaries;
-import org.apache.datasketches.quantilescommon.PartitioningFeature;
-import org.apache.datasketches.quantilescommon.QuantileSearchCriteria;
-import org.apache.datasketches.quantilescommon.QuantilesGenericAPI;
-
-/**
- * A partitioning process that can partition very large data sets into thousands
- * of partitions of approximately the same size.
- *
- * the quantiles sketch that implements both QuantilesGenericAPI and PartitioningFeature.
- */
-public class Partitioner the sketch type
- * @author Lee Rhodes
- */
-public interface SketchFillRequest
- *
- *
- * {@code
- * Long || Start Byte Adr: Common for both QuantilesDoublesSketch and QuantilesItemsSketch
- * Adr:
- * || 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 |
- * 0 ||------unused-----|--------K--------| Flags | FamID | SerVer | Preamble_Longs |
- *
- * || 15 | 14 | 13 | 12 | 11 | 10 | 9 | 8 |
- * 1 ||-----------------------------------N_LONG--------------------------------------|
- *
- * Applies only to QuantilesDoublesSketch:
- * (QuantilesItemsSketch has elements in the same order, but size depends on sizeOf(T)
- *
- * || 23 | 22 | 21 | 20 | 19 | 18 | 17 | 16 |
- * 2 ||---------------------------START OF DATA, MIN_DOUBLE---------------------------|
- *
- * || 31 | 30 | 29 | 28 | 27 | 26 | 25 | 24 |
- * 3 ||----------------------------------MAX_DOUBLE-----------------------------------|
- *
- * || 39 | 38 | 37 | 36 | 35 | 34 | 33 | 32 |
- * 4 ||---------------------------START OF COMBINED BUfFER----------------------------|
- * }
- *
- * @author Lee Rhodes
- */
-final class PreambleUtil {
-
- private PreambleUtil() {}
-
- // ###### DO NOT MESS WITH THIS FROM HERE ...
- // Preamble byte Addresses
- static final int PREAMBLE_LONGS_BYTE = 0;
- static final int SER_VER_BYTE = 1;
- static final int FAMILY_BYTE = 2;
- static final int FLAGS_BYTE = 3;
- static final int K_SHORT = 4; //to 5
- static final int N_LONG = 8; //to 15
-
- //After Preamble:
- static final int MIN_DOUBLE = 16; //to 23 (Only for QuantilesDoublesSketch)
- static final int MAX_DOUBLE = 24; //to 31 (Only for QuantilesDoublesSketch)
- static final int COMBINED_BUFFER = 32; //to 39 (Only for QuantilesDoublesSketch)
-
- // flag bit masks
- static final int RESERVED_FLAG_MASK = 1;
- static final int READ_ONLY_FLAG_MASK = 2;
- static final int EMPTY_FLAG_MASK = 4;
- static final int COMPACT_FLAG_MASK = 8;
- static final int ORDERED_FLAG_MASK = 16;
-
- /**
- * Default K for about 1.7% normalized rank accuracy
- */
- static final int DEFAULT_K = 128;
-
- // ###### TO HERE.
-
- // STRINGS
- /**
- * Returns a human readable string summary of the internal state of the given byte array.
- * Used primarily in testing.
- *
- * @param byteArr the given byte array.
- * @param isDoublesSketch flag to indicate that the byte array represents QuantilesDoublesSketch
- * to output min and max quantiles in the summary
- * @return the summary string.
- */
- static String toString(final byte[] byteArr, final boolean isDoublesSketch) {
- final MemorySegment seg = MemorySegment.ofArray(byteArr);
- return toString(seg, isDoublesSketch);
- }
-
- /**
- * Returns a human readable string summary of the Preamble of the given MemorySegment. If this MemorySegment
- * image is from a QuantilesDoublesSketch, the MinQuantile and MaxQuantile will also be output.
- * Used primarily in testing.
- *
- * @param seg the given MemorySegment
- * @param isDoublesSketch flag to indicate that the byte array represents QuantilesDoublesSketch
- * to output min and max quantiles in the summary
- * @return the summary string.
- */
- static String toString(final MemorySegment seg, final boolean isDoublesSketch) {
- return memorySegmentToString(seg, isDoublesSketch);
- }
-
- private static String memorySegmentToString(final MemorySegment srcSeg, final boolean isDoublesSketch) {
- final int preLongs = extractPreLongs(srcSeg); //either 1 or 2
- final int serVer = extractSerVer(srcSeg);
- final int familyID = extractFamilyID(srcSeg);
- final String famName = idToFamily(familyID).toString();
- final int flags = extractFlags(srcSeg);
- final boolean readOnly = (flags & READ_ONLY_FLAG_MASK) > 0;
- final boolean empty = (flags & EMPTY_FLAG_MASK) > 0;
- final boolean compact = (flags & COMPACT_FLAG_MASK) > 0;
- final boolean ordered = (flags & ORDERED_FLAG_MASK) > 0;
- final int k = extractK(srcSeg);
-
- final long n = (preLongs == 1) ? 0L : extractN(srcSeg);
- double minDouble = Double.NaN;
- double maxDouble = Double.NaN;
- if ((preLongs > 1) && isDoublesSketch) { // preLongs = 2 or 3
- minDouble = extractMinDouble(srcSeg);
- maxDouble = extractMaxDouble(srcSeg);
- }
-
- final StringBuilder sb = new StringBuilder();
- sb.append(LS);
- sb.append("### QUANTILES SKETCH PREAMBLE SUMMARY:").append(LS);
- sb.append("Byte 0: Preamble Longs : ").append(preLongs).append(LS);
- sb.append("Byte 1: Serialization Version: ").append(serVer).append(LS);
- sb.append("Byte 2: Family : ").append(famName).append(LS);
- sb.append("Byte 3: Flags Field : ").append(String.format("%02o", flags)).append(LS);
- sb.append(" RESERVED : ").append(LS);
- sb.append(" READ ONLY : ").append(readOnly).append(LS);
- sb.append(" EMPTY : ").append(empty).append(LS);
- sb.append(" COMPACT : ").append(compact).append(LS);
- sb.append(" ORDERED : ").append(ordered).append(LS);
- sb.append("Bytes 4-5 : K : ").append(k).append(LS);
- if (preLongs == 1) {
- sb.append(" --ABSENT, ASSUMED:").append(LS);
- }
- sb.append("Bytes 8-15 : N : ").append(n).append(LS);
- if (isDoublesSketch) {
- sb.append("MinDouble : ").append(minDouble).append(LS);
- sb.append("MaxDouble : ").append(maxDouble).append(LS);
- }
- sb.append("Retained Items : ").append(computeRetainedItems(k, n)).append(LS);
- sb.append("Total Bytes : ").append(srcSeg.byteSize()).append(LS);
- sb.append("### END SKETCH PREAMBLE SUMMARY").append(LS);
- return sb.toString();
- }
-
- //@formatter:on
- static int extractPreLongs(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, PREAMBLE_LONGS_BYTE) & 0XFF;
- }
-
- static int extractSerVer(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, SER_VER_BYTE) & 0XFF;
- }
-
- static int extractFamilyID(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, FAMILY_BYTE) & 0XFF;
- }
-
- static int extractFlags(final MemorySegment seg) {
- return seg.get(JAVA_BYTE, FLAGS_BYTE) & 0XFF;
- }
-
- static int extractK(final MemorySegment seg) {
- return seg.get(JAVA_SHORT_UNALIGNED, K_SHORT) & 0XFFFF;
- }
-
- static long extractN(final MemorySegment seg) {
- return seg.get(JAVA_LONG_UNALIGNED, N_LONG);
- }
-
- static double extractMinDouble(final MemorySegment seg) {
- return seg.get(JAVA_DOUBLE_UNALIGNED, MIN_DOUBLE);
- }
-
- static double extractMaxDouble(final MemorySegment seg) {
- return seg.get(JAVA_DOUBLE_UNALIGNED, MAX_DOUBLE);
- }
-
- static void insertPreLongs(final MemorySegment wseg, final int numPreLongs) {
- wseg.set(JAVA_BYTE, PREAMBLE_LONGS_BYTE, (byte) numPreLongs);
- }
-
- static void insertSerVer(final MemorySegment wseg, final int serVer) {
- wseg.set(JAVA_BYTE, SER_VER_BYTE, (byte) serVer);
- }
-
- static void insertFamilyID(final MemorySegment wseg, final int famId) {
- wseg.set(JAVA_BYTE, FAMILY_BYTE, (byte) famId);
- }
-
- static void insertFlags(final MemorySegment wseg, final int flags) {
- wseg.set(JAVA_BYTE, FLAGS_BYTE, (byte) flags);
- }
-
- static void insertK(final MemorySegment wseg, final int k) {
- wseg.set(JAVA_SHORT_UNALIGNED, K_SHORT, (short) k);
- }
-
- static void insertN(final MemorySegment wseg, final long n) {
- wseg.set(JAVA_LONG_UNALIGNED, N_LONG, n);
- }
-
- static void insertMinDouble(final MemorySegment wseg, final double minDouble) {
- wseg.set(JAVA_DOUBLE_UNALIGNED, MIN_DOUBLE, minDouble);
- }
-
- static void insertMaxDouble(final MemorySegment wseg, final double maxDouble) {
- wseg.set(JAVA_DOUBLE_UNALIGNED, MAX_DOUBLE, maxDouble);
- }
-}
diff --git a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesSketch.java b/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesSketch.java
deleted file mode 100644
index 85bf957d7..000000000
--- a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesSketch.java
+++ /dev/null
@@ -1,830 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.quantiles;
-
-import static java.lang.Math.max;
-import static java.lang.Math.min;
-import static java.lang.System.arraycopy;
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.common.Util.ceilingPowerOf2;
-import static org.apache.datasketches.quantiles.ClassicUtil.MAX_PRELONGS;
-import static org.apache.datasketches.quantiles.ClassicUtil.MIN_K;
-import static org.apache.datasketches.quantiles.ClassicUtil.checkIsMemorySegmentCompact;
-import static org.apache.datasketches.quantiles.ClassicUtil.checkK;
-import static org.apache.datasketches.quantiles.ClassicUtil.computeNumLevelsNeeded;
-import static org.apache.datasketches.quantiles.ClassicUtil.computeRetainedItems;
-import static org.apache.datasketches.quantiles.DoublesSketchAccessor.BB_LVL_IDX;
-
-import java.lang.foreign.MemorySegment;
-import java.util.Arrays;
-import java.util.Random;
-
-import org.apache.datasketches.common.MemorySegmentRequest;
-import org.apache.datasketches.common.MemorySegmentStatus;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-import org.apache.datasketches.quantilescommon.DoublesSketchSortedView;
-import org.apache.datasketches.quantilescommon.QuantileSearchCriteria;
-import org.apache.datasketches.quantilescommon.QuantilesAPI;
-import org.apache.datasketches.quantilescommon.QuantilesDoublesAPI;
-import org.apache.datasketches.quantilescommon.QuantilesDoublesSketchIteratorAPI;
-
-/**
- * This is an implementation of the Low Discrepancy Mergeable Quantiles Sketch, using doubles,
- * described in section 3.2 of the journal version of the paper "Mergeable Summaries"
- * by Agarwal, Cormode, Huang, Phillips, Wei, and Yi:
- *
- *
- *
- *
-Table Guide for QuantilesDoublesSketch Size in Bytes and Approximate Error:
- K => | 16 32 64 128 256 512 1,024
- ~ Error => | 12.145% 6.359% 3.317% 1.725% 0.894% 0.463% 0.239%
- N | Size in Bytes ->
-------------------------------------------------------------------------
- 0 | 8 8 8 8 8 8 8
- 1 | 72 72 72 72 72 72 72
- 3 | 72 72 72 72 72 72 72
- 7 | 104 104 104 104 104 104 104
- 15 | 168 168 168 168 168 168 168
- 31 | 296 296 296 296 296 296 296
- 63 | 424 552 552 552 552 552 552
- 127 | 552 808 1,064 1,064 1,064 1,064 1,064
- 255 | 680 1,064 1,576 2,088 2,088 2,088 2,088
- 511 | 808 1,320 2,088 3,112 4,136 4,136 4,136
- 1,023 | 936 1,576 2,600 4,136 6,184 8,232 8,232
- 2,047 | 1,064 1,832 3,112 5,160 8,232 12,328 16,424
- 4,095 | 1,192 2,088 3,624 6,184 10,280 16,424 24,616
- 8,191 | 1,320 2,344 4,136 7,208 12,328 20,520 32,808
- 16,383 | 1,448 2,600 4,648 8,232 14,376 24,616 41,000
- 32,767 | 1,576 2,856 5,160 9,256 16,424 28,712 49,192
- 65,535 | 1,704 3,112 5,672 10,280 18,472 32,808 57,384
- 131,071 | 1,832 3,368 6,184 11,304 20,520 36,904 65,576
- 262,143 | 1,960 3,624 6,696 12,328 22,568 41,000 73,768
- 524,287 | 2,088 3,880 7,208 13,352 24,616 45,096 81,960
- 1,048,575 | 2,216 4,136 7,720 14,376 26,664 49,192 90,152
- 2,097,151 | 2,344 4,392 8,232 15,400 28,712 53,288 98,344
- 4,194,303 | 2,472 4,648 8,744 16,424 30,760 57,384 106,536
- 8,388,607 | 2,600 4,904 9,256 17,448 32,808 61,480 114,728
- 16,777,215 | 2,728 5,160 9,768 18,472 34,856 65,576 122,920
- 33,554,431 | 2,856 5,416 10,280 19,496 36,904 69,672 131,112
- 67,108,863 | 2,984 5,672 10,792 20,520 38,952 73,768 139,304
- 134,217,727 | 3,112 5,928 11,304 21,544 41,000 77,864 147,496
- 268,435,455 | 3,240 6,184 11,816 22,568 43,048 81,960 155,688
- 536,870,911 | 3,368 6,440 12,328 23,592 45,096 86,056 163,880
- 1,073,741,823 | 3,496 6,696 12,840 24,616 47,144 90,152 172,072
- 2,147,483,647 | 3,624 6,952 13,352 25,640 49,192 94,248 180,264
- 4,294,967,295 | 3,752 7,208 13,864 26,664 51,240 98,344 188,456
- *
- *
- * @see QuantilesAPI
- */
-public abstract class QuantilesDoublesSketch implements QuantilesDoublesAPI, MemorySegmentStatus {
-
- /**
- * Setting the seed makes the results of the sketch deterministic if the input quantiles are
- * received in exactly the same order. This is only useful when performing test comparisons,
- * otherwise is not recommended.
- */
- static Random rand = new Random();
-
- /**
- * Parameter that controls space usage of sketch and accuracy of estimates.
- */
- final int k_;
-
- /**
- * holder for SortedView
- */
- DoublesSketchSortedView doublesSV = null;
-
- QuantilesDoublesSketch(final int k) {
- checkK(k);
- k_ = k;
- }
-
- synchronized static void setRandom(final long seed) {
- QuantilesDoublesSketch.rand = new Random(seed);
- }
-
- /**
- * Returns a new builder
- * @return a new builder
- */
- public static final QuantilesDoublesSketchBuilder builder() {
- return new QuantilesDoublesSketchBuilder();
- }
-
- /**
- * Heapify takes the sketch image in MemorySegment and instantiates an on-heap Sketch.
- * The resulting sketch will not retain any link to the source MemorySegment.
- * @param srcSeg a MemorySegment image of a Sketch.
- * @return a heap-based Sketch based on the given MemorySegment
- */
- public static QuantilesDoublesSketch heapify(final MemorySegment srcSeg) {
- if (checkIsMemorySegmentCompact(srcSeg)) {
- return HeapCompactDoublesSketch.heapifyInstance(srcSeg);
- }
- return HeapUpdateDoublesSketch.heapifyInstance(srcSeg);
- }
-
- /**
- * Wrap this sketch around the given MemorySegment image of a compact, read-only QuantilesDoublesSketch.
- *
- * @param srcSeg the given MemorySegment image of a compact, read-only QuantilesDoublesSketch.
- * @return a compact, read-only sketch that wraps the given MemorySegment.
- */
- public static QuantilesDoublesSketch wrap(final MemorySegment srcSeg) {
- if (!checkIsMemorySegmentCompact(srcSeg)) {
- throw new SketchesArgumentException(
- "MemorySegment sketch image must be in compact form. "
- + "Use {@link #writableWrap(MemorySegment writableWrap(...)} for updatable sketches.");
- }
- return DirectCompactDoublesSketch.wrapInstance(srcSeg);
- }
-
- /**
- * Wrap this sketch around the given MemorySegment image of an updatable QuantilesDoublesSketch.
- *
- *
- *
- */
- public QuantilesDoublesSketchBuilder() {}
-
- /**
- * Sets the parameter k that determines the accuracy and size of the sketch.
- * @param k determines the accuracy and size of the sketch.
- * It is recommended that k be a power of 2 to enable unioning of sketches with
- * different k. It is only possible to union from
- * larger k to smaller k.
- * @return this builder
- */
- public QuantilesDoublesSketchBuilder setK(final int k) {
- ClassicUtil.checkK(k);
- bK = k;
- return this;
- }
-
- /**
- * Gets the current configured k
- * @return the current configured k
- */
- public int getK() {
- return bK;
- }
-
- /**
- * Returns an UpdatableQuantilesDoublesSketch with the current configuration of this Builder.
- * @return a UpdatableQuantilesDoublesSketch
- */
- public UpdatableQuantilesDoublesSketch build() {
- return HeapUpdateDoublesSketch.newInstance(bK);
- }
-
- /**
- * Returns a UpdatableQuantilesDoublesSketch with the current configuration of this builder
- * and the specified backing destination MemorySegment store that can grow.
- * @param dstSeg destination MemorySegment for use by the sketch
- * @return an UpdatableQuantilesDoublesSketch
- */
- public UpdatableQuantilesDoublesSketch build(final MemorySegment dstSeg) {
- return this.build(dstSeg, null);
- }
-
- /**
- * Returns a UpdatableQuantilesDoublesSketch with the current configuration of this builder
- * and the specified backing destination MemorySegment store that can grow.
- * @param dstSeg destination MemorySegment for use by the sketch
- * @param mSegReq the MemorySegmentRequest used if the given MemorySegment needs to expand.
- * Otherwise, it can be null and the default MemorySegmentRequest will be used.
- * @return an UpdatableQuantilesDoublesSketch
- */
- public UpdatableQuantilesDoublesSketch build(final MemorySegment dstSeg, final MemorySegmentRequest mSegReq) {
- return DirectUpdateDoublesSketch.newInstance(bK, dstSeg, mSegReq);
- }
-
- /**
- * Creates a human readable string that describes the current configuration of this builder.
- */
- @Override
- public String toString() {
- final StringBuilder sb = new StringBuilder();
- sb.append("QuantileSketchBuilder configuration:").append(LS);
- sb.append("K : ").append(TAB).append(bK).append(LS);
- return sb.toString();
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesSketchIterator.java b/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesSketchIterator.java
deleted file mode 100644
index 5b1d9bfbb..000000000
--- a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesSketchIterator.java
+++ /dev/null
@@ -1,79 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.quantiles;
-
-import java.util.Objects;
-
-import org.apache.datasketches.common.SketchesStateException;
-import org.apache.datasketches.quantilescommon.QuantilesDoublesSketchIteratorAPI;
-
-/**
- * Iterator over QuantilesDoublesSketch. The order is not defined.
- */
-public final class QuantilesDoublesSketchIterator implements QuantilesDoublesSketchIteratorAPI {
- private final DoublesSketchAccessor sketchAccessor;
- private long bitPattern;
- private int level;
- private long weight;
- private int index;
-
- QuantilesDoublesSketchIterator(final QuantilesDoublesSketch sketch, final long bitPattern) {
- Objects.requireNonNull(sketch, "sketch must not be null");
- sketchAccessor = DoublesSketchAccessor.wrap(sketch, false);
- this.bitPattern = bitPattern;
- level = -1;
- weight = 1;
- index = -1;
- }
-
- @Override
- public double getQuantile() {
- if (index < 0) { throw new SketchesStateException("index < 0; getQuantile() was called before next()"); }
- return sketchAccessor.get(index);
- }
-
- @Override
- public long getWeight() {
- return weight;
- }
-
- @Override
- public boolean next() {
- index++; // advance index within the current level
- if (index < sketchAccessor.numItems()) {
- return true;
- }
- // go to the next non-empty level
- do {
- level++;
- if (level > 0) {
- bitPattern >>>= 1;
- }
- if (bitPattern == 0L) {
- return false; // run out of levels
- }
- weight *= 2;
- } while ((bitPattern & 1L) == 0L);
- index = 0;
- sketchAccessor.setLevel(level);
- return true;
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnion.java b/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnion.java
deleted file mode 100644
index 707513c65..000000000
--- a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnion.java
+++ /dev/null
@@ -1,211 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.quantiles;
-
-import java.lang.foreign.MemorySegment;
-
-import org.apache.datasketches.common.MemorySegmentRequest;
-import org.apache.datasketches.common.MemorySegmentStatus;
-
-/**
- * The API for Union operations for QuantilesDoublesSketches
- *
- * @author Lee Rhodes
- */
-public abstract class QuantilesDoublesUnion implements MemorySegmentStatus {
-
- /**
- * No argument constructor.
- */
- public QuantilesDoublesUnion() { }
-
- /**
- * Returns a new UnionBuilder
- * @return a new UnionBuilder
- */
- public static QuantilesDoublesUnionBuilder builder() {
- return new QuantilesDoublesUnionBuilder();
- }
-
- /**
- * Returns a Heap Union object that has been initialized with the data from the given sketch.
- * @param sketch A QuantilesDoublesSketch to be used as a source of data only and will not be modified.
- * @return a QuantilesDoublesUnion object
- */
- public static QuantilesDoublesUnion heapify(final QuantilesDoublesSketch sketch) {
- return QuantilesDoublesUnionImpl.heapifyInstance(sketch);
- }
-
- /**
- * Returns a Heap Union object that has been initialized with the data from the given MemorySegment that contains an
- * image of a sketch.
- *
- * @param srcSeg A MemorySegment image of a QuantilesDoublesSketch to be used as a source of data and will not be modified.
- * @return a Union object
- */
- public static QuantilesDoublesUnion heapify(final MemorySegment srcSeg) {
- return QuantilesDoublesUnionImpl.heapifyInstance(srcSeg);
- }
-
- /**
- * Returns an updatable Union object that wraps the given MemorySegment that contains an image of a QuantilesDoublesSketch.
- *
- * @param srcSeg A MemorySegment image of an updatable QuantilesDoublesSketch to be used as the data structure for the union
- * and will be modified.
- * @return a Union object
- */
- public static QuantilesDoublesUnion wrap(final MemorySegment srcSeg) {
- return QuantilesDoublesUnionImpl.wrapInstance(srcSeg, null);
- }
-
- /**
- * Returns an updatable Union object that wraps the given MemorySegment that contains an image of a QuantilesDoublesSketch.
- *
- * @param srcSeg A MemorySegment sketch to be used as the data structure for the union and will be modified.
- * @param mSegReq the MemorySegmentRequest used if the given MemorySegment needs to expand.
- * Otherwise, it can be null and the default MemorySegmentRequest will be used.
- * @return a Union object
- */
- public static QuantilesDoublesUnion wrap(final MemorySegment srcSeg, final MemorySegmentRequest mSegReq) {
- return QuantilesDoublesUnionImpl.wrapInstance(srcSeg, mSegReq);
- }
-
- @Override
- public abstract boolean hasMemorySegment();
-
- @Override
- public abstract boolean isOffHeap();
-
- @Override
- public abstract boolean isSameResource(final MemorySegment that);
-
- /**
- * Returns true if this union is empty
- * @return true if this union is empty
- */
- public abstract boolean isEmpty();
-
- /**
- * Returns the configured maxK of this Union.
- * @return the configured maxK of this Union.
- */
- public abstract int getMaxK();
-
- /**
- * Returns the effective k of this Union.
- * @return the effective k of this Union.
- */
- public abstract int getEffectiveK();
-
- /**
- * Iterative union operation, which means this method can be repeatedly called.
- * Merges the given sketch into this union object.
- * The given sketch is not modified.
- * It is required that the ratio of the two K's be a power of 2.
- * This is easily satisfied if each of the K's are already a power of 2.
- * If the given sketch is null or empty it is ignored.
- *
- *
- *
- */
- public QuantilesDoublesUnionBuilder() {}
-
- /**
- * Sets the parameter masK that determines the maximum size of the sketch that
- * results from a union and its accuracy.
- * @param maxK determines the accuracy and size of the union and is a maximum.
- * The effective k can be smaller due to unions with smaller k sketches.
- * It is recommended that maxK be a power of 2 to enable unioning of sketches with
- * different k.
- * @return this builder
- */
- public QuantilesDoublesUnionBuilder setMaxK(final int maxK) {
- ClassicUtil.checkK(maxK);
- bMaxK = maxK;
- return this;
- }
-
- /**
- * Gets the current configured maxK
- * @return the current configured maxK
- */
- public int getMaxK() {
- return bMaxK;
- }
-
- /**
- * Returns a new empty Union object with the current configuration of this Builder.
- * @return a Union object
- */
- public QuantilesDoublesUnion build() {
- return QuantilesDoublesUnionImpl.heapInstance(bMaxK);
- }
-
- /**
- * Returns a new empty Union object with the current configuration of this Builder
- * and the specified backing destination MemorySegment store.
- * @param dstSeg the destination MemorySegment
- * @param mSegReq the MemorySegmentRequest used if the given MemorySegment needs to expand.
- * Otherwise, it can be null and the default MemorySegmentRequest will be used.
- * @return a Union object
- */
- public QuantilesDoublesUnion build(final MemorySegment dstSeg, final MemorySegmentRequest mSegReq) {
- return QuantilesDoublesUnionImpl.directInstance(bMaxK, dstSeg, mSegReq);
- }
-
-}
diff --git a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnionImpl.java b/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnionImpl.java
deleted file mode 100644
index 8cd4552a3..000000000
--- a/src/main/java/org/apache/datasketches/quantiles/QuantilesDoublesUnionImpl.java
+++ /dev/null
@@ -1,339 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.quantiles;
-
-import static org.apache.datasketches.common.Util.LS;
-import static org.apache.datasketches.quantiles.ClassicUtil.checkIsMemorySegmentCompact;
-import static org.apache.datasketches.quantiles.DoublesUtil.copyToHeap;
-
-import java.lang.foreign.MemorySegment;
-import java.util.Objects;
-
-import org.apache.datasketches.common.MemorySegmentRequest;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesReadOnlyException;
-
-/**
- * Union operation.
- *
- * @author Lee Rhodes
- * @author Kevin Lang
- */
-final class QuantilesDoublesUnionImpl extends QuantilesDoublesUnion {
- int maxK_;
- UpdatableQuantilesDoublesSketch gadget_ = null;
-
- private QuantilesDoublesUnionImpl(final int maxK) {
- maxK_ = maxK;
- }
-
- /**
- * Returns a empty QuantilesDoublesUnion object on the heap.
- * @param maxK determines the accuracy and size of the union and is a maximum.
- * The effective k can be smaller due to unions with smaller k sketches.
- * It is recommended that maxK be a power of 2 to enable unioning of sketches with
- * different k.
- * @return a new QuantilesDoublesUnionImpl on the Java heap
- */
- static QuantilesDoublesUnionImpl heapInstance(final int maxK) {
- return new QuantilesDoublesUnionImpl(maxK);
- }
-
- /**
- * Returns a empty QuantilesDoublesUnion object that uses the given MemorySegment for its internal sketch gadget
- * and will be initialized to the empty state.
- *
- * @param maxK determines the accuracy and size of the union and is a maximum.
- * The effective k can be smaller due to unions with smaller k sketches.
- * It is recommended that maxK be a power of 2 to enable unioning of sketches with
- * different k.
- * @param dstSeg the MemorySegment to be used by the internal sketch and must not be null.
- * @param mSegReq the MemorySegmentRequest used if the given MemorySegment needs to expand.
- * Otherwise, it can be null and the default MemorySegmentRequest will be used.
- * @return a QuantilesDoublesUnion object
- */
- static QuantilesDoublesUnionImpl directInstance(final int maxK, final MemorySegment dstSeg, final MemorySegmentRequest mSegReq) {
- Objects.requireNonNull(dstSeg);
- final DirectUpdateDoublesSketch sketch = DirectUpdateDoublesSketch.newInstance(maxK, dstSeg, mSegReq);
- final QuantilesDoublesUnionImpl union = new QuantilesDoublesUnionImpl(maxK);
- union.maxK_ = maxK;
- union.gadget_ = sketch;
- return union;
- }
-
- /**
- * Returns a Heap QuantilesDoublesUnion object that has been initialized with the data from the given
- * sketch.
- *
- * @param sketch A QuantilesDoublesSketch to be used as a source of data only and will not be modified.
- * @return a QuantilesDoublesUnion object
- */
- static QuantilesDoublesUnionImpl heapifyInstance(final QuantilesDoublesSketch sketch) {
- Objects.requireNonNull(sketch);
- final int k = sketch.getK();
- final QuantilesDoublesUnionImpl union = new QuantilesDoublesUnionImpl(k);
- union.maxK_ = k;
- union.gadget_ = copyToHeap(sketch);
- return union;
- }
-
- /**
- * Returns a Heap QuantilesDoublesUnion object that has been initialized with the data from the given
- * MemorySegment image of a QuantilesDoublesSketch. The srcSeg object will not be modified and a reference to
- * it is not retained. The maxK of the resulting union will be that obtained from
- * the sketch MemorySegment image.
- *
- * @param srcSeg an optionally read-only MemorySegment image of a QuantilesDoublesSketch
- * @return a QuantilesDoublesUnion object
- */
- static QuantilesDoublesUnionImpl heapifyInstance(final MemorySegment srcSeg) {
- Objects.requireNonNull(srcSeg);
- final HeapUpdateDoublesSketch sketch = HeapUpdateDoublesSketch.heapifyInstance(srcSeg);
- final QuantilesDoublesUnionImpl union = new QuantilesDoublesUnionImpl(sketch.getK());
- union.gadget_ = sketch;
- return union;
- }
-
- /**
- * Returns an Union object that wraps the data of the given MemorySegment image of a UpdatableQuantilesDoublesSketch.
- * The data of the Union will remain in the MemorySegment.
- *
- * @param srcSeg A MemorySegment image of an updatable QuantilesDoublesSketch to be used as the data structure for the union
- * and will be modified.
- * @param mSegReq the MemorySegmentRequest used if the given MemorySegment needs to expand.
- * Otherwise, it can be null and the default MemorySegmentRequest will be used.
- * @return a Union object
- */
- static QuantilesDoublesUnionImpl wrapInstance(final MemorySegment srcSeg, final MemorySegmentRequest mSegReq) {
- Objects.requireNonNull(srcSeg);
- if (srcSeg.isReadOnly()) { throw new SketchesReadOnlyException("Cannot create a Union with a Read Only MemorySegment."); }
- final DirectUpdateDoublesSketch sketch = DirectUpdateDoublesSketch.wrapInstance(srcSeg, mSegReq);
- final QuantilesDoublesUnionImpl union = new QuantilesDoublesUnionImpl(sketch.getK());
- union.gadget_ = sketch;
- return union;
- }
-
- @Override
- public void union(final QuantilesDoublesSketch sketchIn) {
- Objects.requireNonNull(sketchIn);
- gadget_ = updateLogic(maxK_, gadget_, sketchIn);
- gadget_.doublesSV = null;
- }
-
- @Override
- public void union(final MemorySegment seg) {
- Objects.requireNonNull(seg);
- if (checkIsMemorySegmentCompact(seg)) {
- gadget_ = updateLogic(maxK_, gadget_, QuantilesDoublesSketch.wrap(seg));
- } else {
- gadget_ = updateLogic(maxK_, gadget_, QuantilesDoublesSketch.writableWrap(seg, null));
- }
-
- gadget_.doublesSV = null;
- }
-
- @Override
- public void update(final double dataItem) {
- if (gadget_ == null) {
- gadget_ = HeapUpdateDoublesSketch.newInstance(maxK_);
- }
- gadget_.update(dataItem);
- gadget_.doublesSV = null;
- }
-
- @Override
- public byte[] toByteArray() {
- if (gadget_ == null) {
- return QuantilesDoublesSketch.builder().setK(maxK_).build().toByteArray();
- }
- return gadget_.toByteArray();
- }
-
- @Override
- public UpdatableQuantilesDoublesSketch getResult() {
- if (gadget_ == null) {
- return HeapUpdateDoublesSketch.newInstance(maxK_);
- }
- return DoublesUtil.copyToHeap(gadget_);
- }
-
- @Override
- public UpdatableQuantilesDoublesSketch getResult(final MemorySegment dstSeg, final MemorySegmentRequest mSegReq) {
- final long segCapBytes = dstSeg.byteSize();
- if (gadget_ == null) {
- if (segCapBytes < QuantilesDoublesSketch.getUpdatableStorageBytes(0, 0)) {
- throw new SketchesArgumentException("Insufficient capacity for result: " + segCapBytes);
- }
- return DirectUpdateDoublesSketch.newInstance(maxK_, dstSeg, mSegReq);
- }
-
- gadget_.putIntoMemorySegment(dstSeg, false);
- return DirectUpdateDoublesSketch.wrapInstance(dstSeg, mSegReq);
- }
-
- @Override
- public UpdatableQuantilesDoublesSketch getResultAndReset() {
- if (gadget_ == null) { return null; } //Intentionally return null here for speed.
- final UpdatableQuantilesDoublesSketch ds = gadget_.getSketchAndReset();
- gadget_ = null;
- return ds;
- }
-
- @Override
- public void reset() {
- gadget_.reset();
- }
-
- @Override
- public boolean hasMemorySegment() {
- return (gadget_ != null) && gadget_.hasMemorySegment();
- }
-
- @Override
- public boolean isOffHeap() {
- return (gadget_ != null) && gadget_.isOffHeap();
- }
-
- @Override
- public boolean isEmpty() {
- return (gadget_ == null) || gadget_.isEmpty();
- }
-
- @Override
- public boolean isSameResource(final MemorySegment that) {
- return (gadget_ == null) ? false : gadget_.isSameResource(that);
- }
-
- @Override
- public int getMaxK() {
- return maxK_;
- }
-
- @Override
- public int getEffectiveK() {
- return (gadget_ != null) ? gadget_.getK() : maxK_;
- }
-
- @Override
- public String toString() {
- return toString(true, false);
- }
-
- @Override
- public String toString(final boolean sketchSummary, final boolean dataDetail) {
- final StringBuilder sb = new StringBuilder();
- final String thisSimpleName = this.getClass().getSimpleName();
- final int maxK = getMaxK();
- final String kStr = String.format("%,d", maxK);
- sb.append(LS).append("### Quantiles ").append(thisSimpleName).append(LS);
- sb.append(" maxK : ").append(kStr);
- if (gadget_ == null) {
- sb.append(HeapUpdateDoublesSketch.newInstance(maxK_).toString());
- return sb.toString();
- }
- sb.append(gadget_.toString(sketchSummary, dataDetail));
- return sb.toString();
- }
-
- //@formatter:off
- static UpdatableQuantilesDoublesSketch updateLogic(final int myMaxK, final UpdatableQuantilesDoublesSketch myQS, final QuantilesDoublesSketch other) {
- int sw1 = ((myQS == null) ? 0 : myQS.isEmpty() ? 4 : 8);
- sw1 |= ((other == null) ? 0 : other.isEmpty() ? 1 : 2);
- int outCase = 0; //0=null, 1=NOOP, 2=copy, 3=merge
-
- switch (sw1) {
- case 0: outCase = 0; break; //myQS = null, other = null ; return null
- case 1: outCase = 4; break; //myQS = null, other = empty; create empty-heap(myMaxK)
- case 2: outCase = 2; break; //myQS = null, other = valid; stream or downsample to myMaxK
- case 4: outCase = 1; break; //myQS = empty, other = null ; no-op
- case 5: outCase = 1; break; //myQS = empty, other = empty; no-op
- case 6: outCase = 3; break; //myQS = empty, other = valid; merge
- case 8: outCase = 1; break; //myQS = valid, other = null ; no-op
- case 9: outCase = 1; break; //myQS = valid, other = empty: no-op
- case 10: outCase = 3; break; //myQS = valid, other = valid; merge
- default: break; //This cannot happen
- }
- UpdatableQuantilesDoublesSketch ret = null;
-
- switch (outCase) {
- case 0: break; //return null
- case 1: ret = myQS; break; //no-op
- case 2: { //myQS = null, other = valid; stream or downsample to myMaxK
- assert other != null;
- if (!other.isEstimationMode()) { //other is exact, stream items in
- ret = HeapUpdateDoublesSketch.newInstance(myMaxK);
- // exact mode, only need copy base buffer
- final DoublesSketchAccessor otherAccessor = DoublesSketchAccessor.wrap(other, false);
- for (int i = 0; i < otherAccessor.numItems(); ++i) {
- ret.update(otherAccessor.get(i));
- }
- }
- else { //myQS = null, other is est mode
- ret = (myMaxK < other.getK())
- ? other.downSampleInternal(other, myMaxK, null, null) //null seg, null mSegReq
- : DoublesUtil.copyToHeap(other); //copy required because caller has handle
- }
- break;
- }
- case 3: { //myQS = empty/valid, other = valid; merge
- assert other != null;
- assert myQS != null;
- if (!other.isEstimationMode()) { //other is exact, stream items in
- ret = myQS;
- // exact mode, only need copy base buffer
- final DoublesSketchAccessor otherAccessor = DoublesSketchAccessor.wrap(other, false);
- for (int i = 0; i < otherAccessor.numItems(); ++i) {
- ret.update(otherAccessor.get(i));
- }
- } else if (myQS.getK() <= other.getK()) { //I am smaller or equal, thus the target
- DoublesMergeImpl.mergeInto(other, myQS);
- ret = myQS;
- } else if (myQS.isEmpty()) {
- if (myQS.hasMemorySegment()) {
- final MemorySegment seg = myQS.getMemorySegment(); //myQS is empty, ok to reconfigure
- other.putIntoMemorySegment(seg, false); // not compact, but BaseBuf ordered
- ret = DirectUpdateDoublesSketch.wrapInstance(seg, null);
- } else { //myQS is empty and on heap
- ret = DoublesUtil.copyToHeap(other);
- }
- } else { //Not Empty: myQS has data, downsample to tmp
- final UpdatableQuantilesDoublesSketch tmp = QuantilesDoublesSketch.builder().setK(other.getK()).build();
-
- DoublesMergeImpl.downSamplingMergeInto(myQS, tmp); //myData -> tmp
- ret = (myQS.hasMemorySegment())
- ? QuantilesDoublesSketch.builder().setK(other.getK()).build(myQS.getMemorySegment())
- : QuantilesDoublesSketch.builder().setK(other.getK()).build();
-
- DoublesMergeImpl.mergeInto(tmp, ret);
- DoublesMergeImpl.mergeInto(other, ret);
- }
- break;
- }
- case 4: { //myQS = null, other = empty; create empty-heap(myMaxK)
- ret = HeapUpdateDoublesSketch.newInstance(myMaxK);
- break;
- }
- default: break; //This cannot happen
- }
- return ret;
- }
- //@formatter:on
-
-}
diff --git a/src/main/java/org/apache/datasketches/quantiles/QuantilesItemsSketch.java b/src/main/java/org/apache/datasketches/quantiles/QuantilesItemsSketch.java
deleted file mode 100644
index 17c94940d..000000000
--- a/src/main/java/org/apache/datasketches/quantiles/QuantilesItemsSketch.java
+++ /dev/null
@@ -1,726 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.datasketches.quantiles;
-
-import static java.lang.Math.max;
-import static java.lang.Math.min;
-import static java.lang.foreign.ValueLayout.JAVA_BYTE;
-import static org.apache.datasketches.quantiles.ClassicUtil.MIN_K;
-import static org.apache.datasketches.quantiles.ClassicUtil.checkFamilyID;
-import static org.apache.datasketches.quantiles.ClassicUtil.checkK;
-import static org.apache.datasketches.quantiles.ClassicUtil.checkPreLongsFlagsCap;
-import static org.apache.datasketches.quantiles.ClassicUtil.computeBaseBufferItems;
-import static org.apache.datasketches.quantiles.ClassicUtil.computeBitPattern;
-import static org.apache.datasketches.quantiles.ClassicUtil.computeCombinedBufferItemCapacity;
-import static org.apache.datasketches.quantiles.ClassicUtil.computeRetainedItems;
-import static org.apache.datasketches.quantiles.PreambleUtil.COMPACT_FLAG_MASK;
-import static org.apache.datasketches.quantiles.PreambleUtil.extractFamilyID;
-import static org.apache.datasketches.quantiles.PreambleUtil.extractFlags;
-import static org.apache.datasketches.quantiles.PreambleUtil.extractK;
-import static org.apache.datasketches.quantiles.PreambleUtil.extractN;
-import static org.apache.datasketches.quantiles.PreambleUtil.extractPreLongs;
-import static org.apache.datasketches.quantiles.PreambleUtil.extractSerVer;
-
-import java.lang.foreign.MemorySegment;
-import java.lang.reflect.Array;
-import java.util.Arrays;
-import java.util.Comparator;
-import java.util.Objects;
-import java.util.Random;
-
-import org.apache.datasketches.common.ArrayOfItemsSerDe;
-import org.apache.datasketches.common.SketchesArgumentException;
-import org.apache.datasketches.common.SketchesStateException;
-import org.apache.datasketches.quantilescommon.GenericPartitionBoundaries;
-import org.apache.datasketches.quantilescommon.ItemsSketchSortedView;
-import org.apache.datasketches.quantilescommon.QuantileSearchCriteria;
-import org.apache.datasketches.quantilescommon.QuantilesAPI;
-import org.apache.datasketches.quantilescommon.QuantilesGenericAPI;
-import org.apache.datasketches.quantilescommon.QuantilesGenericSketchIteratorAPI;
-
-/**
- * This is an implementation of the Low Discrepancy Mergeable Quantiles Sketch, using generic items,
- * described in section 3.2 of the journal version of the paper "Mergeable Summaries"
- * by Agarwal, Cormode, Huang, Phillips, Wei, and Yi:
- *
- *
- *
- *