Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/test-java.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,13 @@ jobs:
run: gradle --stop

- name: Run mvn test
run: mvn test --no-transfer-progress -q -B -pl SQL-compiler -Dsurefire.failIfNoSpecifiedTests=false
run: mvn test --no-transfer-progress -B -pl SQL-compiler -Dsurefire.failIfNoSpecifiedTests=false
working-directory: ./sql-to-dbsp-compiler
if: ${{ vars.CI_DRY_RUN != 'true' }}

- name: Run one quick SLT test
if: ${{ vars.CI_DRY_RUN != 'true' }}
run: mvn test --no-transfer-progress -q -B -Dsurefire.failIfNoSpecifiedTests=false -Dtest=RotateTests#quick
run: mvn test --no-transfer-progress -B -Dsurefire.failIfNoSpecifiedTests=false -Dtest=RotateTests#quick
working-directory: ./sql-to-dbsp-compiler

- name: Print sccache stats
Expand Down
2 changes: 1 addition & 1 deletion crates/adapters/src/integrated.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ where
}

/// Create an instance of an integrated output endpoint given its config
/// and output relation schema.p
/// and output relation schema.
#[allow(unused)]
pub fn create_integrated_output_endpoint(
endpoint_id: EndpointId,
Expand Down
2 changes: 1 addition & 1 deletion crates/sqllib/src/source.rs
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ impl fmt::Display for SourcePositionRange {

/// Maps "keys" to source line position information
#[doc(hidden)]
#[derive(Default)]
#[derive(Debug, Default)]
pub struct SourceMap {
map: BTreeMap<(&'static str, u32), SourcePosition>,
}
Expand Down
2 changes: 1 addition & 1 deletion docs.feldera.com/docs/connectors/sources/debezium.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ CREATE TABLE my_table (
)
```

::: warning
:::warning

Notice that the data format provided by the connector contains data
and metadata columns; for example, a metadata column is the "operation", which
Expand Down
2 changes: 1 addition & 1 deletion docs.feldera.com/docs/connectors/unique_keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ sources and data delivery to external sinks in two key ways:
from tables *without* primary keys. Deleting a non-existent record for
such tables will produce unexpected results or even pipeline crashes.

::: danger
:::danger

Deleting records that do not exist from a table without a primary key is
an undefined operation, and can produce incorrect results or crashes.
Expand Down
65 changes: 63 additions & 2 deletions docs.feldera.com/docs/sql/grammar.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ values
: { VALUES | VALUE } expression [, expression ]*

select
: SELECT [ ALL | DISTINCT ]
: SELECT [ hintComment ] [ ALL | DISTINCT ]
{ projectItem [, projectItem ]* }
FROM tableExpression
[ WHERE booleanExpression ]
Expand All @@ -360,7 +360,7 @@ select
```
tablePrimary
: tableName '(' TABLE tableName ')'
| tablePrimary '(' columnDecl [, columnDecl ]* ')'
| tablePrimary [ hintComment ] '(' columnDecl [, columnDecl ]* ')'
| [ LATERAL ] '(' query ')'
| UNNEST '(' expression ')' [ WITH ORDINALITY ]
| TABLE '(' functionName '(' expression [, expression ]* ')' ')'
Expand Down Expand Up @@ -442,6 +442,67 @@ the column names are *not* used to reorder columns.
In `orderItem`, if expression is a positive integer n, it denotes the
nth item in the `SELECT` clause.

## SQL hints

A hint is an instruction to the optimizer. When writing SQL, you may
know information about the data unknown to the optimizer. Hints
enable you to make decisions normally made by the optimizer.

We support hints in two locations:

- Query Hint: right after the `SELECT` keyword;
- Table Hint: right after the referenced table or view name.

```
SELECT /*+ broadcast(S), shard(T) */
FROM
T /*+ size(5) */
JOIN
S
```

The syntax of hints is:

```
hintComment
: '/*+' hint [, hint ]* '*/'

hint:
hintName
| hintName '(' optionKey '=' optionVal [, optionKey '=' optionVal ]* ')'
| hintName '(' hintOption [, hintOption ]* ')'

optionKey
: simpleIdentifier
| stringLiteral

optionVal
: simpleIdentifier
| stringLiteral

hintOption
: simpleIdentifier
| numericLiteral
| stringLiteral
```

### Supported hints and their impact on query implementation

:::warning

These hints are considered still experimental, and they may change

:::

- `broadcast(`*table*`)`: Indicates that the following `JOIN` should be implemented using
a broadcast-join strategy by broadcasting the input with alias *table*
- `shard(`*table*`)`: Indicates that the following `JOIN` should be implemented using
a hash-join strategy by sharding the input with alias *table*
- `balance(`*table*`)`: Indicates that the following `JOIN` should be implemented using
a balanced strategy by hashing on all fields the input with alias *table*

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc/code mismatch: this lists the hint as balanced(table), but the registered hint name is balanceHINT_BALANCE = "balance" in SqlToRelCompiler, and the new testThreeWayJoinHints writes /*+ ... balance(U) */. A user copying the docs verbatim with /*+ balanced(U) */ will get an unknown hint that silently no-ops. Either rename the constant to balanced or fix the docs to say balance(table).

Note: specifying hints may inhibit some compiler optimizations.

## Creating indexes

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section is empty other than "Support for hints is under development." Since the parser now accepts broadcast, shard, and balance without errors (they are registered in HintStrategyTable), users will discover them and try them. A one-line table listing the three names with "recognized but currently no-op" is more honest than the current placeholder and pre-empts the bug report "my hint did nothing."

Feldera supports the `CREATE INDEX` SQL statement with the following
Expand Down
2 changes: 1 addition & 1 deletion docs.feldera.com/docs/sql/identifiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ GROUP BY x+1
HAVING x+1 > 0;
```

::: danger
:::danger

In order to preserve the semantics of standard SQL programs unchanged,
a new column alias is used only if there is no column with the same
Expand Down
34 changes: 27 additions & 7 deletions js-packages/profiler-lib/src/profile.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ interface JsonMetricBase {
interface MetricValue { };

// Base types
interface PersistentIdMetricValue extends MetricValue {
interface StringMetricValue extends MetricValue {
type: "string";
value: string;
}
Expand Down Expand Up @@ -54,6 +54,11 @@ interface BytesMetricValue {
value: number;
}

interface BoolMetricValue {
type: "bool";
value: boolean;
}

interface PercentMetricValue {
type: "percent";
value: {
Expand Down Expand Up @@ -110,12 +115,13 @@ interface CacheStatsValue extends MetricValue {
}

type JsonMetricValue =
BoolMetricValue |
CountMetricValue |
BytesMetricValue |
PercentMetricValue |
DurationMetricValue |
StatsMetricValue |
PersistentIdMetricValue |
StringMetricValue |
DistributionValue |
RetainmentValue |
IntMetricValue |
Expand Down Expand Up @@ -375,6 +381,10 @@ class BooleanValue extends PropertyValue {
this.value = id;
}

static fromBoolMetric(value: BoolMetricValue): BooleanValue {
return new BooleanValue(value.value);
}

getNumericValue(): Option<number> {
return this.value ? Option.some(0) : Option.some(1);
}
Expand Down Expand Up @@ -721,12 +731,17 @@ export class Measurement {
let metric_id = metric.metric_id;
let kind = metric_id.split("_").pop(); // count, bytes, etc
if (metric.labels) {
let labels = metric.labels.map(e => e.join(":")).join(".");
if (metric.labels.length !== 0) {
metric_id = metric_id + "." + labels;
}
let labels = metric.labels.map(e => e.join(":")).join(".");
if (metric.labels.length !== 0) {
metric_id = metric_id + "." + labels;
}
}
switch (kind) {
case "bool": {
let m = metric.value as BoolMetricValue;
let value = BooleanValue.fromBoolMetric(m);
return [new Measurement(metric_id, Option.some(value))];
}
case "bytes": {
let m = metric.value as BytesMetricValue;
let bytes = BytesValue.fromBytesMetric(m);
Expand Down Expand Up @@ -767,7 +782,7 @@ export class Measurement {
];
}
case "id": { // persistent_id
let s = metric.value as PersistentIdMetricValue;
let s = metric.value as StringMetricValue;
let str = StringValue.fromString(s.value);
return [new Measurement(metric_id, Option.some(str))];
}
Expand Down Expand Up @@ -845,6 +860,11 @@ export class Measurement {
new Measurement(metric_id + ".steps", Option.some(steps)),
];
}
case "policy": {
let s = metric.value as StringMetricValue;
let str = StringValue.fromString(s.value);
return [new Measurement(metric_id, Option.some(str))];
}
}
return [];
}
Expand Down
12 changes: 6 additions & 6 deletions python/tests/runtime_aggtest/arithmetic_tests/test_date.py
Original file line number Diff line number Diff line change
Expand Up @@ -335,7 +335,7 @@ def __init__(self):
"ytm_str": "+10-01",
"dth_str": "+3683 00",
"dtm_str": "+3683 00:00",
"dts_str": "+3683 00:00:00.000000",
"dts_str": "+3683 00:00:00",
"htm_str": "+88392:00",
"hts_str": "+88392:00:00.000000",
"mts_str": "+5303520:00.000000",
Expand All @@ -345,7 +345,7 @@ def __init__(self):
"ytm_str": "-2-08",
"dth_str": "-980 00",
"dtm_str": "-980 00:00",
"dts_str": "-980 00:00:00.000000",
"dts_str": "-980 00:00:00",
"htm_str": "-23520:00",
"hts_str": "-23520:00:00.000000",
"mts_str": "-1411200:00.000000",
Expand All @@ -355,7 +355,7 @@ def __init__(self):
"ytm_str": "+20-06",
"dth_str": "+7506 00",
"dtm_str": "+7506 00:00",
"dts_str": "+7506 00:00:00.000000",
"dts_str": "+7506 00:00:00",
"htm_str": "+180144:00",
"hts_str": "+180144:00:00.000000",
"mts_str": "+10808640:00.000000",
Expand Down Expand Up @@ -646,7 +646,7 @@ def __init__(self):
"ytm_neg": "-10-01",
"dth_neg": "-3683 00",
"dtm_neg": "-3683 00:00",
"dts_neg": "-3683 00:00:00.000000",
"dts_neg": "-3683 00:00:00",
"htm_neg": "-88392:00",
"hts_neg": "-88392:00:00.000000",
"mts_neg": "-5303520:00.000000",
Expand All @@ -656,7 +656,7 @@ def __init__(self):
"ytm_neg": "+2-08",
"dth_neg": "+980 00",
"dtm_neg": "+980 00:00",
"dts_neg": "+980 00:00:00.000000",
"dts_neg": "+980 00:00:00",
"htm_neg": "+23520:00",
"hts_neg": "+23520:00:00.000000",
"mts_neg": "+1411200:00.000000",
Expand All @@ -666,7 +666,7 @@ def __init__(self):
"ytm_neg": "-20-06",
"dth_neg": "-7506 00",
"dtm_neg": "-7506 00:00",
"dts_neg": "-7506 00:00:00.000000",
"dts_neg": "-7506 00:00:00",
"htm_neg": "-180144:00",
"hts_neg": "-180144:00:00.000000",
"mts_neg": "-10808640:00.000000",
Expand Down
Loading