There are several situations where the compiler + runtime could optimize operations on append-only tables:
We need a mechanism by which the compiler can request the DBSP layer to use these operators. One solution is to build custom operators for this case.
Related to the above, but separate, the compiler can use the fact that max grows monotonically to GC relations that get filtered by this max, e.g., select * from my_table where ts > select(max(...)).
There are several situations where the compiler + runtime could optimize operations on append-only tables:
min,max,argmax,argminaggregates can be evaluated by storing the current max only.joinwith foreign keysWe need a mechanism by which the compiler can request the DBSP layer to use these operators. One solution is to build custom operators for this case.
Related to the above, but separate, the compiler can use the fact that
maxgrows monotonically to GC relations that get filtered by thismax, e.g.,select * from my_table where ts > select(max(...)).