-
Notifications
You must be signed in to change notification settings - Fork 111
Expand file tree
/
Copy pathlib.rs
More file actions
119 lines (109 loc) · 4.5 KB
/
lib.rs
File metadata and controls
119 lines (109 loc) · 4.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
//! The `dbsp` crate implements a computational engine for continuous analysis
//! of changing data. With DBSP, a programmer writes code in terms of
//! computations on a complete data set, but DBSP implements it incrementally,
//! meaning that changes to the data set run in time proportional to the size of
//! the change rather than the size of the data set. This is a major advantage
//! for applications that work with large data sets that change frequently in
//! small ways.
//!
//! The [`tutorial`] is a good place to start for a guided tour. After that, if
//! you want to look through the API on your own, the [`circuit`] module is a
//! reasonable starting point. For complete examples, visit the [`examples`][1]
//! directory in the DBSP repository.
//!
//! [1]: https://github.com/feldera/feldera/tree/main/crates/dbsp/examples
//!
//! # Theory
//!
//! DBSP is underpinned by a formal theory:
//!
//! - [Budiu, Chajed, McSherry, Ryzhyk, Tannen. DBSP: Automatic Incremental View
//! Maintenance for Rich Query Languages, Conference on Very Large Databases, August
//! 2023, Vancouver, Canada](https://www.feldera.com/vldb23.pdf)
//!
//! - Here is [a presentation about DBSP](https://www.youtube.com/watch?v=iT4k5DCnvPU)
//! at the 2023 Apache Calcite Meetup.
//!
//! The model provides two things:
//!
//! 1. **Semantics.** DBSP defines a formal language of streaming operators and
//! queries built out of these operators, and precisely specifies how these
//! queries must transform input streams to output streams.
//!
//! 2. **Algorithm.** DBSP also gives an algorithm that takes an arbitrary query
//! and generates an incremental dataflow program that implements this query
//! correctly (in accordance with its formal semantics) and efficiently. Efficiency
//! here means, in a nutshell, that the cost of processing a set of
//! input events is proportional to the size of the input rather than the entire
//! state of the database.
//!
//! # Crate overview
//!
//! This crate consists of several layers.
//!
//! * [`dynamic`] - Types and traits that support dynamic dispatch. We heavily rely on
//! dynamic dispatch to limit the amount of monomorphization performed by the compiler when
//! building complex dataflow graphs, balancing compilation speed and runtime performance.
//! This module implements the type machinery necessary to support this architecture.
//!
//! * [`typed_batch`] - Strongly type wrappers around dynamically typed batches and traces.
//!
//! * [`trace`] - This module implements batches and traces, which are core DBSP data structures
//! that represent tables, indexes and changes to tables and indexes. We provide both in-memory
//! and persistent batch and trace implementations.
//!
//! * [`operator::dynamic`] - Dynamically typed operator API. Operators transform data streams
//! (usually carrying data in the form of batches and traces). DBSP provides many relational
//! operators, such as map, filter, aggregate, join, etc. The operator API in this module is
//! dynamically typed and unsafe.
//!
//! * [`operator`] - Statically typed wrappers around the dynamic API in [`operator::dynamic`].
#![allow(clippy::type_complexity)]
pub mod dynamic;
mod error;
mod hash;
mod num_entries;
//mod ref_pair;
pub mod typed_batch;
#[macro_use]
pub mod circuit;
pub mod algebra;
pub mod ir;
pub mod mimalloc;
pub mod monitor;
pub mod operator;
pub mod profile;
pub mod storage;
pub mod time;
pub mod trace;
pub mod utils;
#[cfg(feature = "backend-mode")]
pub mod mono;
pub use crate::{
error::{DetailedError, Error},
hash::default_hash,
num_entries::NumEntries,
};
// // pub use crate::ref_pair::RefPair;
pub use crate::time::Timestamp;
pub use algebra::{DynZWeight, ZWeight};
pub use circuit::{
ChildCircuit, Circuit, CircuitHandle, DBSPHandle, NestedCircuit, RootCircuit, Runtime,
RuntimeError, SchedulerError, Stream, WeakRuntime,
};
#[cfg(not(feature = "backend-mode"))]
pub use operator::FilterMap;
pub use operator::{
input::{IndexedZSetHandle, InputHandle, MapHandle, SetHandle, ZSetHandle},
CmpFunc, OrdPartitionedIndexedZSet, OutputHandle,
};
pub use trace::{cursor::Position, DBData, DBWeight};
pub use typed_batch::{
Batch, BatchReader, FallbackKeyBatch, FallbackValBatch, FallbackWSet, FallbackZSet,
FileIndexedWSet, FileIndexedZSet, FileKeyBatch, FileValBatch, FileWSet, FileZSet, IndexedZSet,
OrdIndexedWSet, OrdIndexedZSet, OrdWSet, OrdZSet, Trace, TypedBox, ZSet,
};
#[cfg(doc)]
pub mod tutorial;
// TODO: import from `circuit`.
pub type Scope = u16;