|
| 1 | +# Development workflow |
| 2 | + |
| 3 | +## Develop from Devcontainer |
| 4 | + |
| 5 | +A simple way to start interacting with the Feldera source code is via the supplied Devcontainer. It will prepare an environment which will let you build and run all components of Feldera out-of-the-box. |
| 6 | + |
| 7 | +Spin up the Devcontainer, e.g. by opening the Feldera repository in VS Code with [Dev Containers](vscode:extension/ms-vscode-remote.remote-containers) extension enabled. |
| 8 | + |
| 9 | +On your first launch you might need to change the ownership of the files: |
| 10 | +```bash |
| 11 | +sudo chown -R user /workspaces/dbsp |
| 12 | +``` |
| 13 | + |
| 14 | +Build the SQL Compiler: |
| 15 | + |
| 16 | +```bash |
| 17 | +mvn -f ./sql-to-dbsp-compiler/SQL-compiler -DskipTests package |
| 18 | +``` |
| 19 | + |
| 20 | +Build and start the Pipeline Manager (that also serves the Feldera Web Console): |
| 21 | + |
| 22 | +```bash |
| 23 | +RUST_LOG=info RUST_BACKTRACE=1 cargo run --bin dbsp_pipeline_manager --features pg-embed -- --manager-working-directory ~/.dbsp -d postgres-embed --dev-mode --bind-address 0.0.0.0 --sql-compiler-home ./sql-to-dbsp-compiler --dbsp-override-path . --compiler-working-directory ~/.dbsp |
| 24 | +``` |
| 25 | + |
| 26 | +> Here, `~/.dbsp` is the directory that will host compilation artifacts of SQL Compiler that Pipeline Manager will then use. It will be created if it doesn't exist, and you can use another name for it - just replace corresponding arguments in the above command. |
| 27 | +
|
| 28 | +> `--dbsp-override-path .` should be the path of the Feldera repository root - so update the argument if you are running from a different directory. |
| 29 | +
|
| 30 | +You should now be able to access the Web Console at http://localhost:8080/, connected to your local Pipeline Manager instance! |
| 31 | + |
| 32 | +You can also open Web Console in dev mode to be able to see your changes to it live: |
| 33 | + |
| 34 | +```bash |
| 35 | +cd web-ui && yarn install && yarn dev |
| 36 | +``` |
| 37 | + |
| 38 | +The Web Console in dev mode is available at http://localhost:3000/ |
| 39 | + |
| 40 | +Now you can proceed with the [demo](#manually-starting-the-demos). |
| 41 | + |
| 42 | +## Develop on your machine |
| 43 | + |
| 44 | +TODO |
| 45 | + |
| 46 | +## Launch the prepared demo |
| 47 | + |
| 48 | +Refer to the (Get Started page)[../intro] for basic instructions on spinning up and interacting with the demos from a separate docker-compose. |
| 49 | + |
| 50 | +## Manually starting the demos |
| 51 | + |
| 52 | +You can prepare and run multiple demos. To prepare a demo, launch Pipeline Manager and navigate to the directory of the demo, e.g. |
| 53 | +```bash |
| 54 | +cd demo/project_demo00-SecOps |
| 55 | +``` |
| 56 | + |
| 57 | +If the `simulator` directory exists - the following command generates simulated input data and creates the required Kafka input and output topics: |
| 58 | +```bash |
| 59 | +cargo run --manifest-path simulator/Cargo.toml --release -- 300000 |
| 60 | +``` |
| 61 | + |
| 62 | +> For SecOps demo, argument `300000` specifies the number of simulated CI pipelines to be generated, which impacts duration of the demo and system load. YMMV! |
| 63 | +
|
| 64 | +Use `dbsp` python library to create and compile SQL Program, and prepare the Pipeline that utilizes them. |
| 65 | +```bash |
| 66 | +python3 run.py --dbsp_url http://localhost:8080 --actions prepare |
| 67 | +``` |
| 68 | + |
| 69 | +> You should update `--dbsp_url` depending on where Pipeline Manager is getting served from. |
| 70 | +
|
| 71 | +Now you can see the prepared demos on the Pipeline Management page of the Web Console. |
0 commit comments