|
| 1 | +# Feast - Feature Store for Machine Learning |
| 2 | + |
| 3 | +Feast is an open source feature store for machine learning that helps ML platform teams manage features consistently for training and serving. |
| 4 | + |
| 5 | +## Project Overview |
| 6 | + |
| 7 | +Feast (Feature Store) is a Python-based project that provides: |
| 8 | +- **Offline Store**: Process historical data for batch scoring or model training |
| 9 | +- **Online Store**: Power real-time predictions with low-latency features |
| 10 | +- **Feature Server**: Serve pre-computed features online |
| 11 | +- **Point-in-time correctness**: Prevent data leakage during model training |
| 12 | +- **Data infrastructure abstraction**: Decouple ML from data infrastructure |
| 13 | + |
| 14 | +## Development Commands |
| 15 | + |
| 16 | +### Setup |
| 17 | +```bash |
| 18 | +# Install development dependencies |
| 19 | +make install-python-dependencies-dev |
| 20 | + |
| 21 | +# Install minimal dependencies |
| 22 | +make install-python-dependencies-minimal |
| 23 | +``` |
| 24 | + |
| 25 | +### Code Quality |
| 26 | +```bash |
| 27 | +# Format Python code |
| 28 | +make format-python |
| 29 | + |
| 30 | +# Lint Python code |
| 31 | +make lint-python |
| 32 | + |
| 33 | +# Type check |
| 34 | +cd sdk/python && python -m mypy feast |
| 35 | +``` |
| 36 | + |
| 37 | +### Testing |
| 38 | +```bash |
| 39 | +# Run unit tests |
| 40 | +make test-python-unit |
| 41 | + |
| 42 | +# Run integration tests (local) |
| 43 | +make test-python-integration-local |
| 44 | + |
| 45 | +# Run integration tests (CI) |
| 46 | +make test-python-integration |
| 47 | + |
| 48 | +# Run all Python tests |
| 49 | +make test-python-universal |
| 50 | +``` |
| 51 | + |
| 52 | +### Protobuf Compilation |
| 53 | +```bash |
| 54 | +# Compile Python protobuf files |
| 55 | +make compile-protos-python |
| 56 | + |
| 57 | +# Compile all protos |
| 58 | +make protos |
| 59 | +``` |
| 60 | + |
| 61 | +### Go Development |
| 62 | +```bash |
| 63 | +# Build Go code |
| 64 | +make build-go |
| 65 | + |
| 66 | +# Test Go code |
| 67 | +make test-go |
| 68 | + |
| 69 | +# Format Go code |
| 70 | +make format-go |
| 71 | + |
| 72 | +# Lint Go code |
| 73 | +make lint-go |
| 74 | +``` |
| 75 | + |
| 76 | +### Docker |
| 77 | +```bash |
| 78 | +# Build all Docker images |
| 79 | +make build-docker |
| 80 | + |
| 81 | +# Build feature server Docker image |
| 82 | +make build-feature-server-docker |
| 83 | +``` |
| 84 | + |
| 85 | +### Documentation |
| 86 | +```bash |
| 87 | +# Build Sphinx documentation |
| 88 | +make build-sphinx |
| 89 | + |
| 90 | +# Build templates |
| 91 | +make build-templates |
| 92 | + |
| 93 | +# Build Helm docs |
| 94 | +make build-helm-docs |
| 95 | +``` |
| 96 | + |
| 97 | +## Project Structure |
| 98 | + |
| 99 | +``` |
| 100 | +feast/ |
| 101 | +├── sdk/python/ # Python SDK and core implementation |
| 102 | +├── go/ # Go implementation |
| 103 | +├── ui/ # Web UI |
| 104 | +├── docs/ # Documentation |
| 105 | +├── examples/ # Example projects |
| 106 | +├── infra/ # Infrastructure and deployment |
| 107 | +│ ├── charts/ # Helm charts |
| 108 | +│ └── feast-operator/ # Kubernetes operator |
| 109 | +└── protos/ # Protocol buffer definitions |
| 110 | +``` |
| 111 | + |
| 112 | +## Key Technologies |
| 113 | + |
| 114 | +- **Languages**: Python (primary), Go |
| 115 | +- **Dependencies**: pandas, pyarrow, SQLAlchemy, FastAPI, protobuf |
| 116 | +- **Data Sources**: BigQuery, Snowflake, Redshift, Parquet, Postgres, Spark |
| 117 | +- **Online Stores**: Redis, DynamoDB, Bigtable, Snowflake, SQLite, Postgres |
| 118 | +- **Offline Stores**: BigQuery, Snowflake, Redshift, Spark, Dask, DuckDB |
| 119 | +- **Cloud Providers**: AWS, GCP, Azure |
| 120 | + |
| 121 | +## Common Development Tasks |
| 122 | + |
| 123 | +### Running Tests |
| 124 | +The project uses pytest for Python testing with extensive integration test suites for different data sources and stores. |
| 125 | + |
| 126 | +### Code Style |
| 127 | +- Uses `ruff` for Python linting and formatting |
| 128 | +- Go uses standard `gofmt` |
| 129 | + |
| 130 | +### Protobuf Development |
| 131 | +Protocol buffers are used for data serialization and gRPC APIs. Recompile protos after making changes to `.proto` files. |
| 132 | + |
| 133 | +### Multi-language Support |
| 134 | +Feast supports Python and Go SDKs. Changes to core functionality may require updates across both languages. |
| 135 | + |
| 136 | +## Contributing |
| 137 | + |
| 138 | +1. Follow the [contribution guide](docs/project/contributing.md) |
| 139 | +2. Set up your development environment |
| 140 | +3. Run relevant tests before submitting PRs |
| 141 | +4. Ensure code passes linting and type checking |
0 commit comments