Skip to content

feat: Add column reordering to write_to_offline_store #2876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 11 commits into from
Jun 30, 2022
Merged

feat: Add column reordering to write_to_offline_store #2876

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
0a152ee
Add feature extraction logic to batch writer
felixwang9817 Jun 23, 2022
0ce6433
Enable StreamProcessor to write to both online and offline stores
felixwang9817 Jun 27, 2022
a47b39b
Fix incorrect columns error message
felixwang9817 Jun 28, 2022
791322b
Reorder columns in _write_to_offline_store
felixwang9817 Jun 28, 2022
9970032
Make _write_to_offline_store a public method
felixwang9817 Jun 28, 2022
d1ce38f
Import FeatureStore correctly
felixwang9817 Jun 29, 2022
5ff276c
Remove defaults for `processing_time` and `query_timeout`
felixwang9817 Jun 29, 2022
838363a
Clean up `test_offline_write.py`
felixwang9817 Jun 29, 2022
b090a1d
Do not do any custom logic for double underscore columns
felixwang9817 Jun 29, 2022
b75883a
Lint
felixwang9817 Jun 29, 2022
e54ea6e
Switch entity values for all tests using push sources to not affect o…
felixwang9817 Jun 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Import FeatureStore correctly 
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
  • Loading branch information
@felixwang9817
felixwang9817 committed Jun 30, 2022
commit d1ce38fa94c87a2b8b69cbd84c404c2e59632b64
8 changes: 3 additions & 5 deletions sdk/python/feast/infra/contrib/spark_kafka_processor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from types import MethodType
from typing import TYPE_CHECKING, List, Optional
from typing import List, Optional

import pandas as pd
from pyspark.sql import DataFrame, SparkSession
Expand All @@ -8,16 +8,14 @@

from feast.data_format import AvroFormat, JsonFormat
from feast.data_source import KafkaSource, PushMode
from feast.feature_store import FeatureStore
from feast.infra.contrib.stream_processor import (
ProcessorConfig,
StreamProcessor,
StreamTable,
)
from feast.stream_feature_view import StreamFeatureView

if TYPE_CHECKING:
from feast.feature_store import FeatureStore


class SparkProcessorConfig(ProcessorConfig):
spark_session: SparkSession
Expand All @@ -34,7 +32,7 @@ class SparkKafkaProcessor(StreamProcessor):
def __init__(
self,
*,
fs: "FeatureStore",
fs: FeatureStore,
sfv: StreamFeatureView,
config: ProcessorConfig,
preprocess_fn: Optional[MethodType] = None,
Expand Down