Skip to content

Commit aafcd87

Browse files
authored
add firestore online format (#1367)
Signed-off-by: Oleg Avdeev <oleg.v.avdeev@gmail.com>
1 parent 525aa15 commit aafcd87

File tree

3 files changed

+58
-0
lines changed

3 files changed

+58
-0
lines changed
64.5 KB
Loading

docs/specs/online_store_format.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,34 @@ Here's an example of how the entire thing looks like:
5959

6060
However, we'll address this issue in future versions of the protocol.
6161

62+
## Cloud Firestore Online Store Format
63+
64+
[Firebase data model](https://firebase.google.com/docs/firestore/data-model) is a hierarchy of documents that can contain (sub)-collections. This structure can be multiple levels deep; documents and subcollections are alternating in this hierarchy.
65+
66+
We use the following structure to store feature data in the Firestore:
67+
* at the first level, there is a collection for each Feast project
68+
* second level, in each project-collection, there is a Firebase document for each Feature Table
69+
* third level, in the document for the Feature Table, there is a subcollection called `values` that contain a document per feature row. That document contains the following fields:
70+
* `key` contains entity key as serialized `feast.types.EntityKey` proto
71+
* `value` contains value as serialized `feast.types.Value` proto
72+
* `event_ts` contains event timestamp (in the native firestore timestamp format)
73+
* `created_ts` contains write timestamp (in the native firestore timestamp format)
74+
75+
Document id for the feature document is computed by hashing entity key using murmurhash3_128 algorithm as follows:
76+
77+
1. hash utf8-encoded entity names, sorted in alphanumeric order
78+
2. hash the entity values in the same order as corresponding entity names, by serializing them to bytes as follows:
79+
- binary values are hashed as-is
80+
- string values hashed after serializing them as utf8 string
81+
- int64 and int32 hashed as little-endian byte representation (8 and 4 bytes respectively)
82+
- bool hashed as 0 or 1 byte
83+
84+
Other types of entity keys are not supported in this version of the specification, when using Cloud Firestore.
85+
86+
**Example:**
87+
88+
![Firestore Online Example](firebase_online_example.png)
89+
6290
# Appendix
6391

6492
##### Appendix A. Value proto format.

protos/feast/types/EntityKey.proto

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
/*
2+
* Copyright 2018 The Feast Authors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* https://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
syntax = "proto3";
18+
19+
import "feast/types/Value.proto";
20+
21+
package feast.types;
22+
23+
option java_package = "feast.proto.types";
24+
option java_outer_classname = "EntityKeyProto";
25+
option go_package = "github.com/feast-dev/feast/sdk/go/protos/feast/types";
26+
27+
message EntityKey {
28+
repeated string entity_names = 1;
29+
repeated feast.types.Value entity_values = 2;
30+
}

0 commit comments

Comments
 (0)