Skip to content

Commit 758f249

Browse files
rakeshkashyap123bobohu97rkashyap
authored
Add a new compute model to Feathr (#820)
* Add working gradle build * Set up pdl support * Working PDL java code gen * With pdl files from metadata models * With pdl files from compute model * Fix compile for all pdl files * Add working gradle build * Migrate frame-config module into feathr * Migrate fcm graph module to feathr * Add FCM offline execution code, includes FDS metadata code * Add needed jars for feathr-config tests * Switch client to FeathrClient2 for local tests and fix config errors * Fix SWA test * Add gradle wrapper jar * Change name of git PR test from sbt to gradle * Switch python client to use FCM client * Exclude json from dependency * Add hacky solution to handle json dependency conflict in cloud * Add json to local dependency * Add log to debug cloud jar * Add json as dependency * Another attempt at resolving json dependency * Resolve json via shading * Fix json shading * Remove log * Shade typesafe config for cloud jar * Add maven publish code to build.gradle * Add working local maven build and rename frame-config to feathr-config to avoid namespace conflict * Modify sonatype creds * Change so no need to sign if releasing snapshot version * Update build.gradle to allow publishing of all modules * Removed FDS handling from Feathr * All tests working * Deleted FR stuff * Remove dimension and other tensor related stuff * Remove mlfeatureversionurn from defaultvalueresolver * Remove mlfeatureversionurn and featureref * Remove featuredefinition files * Remove featureRef and typedRef * final cleanup * Fix merge conflict bugs * Fix guava error * udf plugin for swa features * row-transformations optimization * fix bug * fix another bug * always execute agg nodes first * Add SWA log * reverse order of execution * group by datasource * Fix bug * Merge main into fcm branch * Remove insecure URLs * Add back removed files * Add back removed files * Add back removed files * Change PR build system to gradle * Change sbt job to gradle jobb * Change sbt workflow:wq * Update maven github workflow to use gradle * fix failing test * remove sbt project module * Remove sbt related files * Change docs to reflect gradle * Remove keywords * Create a single jar * 1. Fix jar not getting populated\n 2. Fix documentation bugs * pubishToMavenLocal Working * With FFE integrated * maven upload working * Update docs and code clean up * add gradle-wrapper file * Push all dependency jars * Update docs * Docs cleanup * Update github workflow commands * Update github workflow * Update workflow syntax * Update version * Add gradle version to github workflow * Update gradle version w/o quotes * Remove github gradle version * Github workflow fix * Github workflow fix-2 * Github workflow fix-4 Co-authored-by: Bozhong Hu <bobhu@linkedin.com> Co-authored-by: rkashyap <rkashyap@linkedin.com>
1 parent 654d56e commit 758f249

1,059 files changed

Lines changed: 31643 additions & 619 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitattributes‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#
2+
# https://help.github.com/articles/dealing-with-line-endings/
3+
#
4+
# These are explicitly windows files and should use crlf
5+
*.bat text eol=crlf
6+

‎.github/workflows/docker-publish.yml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This workflow builds the docker container and publishes to dockerhub with appropriate tag
2-
# It has two triggers,
2+
# It has two triggers,
33
# 1. daily i.e. runs everyday at specific time.
44
# 2. Anytime a new branch is created under releases
55

@@ -22,19 +22,19 @@ jobs:
2222
steps:
2323
- name: Check out the repo
2424
uses: actions/checkout@v3
25-
25+
2626
- name: Log in to Docker Hub
2727
uses: docker/login-action@v2
2828
with:
2929
username: ${{ secrets.DOCKER_USERNAME }}
3030
password: ${{ secrets.DOCKER_PASSWORD }}
31-
31+
3232
- name: Extract metadata (tags, labels) for Docker
3333
id: meta
3434
uses: docker/metadata-action@v4
3535
with:
3636
images: feathrfeaturestore/feathr-registry
37-
37+
3838
- name: Build and push Docker image
3939
uses: docker/build-push-action@v3
4040
with:
@@ -72,4 +72,4 @@ jobs:
7272
id: deploy-to-feathr-registry-sql-rbac
7373
uses: distributhor/workflow-webhook@v3.0.1
7474
env:
75-
webhook_url: ${{ secrets.AZURE_WEBAPP_FEATHR_REGISTRY_SQL_RBAC_WEBHOOK }}
75+
webhook_url: ${{ secrets.AZURE_WEBAPP_FEATHR_REGISTRY_SQL_RBAC_WEBHOOK }}

‎.github/workflows/publish-to-maven.yml‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,18 @@
11
name: Publish package to the Maven Central Repository
2-
on:
2+
on:
33
push:
44
# This pipeline will get triggered everytime there is a new tag created.
5-
# It is required
5+
# It is required
66
tags: ["*"]
77

88
jobs:
99
publish-to-maven:
1010
runs-on: ubuntu-latest
11-
11+
1212
steps:
1313
- name: Checkout source
1414
uses: actions/checkout@v2
15-
15+
1616
# Setting up JDK 8, this is required to build Feathr
1717
- name: Set up JDK 8
1818
uses: actions/setup-java@v2
@@ -27,10 +27,9 @@ jobs:
2727

2828
# CI release command defaults to publishSigned
2929
# Sonatype release command defaults to sonaTypeBundleRelease
30-
# https://github.com/sbt/sbt-ci-release
31-
- name: Sbt ci release
32-
run: |
33-
sbt ci-release
30+
- name: Gradle publish
31+
if: startsWith(github.head_ref, 'release/v')
32+
run: gradle clean publish
3433
env:
3534
PGP_PASSPHRASE: ${{ secrets.MAVEN_GPG_PASSPHRASE }}
3635
PGP_SECRET: ${{ secrets.MAVEN_GPG_PRIVATE_KEY }}

‎.github/workflows/pull_request_push_test.yml‎

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ on:
2828
- cron: '00 13 * * *'
2929

3030
jobs:
31-
sbt_test:
31+
gradle_test:
3232
runs-on: ubuntu-latest
3333
if: github.event_name == 'schedule' || github.event_name == 'push' || github.event_name == 'pull_request' || (github.event_name == 'pull_request_target' && contains(github.event.pull_request.labels.*.name, 'safe to test'))
3434
steps:
@@ -41,7 +41,7 @@ jobs:
4141
java-version: "8"
4242
distribution: "temurin"
4343
- name: Run tests
44-
run: sbt clean && sbt test
44+
run: ./gradlew clean && ./gradlew test
4545

4646
python_lint:
4747
runs-on: ubuntu-latest
@@ -75,15 +75,15 @@ jobs:
7575
with:
7676
java-version: "8"
7777
distribution: "temurin"
78-
- name: Build JAR
78+
- name: Gradle build
7979
run: |
80-
sbt assembly
80+
./gradlew build
8181
# remote folder for CI upload
8282
echo "CI_SPARK_REMOTE_JAR_FOLDER=feathr_jar_github_action_$(date +"%H_%M_%S")" >> $GITHUB_ENV
8383
# get local jar name without paths so version change won't affect it
84-
echo "FEATHR_LOCAL_JAR_NAME=$(ls target/scala-2.12/*.jar| xargs -n 1 basename)" >> $GITHUB_ENV
84+
echo "FEATHR_LOCAL_JAR_NAME=$(ls build/libs/*.jar| xargs -n 1 basename)" >> $GITHUB_ENV
8585
# get local jar name without path
86-
echo "FEATHR_LOCAL_JAR_FULL_NAME_PATH=$(ls target/scala-2.12/*.jar)" >> $GITHUB_ENV
86+
echo "FEATHR_LOCAL_JAR_FULL_NAME_PATH=$(ls build/libs/*.jar)" >> $GITHUB_ENV
8787
- name: Set up Python 3.8
8888
uses: actions/setup-python@v2
8989
with:
@@ -142,15 +142,16 @@ jobs:
142142
with:
143143
java-version: "8"
144144
distribution: "temurin"
145-
- name: Build JAR
145+
146+
- name: Gradle build
146147
run: |
147-
sbt assembly
148+
./gradlew build
148149
# remote folder for CI upload
149150
echo "CI_SPARK_REMOTE_JAR_FOLDER=feathr_jar_github_action_$(date +"%H_%M_%S")" >> $GITHUB_ENV
150151
# get local jar name without paths so version change won't affect it
151-
echo "FEATHR_LOCAL_JAR_NAME=$(ls target/scala-2.12/*.jar| xargs -n 1 basename)" >> $GITHUB_ENV
152+
echo "FEATHR_LOCAL_JAR_NAME=$(ls build/libs/*.jar| xargs -n 1 basename)" >> $GITHUB_ENV
152153
# get local jar name without path
153-
echo "FEATHR_LOCAL_JAR_FULL_NAME_PATH=$(ls target/scala-2.12/*.jar)" >> $GITHUB_ENV
154+
echo "FEATHR_LOCAL_JAR_FULL_NAME_PATH=$(ls build/libs/*.jar)" >> $GITHUB_ENV
154155
- name: Set up Python 3.8
155156
uses: actions/setup-python@v2
156157
with:
@@ -210,15 +211,16 @@ jobs:
210211
with:
211212
java-version: "8"
212213
distribution: "temurin"
213-
- name: Build JAR
214+
215+
- name: Gradle build
214216
run: |
215-
sbt assembly
217+
./gradlew build
216218
# remote folder for CI upload
217219
echo "CI_SPARK_REMOTE_JAR_FOLDER=feathr_jar_github_action_$(date +"%H_%M_%S")" >> $GITHUB_ENV
218220
# get local jar name without paths so version change won't affect it
219-
echo "FEATHR_LOCAL_JAR_NAME=$(ls target/scala-2.12/*.jar| xargs -n 1 basename)" >> $GITHUB_ENV
221+
echo "FEATHR_LOCAL_JAR_NAME=$(ls build/libs/*.jar| xargs -n 1 basename)" >> $GITHUB_ENV
220222
# get local jar name without path
221-
echo "FEATHR_LOCAL_JAR_FULL_NAME_PATH=$(ls target/scala-2.12/*.jar)" >> $GITHUB_ENV
223+
echo "FEATHR_LOCAL_JAR_FULL_NAME_PATH=$(ls build/libs/*.jar)" >> $GITHUB_ENV
222224
- name: Set up Python 3.8
223225
uses: actions/setup-python@v2
224226
with:
@@ -258,7 +260,7 @@ jobs:
258260
259261
failure_notification:
260262
# If any failure, warning message will be sent
261-
needs: [sbt_test, python_lint, databricks_test, azure_synapse_test, local_spark_test]
263+
needs: [gradle_test, python_lint, databricks_test, azure_synapse_test, local_spark_test]
262264
runs-on: ubuntu-latest
263265
if: failure() && github.event_name == 'schedule'
264266
steps:
@@ -268,12 +270,12 @@ jobs:
268270
269271
notification:
270272
# Final Daily Report with all job status
271-
needs: [sbt_test, python_lint, databricks_test, azure_synapse_test, local_spark_test]
273+
needs: [gradle_test, python_lint, databricks_test, azure_synapse_test, local_spark_test]
272274
runs-on: ubuntu-latest
273275
if: always() && github.event_name == 'schedule'
274276
steps:
275277
- name: Get Date
276278
run: echo "NOW=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
277279
- name: Notification
278280
run: |
279-
curl -H 'Content-Type: application/json' -d '{"text": "${{env.NOW}} Daily Report: 1. SBT Test ${{needs.sbt_test.result}}, 2. Python Lint Test ${{needs.python_lint.result}}, 3. Databricks Test ${{needs.databricks_test.result}}, 4. Synapse Test ${{needs.azure_synapse_test.result}} , 5. LOCAL SPARK TEST ${{needs.local_spark_test.result}}. Link: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"}' ${{ secrets.TEAMS_WEBHOOK }}
281+
curl -H 'Content-Type: application/json' -d '{"text": "${{env.NOW}} Daily Report: 1. Gradle Test ${{needs.gradle_test.result}}, 2. Python Lint Test ${{needs.python_lint.result}}, 3. Databricks Test ${{needs.databricks_test.result}}, 4. Synapse Test ${{needs.azure_synapse_test.result}} , 5. LOCAL SPARK TEST ${{needs.local_spark_test.result}}. Link: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"}' ${{ secrets.TEAMS_WEBHOOK }}

‎.gitignore‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
.AppleDouble
44
.LSOverride
55
metastore_db
6-
src/integTest
6+
feathr-impl/src/integTest
77
test-output
88
temp
99

@@ -189,30 +189,37 @@ cython_debug/
189189
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
190190
hs_err_pid*
191191

192-
target/
193192
.idea
194193
.project/target
195194
.project/project
196195
.DS_store
197-
.DS_Store
198196
*.jar
199-
src/main/scala/META-INF/MANIFEST.MF
197+
feathr-impl/src/main/scala/META-INF/MANIFEST.MF
200198
*.MF
201199
feathr_project/feathr_cli.egg-info/*
202200
*.pyc
201+
*.iml
203202

204203
# VS Code
205204
.vscode
206205

207206
#Local Build
208207
null/*
209208

209+
# Ignore Gradle project-specific cache directory
210+
.gradle
211+
212+
# Ignore Gradle build output directory
213+
build
214+
210215
# For Metal Server
211216
.metals/
212217
.bloop/
213218
project/.bloop
214219
metals.sbt
220+
215221
.bsp/sbt.json
216222

217223
# Feathr output debug folder
218224
**/debug/
225+

‎.husky/pre-commit‎

100755100644
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
#!/usr/bin/env sh
22
. "$(dirname -- "$0")/_/husky.sh"
33

4-
npx lint-staged
4+
npx lint-staged

0 commit comments

Comments
 (0)