Skip to content

Add deadletter/read-from-source metrics to batch and stream ingestion - #1223

Merged
feast-ci-bot merged 6 commits into
feast-dev:masterfrom
terryyylim:add-deadletter-metrics
Dec 24, 2020
Merged

Add deadletter/read-from-source metrics to batch and stream ingestion#1223
feast-ci-bot merged 6 commits into
feast-dev:masterfrom
terryyylim:add-deadletter-metrics

Conversation

@terryyylim

@terryyylim terryyylim commented Dec 9, 2020

Copy link
Copy Markdown
Member

Signed-off-by: Terence terencelimxp@gmail.com

What this PR does / why we need it:
There are several new metrics (reported by ingestion job) added:

  • read from source
  • wrote to deadletters

Also few bugs fixed:

  1. Counter reset after sending to statsd.
  2. Generating correct timestamp for test data

Some existing metrics were renamed to unify naming across all metrics.

Which issue(s) this PR fixes:

Fixes #

Does this PR introduce a user-facing change?:

Users can now view deadletter/read-from-source metrics in their configured monitoring system.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we convert this lambda into method of DeadLetterMetrics

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to iterate over all rows instead of just increment metric once by iter.length?

Comment thread spark/ingestion/src/main/scala/feast/ingestion/StreamingPipeline.scala Outdated
@pyalex

pyalex commented Dec 10, 2020

Copy link
Copy Markdown
Collaborator

@terryyylim please add some testing

terryyylim and others added 4 commits December 23, 2020 16:50
Signed-off-by: Terence <terencelimxp@gmail.com>
Signed-off-by: Terence <terencelimxp@gmail.com>
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex
pyalex force-pushed the add-deadletter-metrics branch from 58ebc52 to 372bd69 Compare December 24, 2020 06:24
@pyalex pyalex changed the title Add deadletter metrics to batch and stream ingestion Add deadletter/read-from-source metrics to batch and stream ingestion Dec 24, 2020
@pyalex

pyalex commented Dec 24, 2020

Copy link
Copy Markdown
Collaborator

/test test-end-to-end-gcp

Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@pyalex

pyalex commented Dec 24, 2020

Copy link
Copy Markdown
Collaborator

/test test-end-to-end

Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
@feast-ci-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pyalex, terryyylim

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@pyalex

pyalex commented Dec 24, 2020

Copy link
Copy Markdown
Collaborator

/lgtm

@feast-ci-bot
feast-ci-bot merged commit 4dbdb76 into feast-dev:master Dec 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants