Skip to content

Add request response logging via fluentd - #961

Merged
feast-ci-bot merged 11 commits into
feast-dev:masterfrom
terryyylim:rr-logging
Sep 8, 2020
Merged

Add request response logging via fluentd#961
feast-ci-bot merged 11 commits into
feast-dev:masterfrom
terryyylim:rr-logging

Conversation

@terryyylim

Copy link
Copy Markdown
Member

What this PR does / why we need it:
This PR introduces the option to enable request/response logging to an external fluentd service. This is an addition to the existing Audit logging capabilities which logs to the console.

Which issue(s) this PR fixes:

N/A

Does this PR introduce a user-facing change?:

Users can now enable forwarding of logs to an external fluentd service that's been setup.

@terryyylim

Copy link
Copy Markdown
Member Author

/retest

1 similar comment
@terryyylim

Copy link
Copy Markdown
Member Author

/retest

@terryyylim terryyylim changed the title [WIP] Add request response logging via fluentd Add request response logging via fluentd Aug 26, 2020
Comment thread common/pom.xml Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to pull this into all modules? Can this be moved to serving only?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AuditLogger is used in both Core and Serving, and it exists in common module, so I think that's not possible.

Comment thread common/src/main/java/feast/common/logging/AuditLogger.java Outdated
Comment thread common/src/main/java/feast/common/logging/AuditLogger.java Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request and response should be in Json, just like the existing audit logger. Otherwise its very hard to parse in BQ.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

Comment thread common/src/main/java/feast/common/logging/config/LoggingProperties.java Outdated
@pyalex

pyalex commented Aug 27, 2020

Copy link
Copy Markdown
Collaborator

/test test-end-to-end-batch-dataflow

@terryyylim
terryyylim force-pushed the rr-logging branch 5 times, most recently from 6d00ace to f85f746 Compare August 27, 2020 09:08
@terryyylim

Copy link
Copy Markdown
Member Author

/retest

1 similar comment
@terryyylim

Copy link
Copy Markdown
Member Author

/retest

@terryyylim
terryyylim force-pushed the rr-logging branch 2 times, most recently from 8f0ed5b to 1e6e835 Compare August 31, 2020 07:07
Comment thread common/src/main/java/feast/common/logging/AuditLogger.java Outdated
Comment thread common/src/main/java/feast/common/logging/AuditLogger.java Outdated
log.error(AUDIT_MARKER, entryJSON);
break;
// Either forward log to logging layer or log to console
if (properties.getMessageLogging().getDestination().equals(FLUENTD_DESTINATION)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What prevents this code from executing if message Logging is disabled?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's handled here from #940.

@woop woop Sep 8, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, as a convention we try and initialize variables from getters in order to make code more readable. So something like

String destination = properties.getMessageLogging().getDestination();
if (destination.equals(FLUENTD_DESTINATION)) {

}

@terryyylim

Copy link
Copy Markdown
Member Author

/retest

2 similar comments
@terryyylim

Copy link
Copy Markdown
Member Author

/retest

@terryyylim

Copy link
Copy Markdown
Member Author

/retest

@feast-ci-bot

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: terryyylim, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@woop

woop commented Sep 8, 2020

Copy link
Copy Markdown
Member

/lgtm

@feast-ci-bot
feast-ci-bot merged commit dc860f3 into feast-dev:master Sep 8, 2020
@mrzzy mrzzy added the compat/breaking Breaking user-facing change label Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved compat/breaking Breaking user-facing change kind/feature New feature or request lgtm size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants