This repository was archived by the owner on Dec 3, 2019. It is now read-only.
Repository navigation
This repository was archived by the owner on Dec 3, 2019. It is now read-only.
Make sentry monitoring better #383
Description
Feature
Why is this feature being added?
Sentry is adding a bunch of value to our monitoring, we wish to increase this value in the future.
What should your feature do?
- Configure to include correct build environment (staging, prod)
- Tie the build id of travis, github and sentry so we can relate build issues to sentry issues.
Send logs to chatops for repeated messages, but without spamming them.- Identify routes that require elevated monitoring or de-escalated monitoring. (some routes require every error, some require an increased number of errors).
Activity
kylemh commentedon Aug 22, 2018
That's not really how Raven works though. It essentially alerts us to when users encounter an error in their browser. The input errors aren't browser errors.
apex-omontgomery commentedon Aug 22, 2018
Some of the errors above are due to unknown coding/ infra issues that were only found when a user did some action, from looking at the docs you can modify the config for raven in all sorts of ways.
https://docs.sentry.io/clients/ruby/config/
Currently we only have the default config:
operationcode_backend/config/environments/production.rb
Line 95 in a3027d7
There are some exceptions we don't currently log, but an example that could be helpful would be to capture specific routes we are concerned with and pipe them to slack: https://docs.sentry.io/clients/ruby/usage/#reporting-messages
kylemh commentedon Aug 22, 2018
Right. I'm saying that Raven is client-side only. There's no server-side implementation of Sentry that I'm aware of. #chat-ops is only Raven dumps.
kylemh commentedon Aug 22, 2018
After chatting a bit more, this does seem like an issue. Clearly some services are reporting via Raven. Perhaps it's an aspect of an e2e implementation w/ Raven.
All aside, most of Raven's errors in #chatops seem very cryptic and unhelpful.
apex-omontgomery commentedon Aug 22, 2018
Yeah, I think that we could really benefit from identifying routes that are very important, including context and breadcrumbs and ensuring that these items get properly logged.
apex-omontgomery commentedon Aug 27, 2018
This has been corrected.
Will close the ticket once we document the fix and corrective actions somewhere and what we expect sentry to do.
hollomancer commentedon Aug 27, 2018
The cause of this was twofold;
hollomancer commentedon Aug 27, 2018
@nellshamrell and leads, please add your expectations for logging in #chatops to this thread.
apex-omontgomery commentedon Aug 31, 2018
After playing with sentry here's what I think.
I think of sentry as monitoring for application behavior, so we shouldn't be using it for monitoring CI/CD.
apex-omontgomery commentedon Oct 1, 2018
Okay so sentry is actually working well. I think now with the addition of staging, we need to prioritize targetting environment. Currently the
api.staging....is showing up on sentry as production.[-]Bug?: Sentry isn't alerting to things it should[/-][+]Feature: Make sentry monitoring better[/+][-]Feature: Make sentry monitoring better[/-][+]Make sentry monitoring better[/+]