ExceptionTelemetry with large / deep stack traces are silently omitted

When using the Logback appender ApplicationInsightsAppender and so indirectly the components of the applicationinsights-core artifact from
com.microsoft.azure:applicationinsights-logging-logback:1.0.9
com.microsoft.azure:applicationinsights-core:1.0.9
where is a problem with log statements that include an exception with deep / long stack traces (ExceptionTelemetry).

Deep stack traces are being serialized as event JSON object correctly which has been verified by debugging the applicationinsights-core code.
But when the resulting JSON of the parse stack (shown in Insights Analytics as "details.parsedStack" and stored in Java class ExceptionTelemetry.ExceptionData.ExceptionDetails.parsedStack) exceeds as certain size (which seems to be 32k according to https://github.com/Microsoft/ApplicationInsights-Home/blob/master/EndpointSpecs/Schemas/Docs/ExceptionData.md), the complete log statement is omitted and not shown in Azure Insights Analytics.

I'm not 100% sure if the ExceptionTelemetry is omitted by applicationinsights-core or by Azure Insights instance itself.
But everything indicates that Azure Insights is receiving the JSON and omitting it remotely.

Omitting an ExceptionTelemetry silently is a very problematic behaviour from a development and operations point of view.
The artifact applicationinsights-core or Application Insights itself should know and respect the Azure Insigts limits and provide a robust behaviour:
If stack traces exceed the given limits, at least the log statement without the deep "parsed stack trace" should be stored and shown in Analytics.
Otherwise application exceptions will never be recognized which is a no-go for a monitoring platform.

@LarsGielsok, I see your point here and it is very valid. Here is my understanding:

Application Insights Java SDK - used to run on old schema definition which had lower limits of accepting the stack trace details apparently 1024 character until SDK v1.0.10 where we updated to new schema which supports 32k.
From the SDK v1.0.10 when the exception message exceeds the size of 32k then we truncate it to 32k and still report it to the portal.

Have you got a chance to try the Application Insights SDK v1.0.10?

Hi, Your understanding is correct. Unfortunately, 1.0.10 is Not working with logback. There is already an issue for that. Am 08.12.2017 19:02 schrieb "Dhaval Doshi" <notifications@github.com>: @LarsGielsok <https://github.com/larsgielsok>, I see your point here and it is very valid. Here is my understanding: 1. Application Insights Java SDK - used to run on old schema definition which had lower limits of accepting the stack trace details apparently 1024 character until SDK v1.0.10 where we updated to new schema which supports 32k. 2. From the SDK v1.0.10 when the exception message exceeds the size of 32k then we truncate it to 32k and still report it to the portal. Have you got a chance to try the Application Insights SDK v1.0.10? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#482 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALjMiP2M0ne9UJocHrf_AhHrl9FLNyomks5s-Xk6gaJpZM4Q7XID> .

Yes there was an issue with sanitization and setting the severity level whose changes are already merged in master. So hopefully that should fix this issue also.

when do you plan to release this fix? Is there any workaround in the meanwhile?

@LarsGielsok we plan to release the new version by end of next week most probably if things go right. In the mean while I would be very happy if you would have a chance to build the SDK locally and try and test it out. In this way we would come to know if any issues still persist (however we haven't noticed it on our test applications). Here are the steps to build the SDK locally :

Fork the repository.
Run the following gradle commands from the command prompt the location
gradlew clean, gradlew assemble, gradlew test and gradlew build.
You would need JDK 1.7 / 1.8 and Gradle 3.5 to build this project.
core, web and agent jars would be found in the respective directories of the project.

Please give it a shot if you have some time. Also I would like to point you to #468 and #469 for our future SDK release plan and just in case you are interested to complete brief survey.

@LarsGielsok were you able to try building the SDK locally? The Master branch is now most upto-date and you can build master directly.

Hi Dhaval, using 1.0.9 as well as the current master (1.0.11), the logback appender generally works and pushes traces to Azure Insights (in 1.0.10 this was broken as you know). But unfortunately, with the current master, exceptions with large / deep stack traces are still omitted. Exception with short strack traces are shown in Insights. So pushing logs to ExceptionTelemetry in general works. Regards, Lars 2017-12-13 19:20 GMT+01:00 Dhaval Doshi <notifications@github.com>:

…

@LarsGielsok <https://github.com/larsgielsok> were you able to try building the SDK locally? The Master branch is now most upto-date and you can build master directly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#482 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALjMiK-WzuakU51DOhO-uV-_vSHhysEfks5tABVhgaJpZM4Q7XID> .

@LarsGielsok oh, I see. We might have to dig more into this. We plan to release the SDK 2.0.0-BETA soon so I think it would be hard to lock down this particular issue into the upcoming release. I would investigate this further and update the thread. Hopefully we can get it down in 2.0.1 patch release. (Thank you very much for trying). In the mean while if you find some sort of possible solution which might be lying inside ExceptionTelemetry.java class feel free to send out a PR. Note that max size supported by the end point is 32k. We prune if the size exceeds the max size.

@LarsGielsok there is a PR out there #507 which will fix this issue. Feel free to have a look if you wish to. The max size of exception depth we would allow is 10 (To be in sync with our other SDKs). Also the max size of the parsed stack which would be allowed is 32768 chars.

Hi Dhaval, I tested your PR. Beside a failing jUnit test I could run and use it. But unfortunately the fix does not really work. I guess the size calculation is not 100% correct. At least the implementation of getStackFrameLength seems to be a little bit too basic. I think, the ExceptionTelemetry and its StackFrame elements are serialized as Json. So one might have to check the size of the serialized Json. Just for quick'n'dirty testing, I changed the method as following to get the size of the serialized Json (I know, this is not performant for productive usage) private static int getStackFrameLength(StackFrame stackFrame) { StringWriter sw = new StringWriter(); try { JsonTelemetryDataSerializer s = new JsonTelemetryDataSerializer(sw); stackFrame.serialize(s); } catch (Exception e) { e.printStackTrace(); } return sw.toString().length(); } But this change is also not sufficient: Large StackTraces are still not published to Insights. So, I guess the 32k limit covers not only the size of serialized StackFrames but other structures as well (whole ExceptionTelemetry?). Or there is different limit. Therefore, I reduced the MAX_PARSED_STACK_LENGTH peu a peu. When using the limit 20768, ExceptionTelemetries with large stacktraces are successfully published to Insights. So, as a summary, it is definitely a size limit issue but the current implementation seems to implement the size calculation not correctly. Best Regards Lars 2018-01-03 18:49 GMT+01:00 Dhaval Doshi <notifications@github.com>:

…

@LarsGielsok <https://github.com/larsgielsok> there is a PR out there #507 <#507> which will fix this issue. Feel free to have a look if you wish to. The max size of exception depth we would allow is 10 (To be in sync with our other SDKs). Also the max size of the parsed stack which would be allowed is 32768 chars. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#482 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALjMiFatC5F036oA9XcF2nUAF-beDuiNks5tG72bgaJpZM4Q7XID> .

@LarsGielsok thank you very much for this constructive feedback and taking time to try out the PR. It's much valuable and that's why the PR is not yet merged. Yes I agree with you that the size restriction might be for the entire Packet that gets sent to AI endpoint. Also it might be the case that the implementation of getStackFrameSize() might be too basic. Do you have any better suggestion on how we can efficiently compute the size of stackFrame() {i.e what all parameters to count for etc.} . Also the idea behind this was that if the size exceeds truncate with a self exception indicating truncation.

Further, clearly your experiment by reducing the size to 20k shows that issue is pertaining to the size of the item. If you have any better approach on how to do this, it would be great. Looking forward to hear.

Hi guys,

I think one of the approaches to mitigate this issue could be a parameterization of maximum exception's graph length and trace depth (per exception).

We could add two parameters in TelemetryConfiguration (also in XML and SB config): exceptionDepth and traceDepth.
ExceptionTelemetry.convertExceptionTree method is already able to cut the exceptions graph's length.
Adding capability of cutting trace length should be relatively easy to implement.

If you are ok with this approach, I could provide the PR.

Also, I've found that RuntimeExceptionDecider (from module "agent") can use stackTrace property (which is actually treated as a cap on exceptions graph length). It seems that this property can be configured via "stackSize" element. Maybe it's worth keeping this property name for back compatibility reasons even though it doesn't reflect actual use case.

vlev

vlev that seems very interesting approach. We always appreciate any contributions. I would suggest that you create another issue, and elaborate your apporach a bit more there. You can then create a PR for it. Your contribution would be greatly appreciated.

Dhaval Doshi (@dhaval24)

Is there any news on this issue Dhaval Doshi (@dhaval24) we just recently found out that not all exceptions are logged to application insights. That was very confusing for us. Simple reproduction is just trying to log stackOverflow exception e.g compare following:

private void doInfiniteLoop() {
    doInfiniteLoop();
}

public void logStackOverflow() {
    try {
        doInfiniteLoop();
    } catch (StackOverflowError ex) {
        log.info("Logging StackOverflow");
        log.error("StackOverflow", ex);
    }
}

public void logSimple() {
    try {
        throw new Exception("Oh Dear!");
    } catch (Exception ex) {
        log.info("Logging Simple");
        log.error("Simple", ex);
    }
}

Marcin Bazydlo (@gondar)

Marcin Bazydlo (@gondar) I have responded on the vlev 's PR. Arthur Little (@littleaj) / Trask Stalnaker (@trask) might have more context and can help you more : Let me know if the processor example provided there works for you.

Work around exceptions being silently omitted in Application Insights…

Navigation Menu

Repository navigation

ExceptionTelemetry with large / deep stack traces are silently omitted #482

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ExceptionTelemetry with large / deep stack traces are silently omitted #482

Description

Activity

dhaval24 commented on Dec 8, 2017

larsgx commented on Dec 8, 2017

dhaval24 commented on Dec 8, 2017

larsgx commented on Dec 9, 2017

dhaval24 commented on Dec 9, 2017

dhaval24 commented on Dec 13, 2017

larsgx commented on Dec 13, 2017

dhaval24 commented on Dec 14, 2017

dhaval24 commented on Jan 3, 2018

larsgx commented on Jan 11, 2018

dhaval24 commented on Jan 11, 2018

2 remaining items

vlev commented on Jan 8, 2019

vlev commented on Jan 8, 2019

dhaval24 commented on Jan 17, 2019

gondar commented on May 6, 2019

dhaval24 commented on May 6, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions