{"id":17645,"date":"2025-12-01T08:00:42","date_gmt":"2025-12-01T16:00:42","guid":{"rendered":"https:\/\/slack.engineering\/?p=17645"},"modified":"2025-11-26T16:19:50","modified_gmt":"2025-11-27T00:19:50","slug":"streamlining-security-investigations-with-agents","status":"publish","type":"post","link":"https:\/\/slack.engineering\/streamlining-security-investigations-with-agents\/","title":{"rendered":"Streamlining Security Investigations with Agents"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">Slack\u2019s Security Engineering team is responsible for protecting Slack\u2019s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We\u2019re going to show you how we\u2019re using AI agents to optimize our working efficiency and strengthen Slack\u2019s security defenses. This post is the first in a series that will unpack some of the design choices we\u2019ve made and the many things we\u2019ve learnt along the way.<\/span><\/p>\n<h2>The Development Process<\/h2>\n<h3>The Prototype<\/h3>\n<p><span style=\"font-weight: 400;\">At the end of May 2025 we had a rudimentary prototype of what would grow into our service. Initially, the service was not much more than a 300 word prompt.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The prompt consisted of five sections:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><b>Orientation<\/b><span style=\"font-weight: 400;\">: \u201cYou are a security analyst that investigates security alerts [&#8230;]\u201d<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Manifest<\/b><span style=\"font-weight: 400;\">: \u201cYou have access to the following data sources: [&#8230;]\u201d<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Methodology<\/b><span style=\"font-weight: 400;\">: \u201cYour investigation should follow these steps: [&#8230;] \u201d<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Formatting<\/b><span style=\"font-weight: 400;\">: \u201cProduce a markdown report of the investigation: [&#8230;]\u201d<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Classification<\/b><span style=\"font-weight: 400;\">: \u201cChoose a response classification from: [&#8230;]\u201d<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">We implemented a simple <\/span><a href=\"https:\/\/modelcontextprotocol.io\/docs\/learn\/architecture#transport-layer\"><span style=\"font-weight: 400;\">\u201cstdio\u201d mode MCP server<\/span><\/a><span style=\"font-weight: 400;\"> to safely expose a subset of our data sources through the tool call interface. We repurposed a coding agent CLI as an execution environment for our prototype.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The performance of our prototype implementation was highly variable: sometimes it would produce excellent, insightful results with an impressive ability to cross-reference evidence across different data sources. However, sometimes it would quickly jump to a convenient or spurious conclusion without adequately questioning its own methods. For the tool to be useful, we needed consistent performance. We needed greater control over the investigation process.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We spent some time trying to refine our prompt, stressing the need to question assumptions, to verify data from multiple sources, and to make use of the complete set of data sources. While we did have some success with this approach, ultimately prompts are just guidelines; they\u2019re not an effective method for achieving fine-grained control.<\/span><\/p>\n<h3>The Solution<\/h3>\n<p><span style=\"font-weight: 400;\">Our solution was to break down the complex investigation process we\u2019d described in the prompt of our prototype into a sequence of model invocations, each with a single, well-defined purpose and output structure. These simple tasks are chained together by our application.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Each task was given a structured output format. Structured output is a feature that can be used to restrict a model to using a specific output format defined by a JSON schema. The schema is applied to the last output from the model invocation. Using structured outputs isn\u2019t \u201cfree\u201d; if the output format is too complicated for the model, the execution can fail. Structured outputs are also subject to the usual problems of cheating and hallucination.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In our initial prototype, we included guidance to \u201cquestion your evidence\u201d, but had mixed success. With our structured output approach, that guidance had become a separate task in our investigation flow with much more predictable behavior.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">This approach gave us more precise control at each step of the investigation process.<\/span><\/p>\n<h2><span style=\"font-weight: 400;\">From Prototype to Production<\/span><\/h2>\n<p><span style=\"font-weight: 400;\">While reviewing the literature, two papers particularly influenced our thinking:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><a href=\"https:\/\/arxiv.org\/pdf\/2401.12954\"><span style=\"font-weight: 400;\">Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding<\/span><\/a><span style=\"font-weight: 400;\"> (Stanford, OpenAI)<\/span><\/li>\n<li style=\"font-weight: 400;\"><a href=\"https:\/\/arxiv.org\/pdf\/2307.05300\"><span style=\"font-weight: 400;\">Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration<\/span><\/a><span style=\"font-weight: 400;\"> (Microsoft Research)<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">These papers describe prompting techniques that introduce multiple personas <\/span><b>in the context of a single model invocation<\/b><span style=\"font-weight: 400;\">. The idea of modelling the investigation using defined personas was intriguing, but in order to maintain control we needed to represent our personas as independent model invocations. Security tabletop exercises, and how we might adapt their conventions to our application, were also a major source of inspiration during the design process.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Our chosen design is built around a team of personas (agents) and the tasks they can perform in the investigation process. Each agent\/task pair is modelled with a carefully defined structured output, and our application orchestrates the model invocations, propagating just the right context at each stage.<\/span><\/p>\n<h3>Investigation Loop<\/h3>\n<figure id=\"attachment_17657\" aria-describedby=\"caption-attachment-17657\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-17657 size-large\" src=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?w=1020\" alt=\"Flow diagram illustrating how agents cooperate during security investigations\" width=\"1020\" height=\"1173\" srcset=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png 2202w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=640,736 640w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=768,883 768w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=1280,1472 1280w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=1336,1536 1336w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=1781,2048 1781w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=380,437 380w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=800,920 800w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_round_simple_vertical_ad2b54.png?resize=1160,1334 1160w\" sizes=\"auto, (max-width: 1020px) 100vw, 1020px\" \/><figcaption id=\"caption-attachment-17657\" class=\"wp-caption-text\">The Director agent poses a question and domain expert agents respond, generating findings. The Critic agent reviews findings for quality and assembles a timeline using the most credible. The Director uses the high-quality findings and timeline to determine how to progress the investigation.<\/figcaption><\/figure>\n<p><span style=\"font-weight: 400;\">Our design has three defined persona categories:<\/span><\/p>\n<p><strong>Director Agent<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">The Investigation Director. The Director\u2019s responsibility is to progress the investigation from start to finish. The Director interrogates the experts by forming a question, or set of questions, which become the expert\u2019s prompt. The Director uses a journaling tool for planning and organizing the investigation as it progresses.<\/span><\/p>\n<p><b>Expert Agent<\/b><\/p>\n<p><span style=\"font-weight: 400;\">A domain expert. Each domain expert has a unique set of domain knowledge and data sources. The experts\u2019 responsibility is to produce findings from their data sources in response to the Director\u2019s questions.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We currently have four experts in our team:<\/span><\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li style=\"font-weight: 400;\"><b>Access<\/b><span style=\"font-weight: 400;\">: Authentication, authorization and perimeter services.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Cloud<\/b><span style=\"font-weight: 400;\">: Infrastructure, compute, orchestration, and networking.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Code<\/b><span style=\"font-weight: 400;\">: Analysis of source code and configuration management.<\/span><\/li>\n<li style=\"font-weight: 400;\"><b>Threat<\/b><span style=\"font-weight: 400;\">: Threat analysis and intelligence data sources.<\/span><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><b>Critic Agent<\/b><\/p>\n<p><span style=\"font-weight: 400;\">The Critic is a \u201cmeta-expert\u201d. The Critic\u2019s responsibility is to assess and quantify the quality of findings made by domain experts using a rubric we\u2019ve defined. The Critic annotates the experts\u2019 findings with its own analysis and a credibility score for each finding. The Critic\u2019s conclusions are passed back to the Director, closing the loop. The weakly adversarial relationship between the Critic and the expert group helps to mitigate against hallucinations and variability in the interpretation of evidence.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Because each agent\/task pair is a separate model invocation we can vary all of the inputs, including the model version, output format, prompts, instructions, and tools. One of many ways we\u2019re using this capability is to create a \u201cknowledge pyramid\u201d.<\/span><\/p>\n<h3>Knowledge Pyramid<\/h3>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-17647 size-large\" src=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?w=1020\" alt=\"Pyramid diagram illustrating how investigation knowledge flows up from low to high cost models.\" width=\"1020\" height=\"786\" srcset=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png 3200w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=640,493 640w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=768,592 768w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=1280,987 1280w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=1536,1184 1536w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=2048,1579 2048w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=380,293 380w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=800,617 800w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina.png?resize=1160,894 1160w\" sizes=\"auto, (max-width: 1020px) 100vw, 1020px\" \/><\/p>\n<p><span style=\"font-weight: 400;\">At the bottom of the knowledge pyramid, domain experts generate investigation findings by interrogating complex data sources, requiring many tool calls. Analyzing the returned data can be very token-intensive. Next, the Critic\u2019s review identifies the most interesting findings from that set. During the review process the Critic inspects the experts\u2019 claims and the tool calls and tool results used to support them, which also incurs a significant token overhead. Once the Critic has completed its review, it assembles an up to date investigation timeline, integrating the running investigation timeline and newly gathered findings into a coherent narrative. The condensed timeline, consisting only of the most credible findings, is then passed back to the Director. This design allows us to strategically use low, medium, and high-cost models for the expert, critic, and director functions, respectively.<\/span><\/p>\n<h3>Investigation Flow<\/h3>\n<p><span style=\"font-weight: 400;\">The investigation process is broken into several phases. Phases allow us to vary the structure of the investigation loop as the investigation proceeds. At the moment, we have three phases, but it is simple to add more. The Director persona is responsible for advancing the phase.<\/span><\/p>\n<figure id=\"attachment_17666\" aria-describedby=\"caption-attachment-17666\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-17666 size-large\" src=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?w=1020\" alt=\"Flow diagram illustrating how the Director progresses the investigation through distinct phases.\" width=\"1020\" height=\"1564\" srcset=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png 1750w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=640,982 640w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=768,1178 768w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=1280,1963 1280w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=1001,1536 1001w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=1335,2048 1335w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=380,583 380w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=800,1227 800w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/investigation_phases_vertical_final_300dpi.png?resize=1160,1779 1160w\" sizes=\"auto, (max-width: 1020px) 100vw, 1020px\" \/><figcaption id=\"caption-attachment-17666\" class=\"wp-caption-text\">Investigations begin in the discovery phase. After each round of investigation the Director decides whether to remain in the current phase or to progress to a new phase.<\/figcaption><\/figure>\n<p><strong>Discovery<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">The first phase of each investigation. The goal in the discovery phase is to ensure that every available data source is examined. The Director reviews the state of the investigation and generates a question that is broadcast to the entire expert team.<\/span><\/p>\n<p><strong>Director Decision<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">A \u201cmeta-phase\u201d in which the Director decides whether to advance to the next investigation phase or continue in the current one. The task\u2019s prompt includes advice on when to advance to each phase.<\/span><\/p>\n<p><strong>Trace<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">Once the discovery phase has made clear which experts are able to produce relevant findings, the Director transitions the investigation to the trace phase. In the trace phase, the Director chooses a specific expert to question. We also have the flexibility to vary the model invocation parameters by phase, allowing us to use a different model or enhanced token budget.<\/span><\/p>\n<p><strong>Conclude<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">The Director transitions the investigation to the concluding phase when sufficient information has been gathered to produce the final report.<\/span><\/p>\n<h2>Service Architecture<\/h2>\n<p><span style=\"font-weight: 400;\">Our prototype used a coding agent CLI as an execution harness, but that wasn\u2019t suitable for a practical implementation. We needed an interface that would let us observe investigations occurring in realtime, view and share past investigations, and launch ad-hoc investigations. Critically, we needed a way of integrating the system into our existing stack, allowing investigations to be triggered by our existing detection tools. The service architecture we created does all of these things and is quite simple.<\/span><\/p>\n<p><strong>Hub<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">The hub provides the service API and an interface to persistent storage. Besides the usual CRUD-like API, the hub also provides a metrics endpoint so we can visualise system activity, token usage, and manage cost.<\/span><\/p>\n<p><strong>Worker<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">Investigation workers pick up queued investigation tasks from the API. Investigations produce an event stream which is streamed back to the hub through the API. Workers can be scaled to increase throughput as needed.<\/span><\/p>\n<p><strong>Dashboard<\/strong><\/p>\n<p><span style=\"font-weight: 400;\">The Dashboard is used by staff to interact with the service. Running investigations can be observed in real-time, consuming the event stream from the hub. Additionally the dashboard provides management tools, letting us view the details of each model invocation. This capability is invaluable when debugging the system.<\/span><\/p>\n<h2>Example Report<\/h2>\n<p><span style=\"font-weight: 400;\">We\u2019ve included an edited investigation report which demonstrates the potential of the agents to exhibit novel emergent behavior. In this case, the original alert was raised for a specific command sequence, which we analyze because it can be an indicator of compromise. In the course of investigating the alert, the agents independently discovered a separate credential exposure elsewhere in the process ancestry.<\/span><\/p>\n<figure id=\"attachment_17650\" aria-describedby=\"caption-attachment-17650\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-17650 size-large\" src=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png?w=1020\" alt=\"Tree diagram illustrating how agents navigated the process tree.\" width=\"1020\" height=\"746\" srcset=\"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png 1317w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png?resize=640,468 640w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png?resize=768,562 768w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png?resize=1280,936 1280w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png?resize=380,278 380w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png?resize=800,585 800w, https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/process_ancestry_retina.png?resize=1160,848 1160w\" sizes=\"auto, (max-width: 1020px) 100vw, 1020px\" \/><figcaption id=\"caption-attachment-17650\" class=\"wp-caption-text\">The highlighted leaf process triggered the investigation, but the agents traced the process hierarchy and discovered a different issue in an ancestor process.<\/figcaption><\/figure>\n<p><span style=\"font-weight: 400;\">The text below is a lightly edited version of the report summary from this investigation.<\/span><\/p>\n<hr \/>\n<p><em><strong>Investigation Report<\/strong>: Credential Exposure in Monitoring Workflow <strong>[ESCALATE]<\/strong><\/em><\/p>\n<p><em><strong>Summary<\/strong>: While investigating [command sequence], the investigation uncovered a credential exposure elsewhere in the process ancestry chain.<\/em><\/p>\n<p><strong><em>Analysis<\/em><\/strong><\/p>\n<p><em>The investigation confirmed that the command execution on [TIMESTAMP] was part of a legitimate monitoring workflow using [diagnostic tool]. The process ancestry shows the expected execution chain. However, critical security concerns were identified:<\/em><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><em><strong>Credential Exposure<\/strong>: A credential was exposed in process command line parameters within the ancestry chain, creating significant security risk.<\/em><\/li>\n<li style=\"font-weight: 400;\"><em><strong>Expert-Critic Contradiction<\/strong><span style=\"font-size: 1.25rem;\">: The expert incorrectly assessed credential handling as secure while the critic correctly identified exposed credentials, indicating analysis blind spots that require attention.<\/span><\/em><\/li>\n<\/ol>\n<hr \/>\n<p><span style=\"font-weight: 400;\">What is notable about this result is that the expert did not raise the credential exposure in its findings; the Critic noticed it as part of its meta-analysis of the expert&#8217;s work. The Director then chose to pivot the investigation to focus on this issue instead. In the report, the Director highlights both the need to mitigate the security issue, and to follow-up on the expert\u2019s failure to properly identify the risk. We referred the credential exposure to the service owning team to resolve.<\/span><\/p>\n<h2>Conclusion<\/h2>\n<p><span style=\"font-weight: 400;\">We\u2019re still at an early phase of our journey to streamline security investigations using AI agents, but we\u2019re starting to see meaningful benefits. Our web-based dashboard allows us to launch and watch investigations in real time, and investigations yield interactive, verifiable reports that show how evidence was collected, interpreted, and judged. During our on-call shifts, we\u2019re switching to supervising investigation teams, rather than doing the laborious work of gathering evidence. Unlike static detection rules, our agents often make spontaneous and unprompted discoveries, as we demonstrated in our example report. We\u2019ve seen this occur many times, from highlighting weakness in IAM policies, to identifying problematic code and more. <\/span><\/p>\n<p><span style=\"font-weight: 400;\">There\u2019s a great deal more to say. We look forward to sharing more details of how our system works in future blog posts. As a preview of some future content from the series:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Maintaining alignment and orientation during multi-persona investigations<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Using artifacts as a communication channel between investigation participants<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Human in the loop: human \/ agent collaboration in security investigations<\/span><\/li>\n<\/ul>\n<h3>Acknowledgements<\/h3>\n<p><span style=\"font-weight: 400;\">We wanted to give a shout out to all the people that have contributed to this journey:\u00a0<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Chris Smith<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Abhi Rathod<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Dave Russell<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Nate Reeves<\/span><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n\t\t<div class=\"hiring\">\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"26\" height=\"37\" fill=\"none\" viewbox=\"0 0 26 37\"><path stroke=\"#032d60\" stroke-linejoin=\"round\" stroke-width=\"5\" d=\"m4.112 1c-2.5 6.167-2.4 21.1 18 31.5\"\/><path stroke=\"#032d60\" stroke-width=\"5\" d=\"m20.112 18 2.5 14.5-13.5 1.5\"\/><\/svg>\n\t\t\t<p>Interested in taking on interesting projects, making people\u2019s work lives easier, or just building some pretty cool forms? We\u2019re hiring! \ud83d\udcbc<\/p>\n\t\t\t<a href=\"https:\/\/slack.com\/jobs\/dept\/engineering\"\n\t\t\t\tclass=\"\" target=\"_blank\"\n\t\t\t\tdata-clog-click=\"\"\n\t\t\t\tdata-clog-trigger=\"trigger=\"\n\t\t\t\tdata-clog-ui-element=\"\"\n\t\t\t\tdata-clog-ui-component=\"\">Apply now<\/a>\n\t\t<\/div>\n\t\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"Slack\u2019s Security Engineering team is responsible for protecting Slack\u2019s core infrastructure and services. Our security event ingestion pipeline handles billions of events per day from a diverse array of data sources. Reviewing alerts produced by our security detection system is our primary responsibility during on-call shifts. We\u2019re going to show you how we\u2019re using AI&hellip;","protected":false},"author":560,"featured_media":17671,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[3],"tags":[544,628,634],"class_list":{"0":"post-17645","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-uncategorized","8":"tag-development","9":"tag-security","10":"tag-software-engineering","11":"ts-entry"},"acf":{"subtitle":"","excerpt":"We built an agentic security investigation service to help us research alerts as part of our mission to keep Slack secure and protect our customers. Our service deploys teams of AI agents that collaboratively perform security investigations. AI agents free human analysts from tedious data gathering tasks. Over just the first quarter of their deployment, our agents have performed over 7,500 investigations, issuing over 500,000 tool calls. Our agents are enabling us to gain unprecedented real-time insight into Slack\u2019s infrastructure in a way we could never do with human labor alone.","has_toc":true,"author_group":{"configure_author":"wordpress","authors":[{"ID":17643,"post_author":"560","post_date":"2025-11-25 18:08:45","post_date_gmt":"2025-11-26 02:08:45","post_content":"","post_title":"Dominic Marks","post_excerpt":"","post_status":"publish","comment_status":"closed","ping_status":"closed","post_password":"","post_name":"dominic-marks","to_ping":"","pinged":"","post_modified":"2025-11-25 18:08:45","post_modified_gmt":"2025-11-26 02:08:45","post_content_filtered":"","post_parent":0,"guid":"https:\/\/slack.engineering\/?post_type=author&#038;p=17643","menu_order":0,"post_type":"author","post_mime_type":"","comment_count":"0","filter":"raw"}],"custom_author":""},"series":false,"tags":[628,544,634]},"jetpack_featured_media_url":"https:\/\/slack.engineering\/wp-content\/uploads\/sites\/7\/2025\/11\/knowledge_pyramid_retina_5f44b2.png","_links":{"self":[{"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/posts\/17645","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/users\/560"}],"replies":[{"embeddable":true,"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/comments?post=17645"}],"version-history":[{"count":14,"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/posts\/17645\/revisions"}],"predecessor-version":[{"id":17672,"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/posts\/17645\/revisions\/17672"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/media\/17671"}],"wp:attachment":[{"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/media?parent=17645"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/categories?post=17645"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/slack.engineering\/wp-json\/wp\/v2\/tags?post=17645"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}