Last week, an active supply chain attack called ForceMemo was compromising hundreds of GitHub repositories in real time. I needed to run a structured threat hunt across our environment — 10 sequential phases of KQL queries in Microsoft Sentinel, each building on the findings of the last. Phase 1 identifies compromised devices. Phase 3 maps which developers installed from suspect repos. Phase 10 takes the device names from Phase 1 and hunts for lateral movement.
I decided to use Microsoft 365 Copilot as my hunting partner. The idea was straightforward: feed it the campaign context, the IOCs, and the KQL queries for each phase. Copilot would help me refine queries, interpret the results I pasted back from Sentinel, track findings across phases, and flag what to investigate next. I built a detailed prompt — campaign briefing, IOCs, all 10 queries with interpretation guides, instructions to track findings across phases and wait for my confirmation before advancing. A complete, interactive playbook.
It worked well for the first few phases. Clear interpretation, sharp analysis, smooth back-and-forth. Then somewhere around Phase 5, something shifted. Copilot started losing the thread. It forgot the device names we’d flagged in Phase 1. It was asking me questions I’d already answered. My AI co-pilot had developed amnesia in the middle of an active investigation.
The problem wasn’t intelligence. It was context.
The context window problem
Every AI model — M365 Copilot, Claude, ChatGPT, Gemini — has a finite context window. That’s the total amount of text it can “see” at once: your prompt, its responses, your follow-ups, the query results, all of it. When the conversation exceeds that window, earlier content is no longer visible to the model. It doesn’t know it’s lost access — it just stops referencing information it can no longer see.
For a single question — “what does this KQL query do?” — this doesn’t matter. The question and answer fit comfortably in one window.
For a 10-phase threat hunt that accumulates findings over an hour of back-and-forth, it’s a hard wall. Each phase generates query results, interpretation, and discussion. After several phases, I noticed the AI losing reference to earlier findings. It was analyzing later phases in isolation, without the context that made those results meaningful.
This isn’t a knock on M365 Copilot specifically. I hit the wall there because that’s what I was using, but the constraint is fundamental to how large language models work right now — Claude, ChatGPT, Gemini, dedicated tools like Copilot for Security, any of them would hit the same limit on a sufficiently complex investigation. And in practice, the effective context is often smaller than the model’s theoretical maximum — system prompts, plugin schemas, and safety layers all consume tokens before your conversation even starts. The security workflows where AI could add the most value — threat hunting, incident response, forensic analysis — are exactly the workflows that are stateful, sequential, and accumulative. They’re the ones that exceed the effective context first.
Ways to manage state
After hitting this wall, I worked through several approaches to keep the hunt moving. There are more than what I’ll cover here — RAG-based retrieval over your own investigation history, server-side compaction features some AI platforms are starting to offer — but these are the ones that are practical today for a security practitioner who isn’t building custom tooling.
Modular prompts with manual state tracking. The simplest fix. Break the hunt into self-contained, single-phase prompts. Keep a findings tracker you fill in after each phase. When Phase 10 needs device names from Phase 1, you paste them in yourself. The AI handles analysis. You handle continuity.
Context compression — manual or AI-assisted. This is a spectrum. On the simple end, you manually strip raw result tables between phases and carry forward only the essentials: device names, risk assessments, key IOCs. On the more powerful end, you have the AI compress each phase’s findings into a structured summary block that you carry forward. The second version — progressive summarization — is the technique that changed things for me. More on this below.
Notebook and pipeline orchestration. Move the query execution out of the AI entirely. Jupyter notebooks with KQL magic commands, or Azure Logic Apps chaining Sentinel API calls. State lives in Python variables, not in the AI’s context window. Eliminates the context problem but requires engineering investment.
Sentinel workbooks. Build the hunt as a parameterized workbook where each query tile feeds results into the next. The most production-ready approach, but you trade away the interactive AI experience.
Each approach trades off differently between effort and fidelity. In the moment, with an active campaign, I went with modular prompts — breaking the hunt into single-phase chunks and tracking findings manually between sessions. It worked. But it also meant I was the state manager, copying device names and assessments between prompts by hand, making judgment calls about what to carry forward.
After the hunt, I started thinking about a better approach — one that keeps the interactive AI experience but solves the memory problem more cleanly. That’s progressive summarization.
Progressive summarization: how I’d do it next time
The idea is simple. After each phase, before moving on, you ask the AI to compress its findings into a structured summary block — a fixed format, a few lines, just the facts that downstream phases need. Then you start the next phase in a new session, pasting the compressed summaries from all previous phases as context instead of carrying the full conversation history.
You’re not fighting the context window. You’re fitting inside it by controlling what takes up space.
Here’s how it works in practice. After Phase 1 (Solana C2 detection) returns results, instead of just moving to Phase 2, you say:
“Before we continue, compress your Phase 1 findings into this exact format:”
PHASE 1 SUMMARY | Solana C2 Detection | Assessment: [CLEAN/SUSPICIOUS/COMPROMISED]
Devices flagged: [list]
Users flagged: [list]
Key finding: [one sentence]
Action taken: [containment status]
The AI produces five lines. You copy them. When you start Phase 2, your prompt is: the Phase 2 instructions, the Phase 1 summary block, and the Phase 2 KQL query. Total context consumed by Phase 1’s findings: five lines instead of the full multi-turn conversation.
By Phase 10, you’re carrying nine summary blocks — maybe 50 lines total. That fits easily in any model’s context window. And every phase has access to the key findings from every previous phase: device names, user accounts, risk assessments, containment actions.
The compression step does something else that’s surprisingly valuable. It forces the AI to distinguish between what matters and what’s noise in its own analysis. Raw query results include dozens of columns and rows. The summary forces extraction of only the facts that downstream decisions depend on. It’s a form of analytical discipline that actually improves the quality of the hunt, not just the context management.
There are a few principles I’d follow to make this work well.
Structure the summary format tightly. Don’t ask for “a summary.” Give the AI an exact template with fields. Assessment (clean/suspicious/compromised), devices, users, key finding, action taken. The more rigid the format, the more consistent and compact the output. Consistency matters because you’re stacking these summaries across 10 phases — if each one is formatted differently, they become hard to parse at a glance.
Summarize at the phase boundary, not after the fact. The compression needs to happen while the AI still has the full query results in context. If you wait, you’re asking it to summarize something it can no longer see.
Carry all previous summaries forward, not just the last one. Phase 10 might need device names from Phase 1 and repository names from Phase 3. Don’t assume which earlier findings will matter — carry all the summaries. They’re compact enough that this works.
Start a new session for each phase. Based on what I saw during the ForceMemo hunt, long conversations degrade quality even before the context window technically fills. A fresh session per phase with the compressed summaries pasted in should give you a clean slate with full history intact.
What this means for AI adoption in security
The vendor pitch for AI in security operations is “autonomous investigation.” The reality, right now, is “powerful analytical partner with short-term memory.” That’s not a criticism — it’s a design constraint, and understanding it is the difference between getting real value from these tools and getting frustrated by them.
The context window will get bigger. Models will get better at long-range coherence. Agentic frameworks will eventually manage state externally. But bigger windows don’t fully solve this — research shows that models struggle to retrieve information buried in the middle of very long contexts, even when it technically fits. And we’re not waiting for the future. Security teams are adopting AI tools today, for real investigations, against real threats.
If you’re building AI into your security workflows, design for the constraint. Break complex investigations into bounded phases. Use progressive summarization to carry state forward. Keep a human in the loop as the state manager — not because the AI can’t be trusted, but because the architecture requires it right now.
The teams that figure out how to work with AI’s current limitations will be the ones ready to scale when those limitations shrink.
-----
If you found this useful, subscribe to get the next one.