Many large enterprises discard most of the log data their systems generate, and they do it on purpose to keep costs down. A Dynatrace survey of 450 senior IT leaders at large enterprises found that half of organizations drop or never collect an average of 86 percent of their logs, even after filtering and aggregation. Many also limit how long they retain the logs they do keep.
That choice carries a security cost of its own.
What logs do for an investigation
Logs are the record of what happened inside an application or a piece of infrastructure. They capture errors, events, and actions in sequence, which makes them the raw material for threat hunting, incident response, and forensics. When an organization conducts cyber forensics or runs a security investigation, log data is among the first things it reaches for. Security investigations rank among the most common uses for logs at the enterprises polled.
A decision to drop the bulk of that material, or to age it out after a short window, lands directly on this work. An intrusion can sit undetected for weeks or months before anyone notices. When the alert finally arrives and an investigator goes looking for the trail, the relevant entries may have been sampled away or deleted long before the breach surfaced. The evidence is gone, and the budget owner who removed it often worked in a separate team with a separate mandate.
The decision sits outside security
Log retention and ingestion are usually managed by observability, platform engineering, or cost-control functions. Those teams answer to spending targets. Two thirds of the organizations in the survey said the cost of their log management approach has grown larger than the value they get from it, and most reported higher log costs over the past year. Spending on logging tools at a single large enterprise averages close to $2.5 million a year and consumes roughly half of the money set aside for observability and monitoring.
Given bills like that, teams trim. They limit storage duration, sample a subset of common logs, and stop collecting categories of data they judge to be repetitive. Each step lowers the bill. Each step also narrows the field of view that a security team depends on after the fact.
AI tightens the squeeze
The cost pressure has a driver behind it. Organizations running AI workloads report that their log and telemetry volume has climbed sharply over the past year. More data means higher ingestion, storage, and query costs, which pushes the cost-cutting behavior harder. AI adds to a cost problem that already existed, and it accelerates the conditions that lead to deletion.
The same workloads make the lost visibility more consequential. AI systems behave in ways that are difficult to predict, and understanding why one produced a given output depends on having a detailed record of the inputs, the calls it made, and the services it touched. Many organizations already say their logs show only part of what is happening inside their AI applications. Discarding more of that record leaves them with even less to work from.
Agents read the logs
One detail in the research points to a security question that the cost story tends to bury. AI agents write logs and read them, which turns log data into a shared language between software and the people running it. An agent that consumes logs and acts on what it finds becomes a target.
Tampered or injected log entries could steer an automated system toward the wrong action. The research notes training data poisoning as a concern for a small share of respondents and stops short of connecting it to the logs that agents consume, which leaves an open area for security teams to examine on their own.
A question of who decides
The practical issue is governance. The people deleting logs and the people who need them during an incident are frequently different people, measured against different goals. Security leaders who assume their organization retains the telemetry needed for an investigation may want to confirm what is being collected, what is being dropped, and how long any of it survives. The answer, for many enterprises, is less than they expect.
Leave A Comment