Security Data Pipelines

Every security tool an organisation runs produces logs, and for twenty years the answer to what to do with them was the same: send everything to the SIEM. That answer has stopped working. Log volume roughly doubles every eighteen months while SIEM pricing is still largely a function of how much data you ingest, so the bill grows faster than the security team’s budget and faster than the value of the data being stored. A security data pipeline sits between the sources and the destination and decides what actually needs to go there.

The category has moved quickly from an optimisation to an architectural layer. The pitch began as cost reduction: filter the noise, drop the duplicate Windows events, and cut the SIEM bill by half. It has since become something more structural, because once a pipeline is parsing, normalising and enriching data in stream, it is a control point. It decides what the SIEM sees, what goes to cheap object storage for compliance, and increasingly what gets detected on the way past.

Why this market exists

Four pressures created it, and none of them is easing.

Volume. Cloud estates, identity providers, EDR agents and now AI applications each emit their own telemetry, and the number of sources grows with every tool added. A large enterprise commonly runs forty to fifty security products, each with its own log format.

SIEM economics. Ingest-based pricing was tolerable when the data was firewall and endpoint logs. It is not tolerable when a single cloud provider’s audit trail can exceed the whole previous estate. The pipeline vendors’ most quoted number is the reduction they achieve before ingest, typically claimed between 50% and 80%.

Format sprawl. Detection content is written against fields. When every source names its fields differently, the security team spends its time on parsers rather than on detections. Normalisation in the pipeline moves that work to one place.

Retention rules that differ from detection needs. Regulation often requires keeping data for years that nobody will query in anger. Splitting the hot path from the archive path is cheaper than storing everything where it is most expensive.

What changed in 2025

Two acquisitions inside a fortnight moved this from an emerging category to a contested one. On 27 August 2025 CrowdStrike agreed to buy Onum, a Madrid telemetry-pipeline company, for around $290m. On 8 September SentinelOne agreed to buy Observo AI for roughly $225m in cash and stock. Both buyers sell SIEM products; both bought the layer that decides what reaches a SIEM.

That is worth reading carefully if you are choosing a pipeline. The independent vendors argue their value is vendor-neutrality: a pipeline that will route to any destination gives you leverage over the SIEM contract. When a SIEM vendor owns the pipeline, that argument does not disappear, but it does become a claim the buyer has to test rather than assume. Cribl, the largest independent, has made neutrality its explicit position.

How to evaluate one

The demos in this category look alike, because they all show the same thing: a stream of noisy logs going in and a smaller, tidier stream coming out. The questions that separate the products are less photogenic.

What happens to what you drop? Reduction that discards data is different from reduction that routes it to cheap storage. If an incident later requires the discarded events, the difference is the investigation.

Who writes the parsers, and what happens when a source changes its format? This is the recurring maintenance cost of the category, and the main thing the AI-native entrants claim to solve.

Where does it run? Data residency, air-gapped estates and egress costs all push processing toward the edge. Support for on-premise and hybrid deployment varies sharply.

What does exit look like? A pipeline is a control point by design, which means it is also a dependency. Ask what happens to your parsing and routing logic if you leave.

Where it is going

The clear direction is detection moving upstream. If the pipeline is already parsing and enriching every event in stream, running detection logic there is a short step, and several vendors have taken it, which starts to make the SIEM a query and storage layer rather than the place where analysis happens. Abstract Security has been most explicit about this, positioning a streaming pipeline with embedded detection as a SIEM replacement rather than an adjunct.

The counter-pressure is consolidation. If the major SIEM and endpoint vendors each own a pipeline, the independent market narrows to organisations that specifically want neutrality, which is a real constituency but a smaller one than the whole market. The next year of deals will show which way that resolves.