VirtualMetric DataStream is VirtualMetric DataStream is a data pipeline and transformation layer built for Elasticsearch, Elastic Cloud, and Elastic Security.
It collects logs from on-premises, cloud, legacy, OT/ICS, IoT, and custom application sources using Elastic Agents, Beats, agentless collection (WinRM/SSH), Syslog, CEF, LEEF, HTTP, or direct APIs.
The pipeline parses and normalizes data to the Elastic Common Schema (ECS) using vendor-specific mappings, then enriches events with contextual metadata and validates them against ECS requirements. It filters, deduplicates, samples, and extracts fields to reduce data ingest volume before it reaches Elasticsearch, with vendors citing a 50-90% reduction in ingest volume.
DataStream uses multi-stage routing to send ECS-normalized security events to Elastic Security, full data to Elasticsearch storage tiers, and raw data as JSON or Parquet to cloud storage (AWS S3, Azure Blob, Google Cloud Storage) with correlation IDs for later retrieval. It includes schema drift detection that validates against ECS to prevent breaking changes to detection rules, Kibana visualizations, or compliance reports.
Deployment options include Docker/Kubernetes containers, on-premises agents, cloud-native deployment (AWS, Azure, GCP, Elastic Cloud), air-gapped/data-residency configurations, and multi-tenant MSSP configurations. It uses the Elasticsearch Bulk API with API key and basic authentication over TLS, native Elasticsearch Ingest Pipeline execution, and high-throughput batching with retry and rate-limiting logic.
Data pipeline that normalizes logs to ECS and cuts ingest volume for Elastic Stack, developed by VirtualMetric. It is a Security Operations solution designed to help security teams with Log Management, Security Compliance Training, Data Exfiltration.
