Every story tagged Data Pipeline, curated for CIOs and IT leaders — ranked by source credibility, engagement, and freshness.
3 stories · open in the command center
Streambed enables organizations to stream changes from production PostgreSQL databases directly to Apache Iceberg tables on S3, eliminating the need for complex ETL pipelines and reducing load on transactional systems. This open-source tool uses logical replication to capture database changes, converts them to Parquet files, and exposes them via the Postgres wire protocol, allowing analytics teams to query production data without impacting application performance. For IT organizations, this represents a cost-effective alternative to traditional data warehouse solutions, simplifying the data pipeline architecture while maintaining familiar Postgres query interfaces.
Definity's embedded agent technology fundamentally shifts data pipeline operations from reactive post-failure monitoring to proactive, in-execution intervention, directly addressing the critical data quality and reliability requirements that agentic AI systems demand. By installing agents inside Spark execution layers rather than monitoring externally, enterprises can detect and prevent failures in real-time, reduce troubleshooting effort by 70%, and unlock significant infrastructure optimization—with one customer identifying one-third of optimization opportunities in the first week. This architectural approach has strategic implications for IT organizations: it redefines the data reliability challenge from a tools problem to an actionable intelligence problem, enabling shift-left data operations and reducing the costly downstream impact of data failures on AI-dependent business processes.
Airbnb successfully migrated from StatsD to OpenTelemetry Protocol (OTLP) for their billion-series Prometheus metrics pipeline, reducing CPU overhead from 10% to under 1% while improving reliability and enabling advanced Prometheus-native capabilities. The migration required careful handling of high-cardinality metrics through delta temporality for critical services and strategic dual-write approaches for low-friction adoption. This positions Airbnb with a vendor-neutral, CNCF-standard telemetry infrastructure that supports future observability needs while maintaining cost control through streaming aggregation.