Indexing the Data Lake for Online Point Queries

Spotify's Random Access Parquet (RAP) indexing solution enables sub-second point queries on exabyte-scale data lakes by replacing expensive dependent read chains with direct key-to-location lookups, allowing organizations to serve interactive user experiences and power AI agents without maintaining separate specialized serving systems. This approach maintains data in cost-efficient cloud object storage (GCS, S3) while achieving latency comparable to traditional key-value stores, with index overhead scaling predictably (terabytes of index per petabytes of data). For IT organizations, RAP represents a strategic shift toward unified data architectures that eliminate data silos and reduce operational complexity while improving both performance and economics.