field notes
Quick observations and lessons learned in the field.
Reading List & Resources
Blogs
- A collection of data engineering blogs from across organizations
- Semantic Search and RAG for Hacker News and a vector search application example
- Open Source Data Engineering
- A good read about LTAP by Reynold Xin
- Spotify blog on Indexing the Data Lake for Online Point Queries using Random Access Parquet
- Atlassian blog on scaling StreamHub, transitioning from Kinesis to Kafka for 145 billion daily events
- Databricks August 2026 release notes, including Unity AI Gateway going GA
Papers & Research
- BAIR blog on Compound AI systems and what are they?
- Some good posts on text retrieval and LLMs
- Readings in Databases from rxin.
- Zerrow: True Zero-Copy Arrow Pipelines in Bauplan: I was playing around with Polars for a side project recently and this is an interesting related read on zero-copy.
Conference Talks
- Data-Driven Detection using PySpark - I was part of this during my time at Databricks