🌊 The First-Ever Streamkap Newsletter
Streaming data, shift-left architecture, and BYOC (bring your own cloud) deployments.
Hi Folks,
Welcome to the Streamkap Rekap Newsletter! We’ll be sharing weekly news from the world of data streaming, articles from our blog for deep dives, and updates from our product team.
Let’s start with a trend near and dear to our hearts at Streamkap: streaming data, shift-left architecture, and BYOC (bring your own cloud) deployments.
This past week saw rumors swirling about a potential acquisition of Redpanda by Snowflake with Redpanda positioning for a $1.5B valuation on a reported $20m revenue in 2023. Redpanda is growing 300% in FY24, so I’m sure that revenue number is low, but this acquisition wouldn’t be primarily motivated by Redpanda’s revenue. Let’s get into it.
First of all, Snowflake is not the first mover in this space, streaming giant Confluent acquired 16-month-old startup Warpstream for $220m last year to fast-track their ability to provide BYOC, a gap in their offering relative to Redpanda and Streamkap.
BYOC is emerging as the preferred deployment model for data infrastructure, companies are increasingly aware of the value of their data with the growth in AI and of the risks of sharing data outside their environments.
So why does Snowflake need Redpanda? Snowflake led the push to the cloud for data warehousing, innovated with the separation of compute and storage, and built a $3B ARR behemoth. While they are well positioned, albeit with strong competition from Databricks, architectures are changing.
Over the past 10 years, the ELT (extract load transform) model dominated, dump your raw data into the data warehouse and model it from there. While this had advantages over traditional slow, brittle, opaque, and expensive ETL (extract, transform, load) it is not perfect.
What is new?
Shift left architecture - Traditional ELT processes all data in a warehouse, often resulting in inefficiencies and delays. By shifting processing earlier in the data pipeline—within the stream itself—several key advantages emerge:
Faster, More Efficient Processing – Data quality can be assured in flight, reducing errors and the need for costly reprocessing.
Real-Time Insights and Actions—Instead of waiting for batch jobs, businesses can make immediate decisions based on live data.
Cost Savings – Compute resources are only used where needed, avoiding unnecessary processing of low-value data.
Apache Iceberg – Decoupling Compute and Storage
Many organizations today store their data in Snowflake, locking themselves into a proprietary ecosystem where only Snowflake’s compute can be used. Iceberg changes this dynamic:
Open Storage Format – Data is stored in an object storage system like AWS S3, not inside a vendor’s proprietary system.
Compute Flexibility – Any processing engine—Databricks, DuckDB, Apache Flink, or others—can analyze the data, reducing vendor lock-in.
Optimized Cost Structure – Instead of paying a premium for bundled storage and compute, companies can choose best-of-breed compute solutions while keeping storage costs low.
Where does Streamkap fit into this? Streamkap is a streaming platform, like Redpanda and Confluent, and we offer BYOC like they now both do. While Redpanda and Confluent offer managed infrastructure, they still require building up from lower-level infrastructure, Streamkap prioritizes end-to-end workflows with an application layer to set things up (or API/Terraform) and supports streaming to a variety of destinations with Iceberg as a pillar of how our customers will be using Streamkap in 2025.
Streamkap is doing it for me streaming, vs. do it yourself with the others.
What’s happening in the physical world:
🤫 If you’re in San Francisco, I am hosting the monthly SF Low-Key Data Happy Hour on February 18th. If you're nearby, we'd love for you to stop by! It would be great to see you, catch up, and chat about the latest and most exciting news in the field.
Latest From the Blog
The latest blog post explores Change Data Capture (CDC)—a method for detecting and capturing changes in data as they happen. We’ll look at what CDC is, how it works, and the advantages of streaming change events.
You'll also find insights into the business benefits of CDC, from improving operational efficiency to enabling real-time analytics, along with key use cases where CDC can make a difference. Read here Change Data Capture for Streaming ETL.
Warmly,
Paul Dudley
Great read 🔥