Pipelines that stay correct at ten million rows, not just ten.
I design and build the ingestion, transformation, and storage layers behind your data — architected for the volume you'll actually hit, not just the sample dataset in the demo.
Pipeline idle.
what makes it "big"
The four properties that break a normal database
These are the actual constraints that decide whether you need a data pipeline at all, or just a bigger Postgres instance.
Volume
Enough rows that a full table scan, a naive join, or an unindexed query stops being a rounding error and starts being an outage.
Velocity
Data arriving continuously rather than in a nightly batch — clickstreams, IoT telemetry, transaction feeds that need near-real-time handling.
Variety
Structured rows, semi-structured JSON, logs, and documents all landing in the same pipeline, each needing a different validation path.
Veracity
Duplicate events, late arrivals, and schema drift are the default, not the exception — the pipeline has to expect and handle them.
what I build
Pipeline and platform engineering
From a first ETL job to a full lakehouse serving your BI stack.
ETL / ELT pipeline design
Extraction from operational databases, APIs, and files, with transformation logic in dbt or custom code and orchestration via Airflow or Dagster.
Data warehouse & lakehouse architecture
Schema design, partitioning, and tiered hot/warm/cold storage on Snowflake, BigQuery, Redshift, or an open lakehouse on S3 with Iceberg or Delta.
Streaming analytics
Kafka or Kinesis-based streams with consumers that aggregate, enrich, or alert on events within seconds of them happening.
Scheduled processing systems
Idempotent, re-runnable batch jobs with backfill support, so a failed run at 3am doesn't mean manually reconstructing yesterday's numbers.
Data quality & governance
Automated validation, schema contracts, and lineage tracking so a broken upstream field is caught before it reaches a dashboard.
BI & reporting integration
Modeling the warehouse for the tools your team already uses — Looker, Metabase, Power BI — so numbers match across every report.
engagement
How a data platform project runs
Architecture decided before infrastructure, because storage and partitioning choices are expensive to reverse later.
Profile the data and the questions it needs to answer
Volume, arrival pattern, and the actual business questions the pipeline needs to support — this decides streaming versus batch, and how the warehouse gets modeled.
Design the storage and partitioning strategy
Schema, partition keys, and hot/warm/cold tiering decided upfront, since re-partitioning a production warehouse later means downtime and cost.
Build ingestion and transformation
Idempotent, re-runnable jobs with schema validation at the boundary, so bad upstream data is caught before it lands in the warehouse.
Load-test at realistic volume
Pipelines are run against data sized to match production, not a sample — this is where partitioning mistakes and slow joins actually surface.
Ship with monitoring and a runbook
Deployed with pipeline health monitoring, data quality alerts, and a runbook your team can follow when a job fails at 3am without me.
stack
Tools and platforms I work with
Warehouses & lakes
Processing & orchestration
Databases
questions
Frequently asked
How do I know if we actually need a "big data" pipeline?+
If a well-indexed Postgres or MySQL instance handles your queries in an acceptable time, you don't. The signal is usually one of: queries slowing down as tables grow past tens of millions of rows, needing near-real-time data rather than nightly reports, or combining structured and semi-structured sources that don't fit one schema.
Should we use streaming or batch processing?+
Batch is simpler to build, test, and debug, and is the right default unless a specific use case needs data within seconds rather than minutes or hours — fraud detection and live dashboards are typical reasons to reach for streaming; most reporting isn't.
How do you decide between Snowflake, BigQuery, Redshift, or a self-managed lakehouse?+
It comes down to existing cloud provider, query patterns, and cost model — usage-based warehouses suit spiky workloads, while a self-managed lakehouse on object storage gives more control at the cost of more operational overhead. I'll walk through the trade-offs against your actual data before recommending one.
What is hot/warm/cold storage tiering, and do we need it?
It's keeping recently accessed data on fast, more expensive storage and moving older data to cheaper, slower storage automatically. It's worth setting up once storage cost becomes noticeable — usually once historical data significantly outweighs the data you actually query day to day.
Can you migrate an existing pipeline without downtime?
In most cases, yes — by running the new pipeline in parallel against the same source, validating outputs match, and cutting over once confidence is established, rather than a hard switch.
What does a typical big data project cost and take?
A single well-scoped pipeline — one source, one destination, with proper validation — usually runs two to four weeks. Full warehouse or lakehouse builds take longer. I'll give a fixed-scope estimate after a short discovery call once I understand your data volume and sources.
Build the pipeline once, trust the numbers every time after.
Tell me what data you're moving and where it needs to end up — I'll tell you honestly what scale of solution it actually needs.