The Six Month YAML Problem

Most companies hit the same wall around Series B.

The data team is two or three people. They're good. They've been doing heroic work, stitching together pipelines in notebooks, running ad hoc queries, keeping the lights on with duct tape and institutional knowledge. The business has grown around them without anyone really noticing how fragile the foundation is.

Then someone (usually a new VP of Data, or a board member who just came from a bigger company) says: we need a real data platform. Airflow. Airbyte. A proper transformation layer. A governance story. Something that looks like what the grown-up companies are running.

Everyone agrees. It gets put on the roadmap. And then it takes six months. Sometimes a year. Here's exactly why.

The architecture debate

Before a single line of infrastructure gets written, there's a meeting. Then another meeting. Airflow vs Prefect. Airbyte vs Fivetran. Do we use dbt Cloud or run dbt Core ourselves. Kubernetes or ECS. Which cloud, which region, which account structure. Everyone has an opinion. The senior engineer who's done this before advocates for the conservative choice. The new hire who just came from a bigger company wants the architecture they had there. The Head of Data is trying to keep everyone aligned while also answering Slack messages about why last Tuesday's revenue number looks different in two dashboards.

Nothing gets decided in two weeks. Usually nothing gets decided in four.

The YAML phase

Once a direction is chosen, someone has to actually build the thing. In most companies this means one engineer becomes the de facto platform engineer. Let's call him the person who got voluntold. His actual job was data engineering. His new job is infrastructure.

He starts with Airflow. Airflow is powerful and Airflow is a beast. The Helm chart alone has 700 configurable parameters. He needs to understand Kubernetes well enough to not shoot himself in the foot. He needs to configure the scheduler, the webserver, the workers, the metadata database, the log storage, the secret management. He needs to decide how DAGs get deployed. Sync from Git? CI/CD pipeline? Bake them into the image? Each of these decisions has downstream consequences he won't discover for three months.

Then Airbyte. Then DataHub, which has its own sprawling configuration surface and a community Slack where the answers to your questions are usually "it depends" or a link to a GitHub issue from 2021 that was closed without resolution.

Each tool takes longer than estimated. Not because the engineer is slow. Because this is genuinely hard and nobody told him it would take this long because nobody had actually done all of it together before.

The unplanned interruptions

While the platform engineer is heads down, the business doesn't stop. A pipeline breaks and he gets pulled in because he's now the person who knows where things live. A new data source needs to be connected before the board meeting. Someone in finance needs a dashboard by Thursday. The on-call rotation didn't exist six months ago but it does now and his name is on it.

Each interruption costs more than its face value. Context switching on infrastructure work is brutal. An hour of interruption on a hard debugging session can cost half a day of actual progress. The six-month estimate was made assuming focused work. Focused work is not what happens.

The "it works on my machine" phase

Eventually something that looks like an environment exists. It works. Mostly. The engineer demos it and people are excited. Then someone tries to use it and hits an error that only happens in production. Then another. The error messages from Airflow are famously unhelpful: a task failed, here is a wall of Python traceback, good luck. The engineer spends a week debugging something that turns out to be a permissions issue in the IAM role attached to the ECS task definition.

This is not in anyone's six-month estimate.

The documentation that doesn't exist

The platform is running. The engineer who built it knows how it works. Nobody else does. There is no runbook for what to do when the scheduler goes down at 2am. There is no documentation for how to add a new connector. There is no onboarding guide for the next data engineer who joins in four months and needs to be productive in week one.

The engineer means to write this documentation. He never has time. The documentation lives in his head.

This is the moment the company has traded one kind of fragility for another. Before, the fragility was the pipelines themselves. Now, the fragility is the person who built them. If he leaves, the company starts over. And he might leave. He was hired to do data engineering, not platform operations.

What SkaleData is

SkaleData is the answer to the question: what if you could have the open-source stack, running in your own cloud account, without any of that?

Not a SaaS platform that takes your data and runs it through their infrastructure. Not a consulting engagement that produces a handoff document and a thank you note. A fully managed, fully operated data stack that lives inside your AWS, GCP, or Azure account. Your data never leaves. Your team never gets voluntold. The first response on infrastructure issues moves to us, not your best engineer.

The stack is Airflow, Airbyte, DataHub, and Superset. These are not our proprietary tools. They're the same open-source tools your team would choose if they were building it themselves. We run them, maintain them, upgrade them, and own the on-call escalation for them.

The six months becomes a conversation and a deployment. The YAML is our problem. The on-call rotation is our problem. The IAM permissions rabbit hole is our problem.

Your engineers get to do data engineering.

Request Early Access or book a demo and we'll show you what the deployment looks like in your account.