More than a decade ago, I was working on a job scheduler similar to Airflow. We called it Woodsy internally. At first, it was quite simply an application that only uploaded CSV files to Redshift. People started using it, and before long it had grown well beyond its original purpose. It eventually became critical infrastructure for data processing at TripAdvisor. Many critical business functions started depending on it, so it became more than a toy project. So, we hardened it.
We separated the UX from the backend, added queuing and distributed task processing, made the database highly available, and so on. We also added telemetry and alerts to all of the components. It worked fairly well, and we did get paged from time to time for the right reasons. One day, a few of the business analysts were complaining that the jobs weren’t submitting. I looked at all possible avenues. Everything looked good. In theory, everything looked perfect, but I then realized someone had started uploading a huge CSV file that blocked the entire API surface and hosed the application.
That's when we thought we needed an alert that somehow measured the user experience, not something internal to the application. That's when I coined the term void job, simply writing a partition to HDFS to check if, from scheduling all the way to job completion, everything works. It validates the database connection, scheduling, and task completion. This is especially important when components are distributed across the UX, scheduler, queue and all that. If, in the last five minutes, you scheduled 10 void jobs and they all completed, you are simply up and running. It doesn’t prove every workload is healthy, but it gives you a continuous signal that the basic customer path is functioning.
Woodsy - Your One stop for transferring, consolidating, transforming, and scheduling data!
Control the Input
Since Woodsy, I took the idea with me. The concept applies to many domains. Say you run an email delivery system. You can periodically send a synthetic message through the same API, queue, workers and delivery path as normal traffic, then verify that it actually arrives in a controlled inbox. Or say you run a real-time analytics pipeline. You can inject a known event at ingestion and check that the expected result eventually appears in the downstream aggregation. In both cases, you are exercising the path the customer depends on rather than relying on each component health telemetry. If the synthetic operation stops completing, you know something meaningful has broken.
There is another reason I prefer a synthetic operation over treating every failed customer action as an availability signal. Customer requests are messy. Someone can upload a malformed file, use the wrong credentials, or lose their network connection. If I counted all of those as service failures, I would end up alerting for things the system handled correctly.
A void job removes most of that ambiguity. I control the input, I know the credentials work, I know what the expected result is, and I can keep the workload deliberately simple and stupid. If that operation stops completing, the signal means considerably more.
I later learned that the broader industry already had names for this family of ideas: synthetic transactions, black-box monitoring, canary transactions. The name doesn't matter much to me. What mattered was where I chose to measure the system. This is why I still believe this is a good way of measuring an end-to-end SLO. Send something through the normal entry point and see if it makes it to the end. Individual component health might be evidence about those components, but in a complex distributed system, component evidence isn’t a promise about the customer experience.
Measure the Promise
A database can be operational while the application cannot connect to it. A queue can accept work while no job completes. Every worker can be sending heartbeats while customers are still unable to get a result. Each component might be telling the truth in its own realm. The problem is that they are answering a smaller question.
If the system is meant to let you submit a job and get a result, then I want the SLO to sit as close to that experience as possible. Database availability, queue depth, and worker health are independently useful evidence. Job completion is what Woodsy was there to deliver.
The same goes for latency. I can measure how quickly the scheduler responds, how long something sits in the queue, or how long a worker takes to execute. Those numbers matter when I need to understand where time went. The person using the system experiences something simpler: how long it took from submitting the job to getting the result.
Void jobs are not proof that everything works. A tiny HDFS write completing didn’t tell me that a massive job would finish or that every customer was having a good experience. Real workloads still matter. So does all the component telemetry we added in the first place.
But those signals serve a different purpose. The void job tells me that something meaningful is broken. The component telemetry helps me figure out why. More than a decade later, that is still roughly where I want to measure health: as close as possible to the thing the system was built to do.
About the Name
About two years into Woodsy, we briefly considered open-sourcing it. We quickly discovered that the name would have to change because Woodsy Owl is protected under US federal law. We never ended up doing it, which still annoys me a little.
Woodsy had a SQL editor that worked across all configured databases, distributed job execution, pipelining, cached query results in S3 or locally, reporting, and a UI that let analysts build and run the whole thing without dropping into code. A few years later, Airbnb open-sourced Airflow and that became the name everyone remembers. Woodsy never got that chance.
I still think the early version of Woodsy was better.
Perhaps I am biased because I spent so much time building it, but I am also not completely joking. At least we avoided having to explain to legal why our data tool had accidentally borrowed a federally protected cartoon owl.