How we use Airflow and Astronomer to find restaurants worth talking to

Every early-stage product has the same unglamorous problem: the people who would genuinely benefit from it don’t know it exists. For us those people are independent restaurants, cafés, bars, and food trucks — and there is no list of them you can simply buy that is any good.

So we treat it as what it is: a data problem. This post is about the technologies we point at it. It’s deliberately light on specifics — the shape of the pipeline is standard, and the parts that took real work aren’t the interesting bit for a reader anyway.

Why an orchestrator at all

The naive version of this is a script on a laptop. That works exactly once.

The real job has the properties that make people reach for an orchestrator: it runs in stages that depend on each other, any stage can fail on someone else’s network, it has to be safely re-runnable without creating duplicates, and it needs to be politely rate-limited against every source it touches. That last constraint means the whole thing is long-running and bursty, which is precisely where a script’s error handling becomes a pile of try blocks nobody trusts.

That’s the job description of a workflow orchestrator, so we use one.

Apache Airflow, with Astronomer’s tooling

Apache Airflow 3 is the orchestrator. Pipelines are Python, dependencies between stages are explicit, every run is observable, and individual stages can be retried in isolation rather than re-running the whole job. When a stage fails because a third party returned a 503, Airflow retries it on its own and we read about it afterwards.

Astronomer’s runtime image and CLI are how we get Airflow without assembling Airflow. A working Airflow is a scheduler, a metadata database, and a web server that all have to agree with each other; their runtime packages that into one reproducible container definition, and the CLI brings it up in Docker. Standing that stack up by hand is a genuine infrastructure project, and we have one product to ship.

That’s the same instinct as the rest of our stack: take the well-trodden version of the boring part, and spend the effort on the part that’s actually ours.

The shape of a run

Pub/Sub carries the work requests. Worth being precise about what that does and doesn’t do: a request enqueues work, it doesn’t start a run. Decoupling “something should be looked at” from “a pipeline is running” means the two never have to be available at the same moment — requests accumulate harmlessly, and the next run picks up whatever is waiting.

Discover reaches out to public sources of business information, politely. Rate limiting here isn’t a performance optimization, it’s a condition of use, and treating it as optional is how you lose access to a source permanently.

Normalize is the stage that does more work than anyone expects. The same restaurant appears across sources with a different name, a differently formatted address, and a phone number written four ways. Getting to one confident record per real-world business is most of the difficulty in this kind of pipeline, and it is very much not a solved problem you can import.

Enrich calls Vertex AI with Gemini models to classify and score what we found — the same AI platform the product itself uses for reading menus, which keeps one fewer vendor in our life.

Load writes into BigQuery as an idempotent upsert. This is what makes re-runs safe: running the same work twice converges on the same table rather than doubling it. If you build one thing carefully in a pipeline like this, build this.

From there an internal, access-controlled tool presents the results for a human to review. Nothing here contacts anyone automatically.

The Python underneath

The orchestration is Airflow, but the logic is ordinary Python, kept deliberately separate from it. Stages call into plain modules that know nothing about Airflow, which means the substantial part of the codebase is testable with pytest in milliseconds, with no scheduler and no containers running.

That separation is the single design decision we’d recommend to anyone building on an orchestrator. Logic that can only be exercised by triggering a pipeline run is logic that effectively isn’t tested. We use the Google Cloud client libraries directly rather than heavier provider packages, which keeps the runtime image small and the build fast.

On doing this responsibly

Since this is a post about finding potential customers, it’s worth being direct about the boundaries.

We work from publicly available business information — the same things you’d see looking up a restaurant yourself. We respect the rate limits and terms of every source. We don’t scrape personal data, we’re building a picture of businesses rather than individuals, and a human reviews everything before anyone hears from us.

None of that is legally mandated in every jurisdiction we operate in. We do it because a prospecting pipeline that annoys restaurant owners is worse than no pipeline at all, and because the alternative is the exact thing everyone hates about their inbox.

What this bought us

A repeatable way to answer “which independent restaurants in this area are still handing out laminated menus, and might genuinely be better off with something else” — and to answer it again next month without redoing the work.

The technologies are all off-the-shelf: Airflow, Astronomer, Pub/Sub, BigQuery, Vertex AI, Python. There’s nothing exotic in the list. What’s ours is the judgment about which restaurants are actually a fit, and that isn’t a technology at all.