Tracking flight prices,
live.

Where the project stands right now — this section will grow as more data comes in.

This is a live project. The numbers above reflect pipeline scope and cadence, not yet outcome metrics — those will be added as the collection window matures.

The client, a travel company, organises trips and events for clients across a recurring set of European and long-haul routes. Flight costs are one of the largest and most volatile line items in event planning, and pricing decisions were being made on instinct and spot-checks rather than data.

I'm building a continuous price-tracking pipeline that watches 28 routes daily, across two independently verified data sources covering both budget and traditional carriers, and records how price moves as departure approaches. The goal is a data-backed answer to a simple but high-value question: when should a booking actually be made for a given route to get the best realistic price — not the theoretical cheapest fare, but the price a client would actually see when the team needs to book.

Event and trip planning runs on deadlines that don't always align with when fares are cheapest. Without visibility into how prices move over the weeks before departure, bookings were made reactively — either too early, missing later dips, or too late, absorbing the pre-departure price climb.

The question: could a lightweight, self-hosted pipeline produce a reliable enough signal to change when bookings get made — without the cost of an enterprise travel-data subscription?

The first real obstacle wasn't technical — it was availability. Three candidate data sources were evaluated at the start of the project. Two of them turned out to have closed their free/self-serve developer access during the research phase itself — one had shut its public developer portal entirely just weeks before, the other had quietly restricted new registrations to approved partners only. Neither was discoverable as "closed" without actually trying to sign up; both still showed up in search results and older tutorials as viable options.

The third source covered most, but not all, of the routes the client actually needed — the budget carrier most relevant to several client itineraries doesn't participate in third-party distribution at all, through any of the standard channels. Reaching it required going directly to the same publicly accessible search endpoint its own website uses, rather than a conventional developer API — verified as safe and appropriate for a low-frequency, once-daily monitoring use case.

• Each source has its own rate limits, response shape, and failure modes
• No single API covers budget + traditional + long-haul routes reliably
• Free-tier APIs in this space are volatile — access can change with little or no notice

A high-level view of the pipeline flow: from data collection through cleaning, storage, and delivery to the client dashboard. Each stage is automated and monitored independently.

The full architecture diagram with infrastructure-level detail is provided separately and linked below. This section shows the logical flow that any stakeholder — not just a technical reviewer — should be able to follow.

The architecture diagram below shows each stage of the pipeline, from data collection through cleaning, storage, and delivery to the client dashboard.

Note: the architecture diagram is still being refined as more data flows through the pipeline. Stages shown above reflect the current design; details may change as new sources or optimisations are added.

This project is still in the data collection phase. Results and impact metrics are not yet available — the pipeline needs more accumulated data before booking-window patterns can be identified and validated.

• ✓ Pipeline deployed and running — data collection active across 28 routes
• ✓ Two independent data sources integrated and verified
• ✓ Dashboard live — client can view current pricing data by route
• — Booking-window analysis — requires additional data accumulation (in progress)
• — Actionable recommendations — will be built once patterns are validated

Early lessons from building a multi-source data pipeline in a space where free APIs are volatile and unreliable:

1. The "one API" assumption is dangerous. Flight pricing doesn't work that way. Budget carriers, traditional carriers, and long-haul routes often live on completely different data feeds — if they're accessible at all. Designing around a single source would have collapsed the whole pipeline as soon as that source changed its terms.
2. Volatile APIs require graceful degradation. If a source goes down or changes its response format, the rest of the pipeline must keep working. This means designing each source as an independent module with its own error handling, rather than a chain where one broken link breaks everything.
3. Verify before you commit. Before investing days into building a collector for a specific API, verify that it's actually accessible, stable, and likely to remain so. The two APIs that shut down during our research phase had been operational for years — their closures came as surprises because there was no public signal until they happened.

The pipeline runs on a daily cadence. Each morning, it queries both data sources for the 28 tracked routes, cleans and normalises the results into a unified schema, and stores them in a local database. The client dashboard reads from that database and presents the data in an accessible format.

Every stage is automated — no manual data entry, no spreadsheets. The client interacts with the dashboard, not with raw API responses or CSV files.

• 01 Automated daily queries run against both data sources at a fixed time
• 02 Each response is parsed, validated, and normalised into a unified schema
• 03 Normalised data is persisted to a local database with historical records
• 04 The dashboard reads from the database and presents price trends by route
• 05 Booking-window cohorts (45 / 60 / 90 days before departure) are tracked separately
• 06 As data accumulates, trend analysis and recommendations can be layered on top

This is one of the more interesting projects I've worked on because it sits at the intersection of data engineering and real-world decision-making. The flight pricing space is notoriously opaque — and the fact that free APIs in this area can shut down without warning means you can't just grab a dataset and start building. You have to build resilience into the design from day one.

The technical challenge of handling multiple data sources with different formats and reliability profiles is interesting, but the real value for the client isn't in the architecture — it's in answering a simple question they couldn't answer before: when should I book this flight?