Shipment Tracking Automation for a Freight Forwarding Company

Client context

A mid-sized freight forwarder managing around 900 active shipments at any given time — ocean containers, air freight, road legs — spread across roughly forty carriers. I built the integrations, the event model, and the monitoring design.

The problem

Each carrier reports status its own way: some have APIs, some have tracking pages, some only send emails, and a few do all three, inconsistently. Keeping the transport management system current meant coordinators visiting carrier sites one by one, reading the latest event, and typing it into the TMS. Nobody could do that for 900 shipments daily, so in practice the TMS was always partly stale.

Customers knew it, which is why "can you check on my container?" emails arrived constantly — each one triggering exactly the manual lookup the TMS was supposed to make unnecessary. The expensive version of the problem was delays discovered late: a container sitting unnoticed at a transshipment port for four days can turn into demurrage charges and a missed delivery window that a two-day head start might have prevented.

The automation I built

There was never going to be one integration pattern for forty carriers, so I built three ingestion paths and a common core. Carriers with APIs got direct integrations — the reliable, preferred route. Carriers with only a tracking page got Playwright-based web automation, polling on a schedule tuned to the shipment's leg: an ocean crossing doesn't need hourly checks, but a container due at port today does. Carriers that send status emails got a parser watching a dedicated mailbox.

All three paths feed a normalization layer, which was the real work. Every carrier names events differently — "Discharged," "Unloaded from vessel," and "POD arrival" can all mean the same thing — so raw events map to a canonical milestone model, maintained as a data table rather than code so new mappings don't need a deployment. Normalized events are written to the TMS with full history, and a delta detector watches for changes worth acting on: a slipped ETA raises a delay alert for the ops team, and key milestones trigger customer notification emails built on the forwarder's own templates.

Just as important was deciding what not to automate. Shipment volume follows the usual curve — a dozen carriers cover most of it. We integrated the head of the distribution properly and left the long tail manual, with those shipments clearly flagged in a checking queue. Automating a carrier the client uses twice a year would have been effort spent where it mattered least.

• Coverage of ~40 carriers through a mix of REST APIs, web automation, and email parsing
• A canonical milestone model that makes events from different carriers comparable
• Polling schedules tuned per carrier and per shipment leg
• Automatic TMS updates with a full event history for every shipment
• ETA-change detection with proactive delay alerts to the operations team
• Customer notification emails on key milestones, using the forwarder's templates
• Per-carrier health monitoring — a broken integration routes its shipments to a manual queue instead of going silent
• Nightly data-completeness report across all active shipments

What made it hard

With forty carriers, something is always broken. Carrier websites change without notice, and a scraper that fails silently is worse than no scraper — the shipment just goes stale while everyone assumes it's covered. So failure handling got more design attention than the scraping itself: every carrier integration has a health check, and when one goes down, its shipments move automatically to the manual queue and the team is told. Coverage degrades visibly instead of silently.

Timezones were sneakier than expected. Carriers publish event times with no timezone, in local port time, in UTC, or inconsistently between their API and their own website. An event that appears to happen before the previous one breaks ETA logic in confusing ways. The fix was a per-carrier interpretation rule set, plus a validator that quarantines chronologically impossible events for review instead of writing them to the TMS.

A couple of carriers actively resist automated access to their tracking pages. Rather than escalate a technical arms race — fragile, and not a game worth playing — we switched those carriers to the email-notification path, which they happily support.

Results

The daily portal-checking grind is gone, and the TMS finally tells the truth:

• More than 30 staff-hours per week returned to the operations team
• Customer status-request emails dropped by about half once proactive notifications went out
• Delays are flagged on average two days earlier, cutting demurrage and detention surprises
• TMS milestone completeness rose from roughly 70% to 97%
• Coordinators each handle more shipments without the daily checking routine

Business impact

Some benefits land directly on the invoice: demurrage and detention charges avoided because a delay was caught early are hard savings, and they accumulated quickly. Others compound more quietly. Proactive notifications changed the customer relationship from reactive to informative — several customers commented on it unprompted. And a fully populated TMS produced something the client hadn't asked for: enough clean historical data to compare carriers on actual transit performance, which now feeds their rate negotiations. A small, unplanned piece of digital transformation.

Lessons learned

In enterprise automation across systems you don't control, resilience beats elegance. The scraping code here is ordinary; the value sits in the health checks, the fallback queue, and the refusal to fail silently. I also came away convinced that partial coverage, honestly flagged, beats total coverage badly maintained — the 80/20 carrier decision felt like a compromise at the time and turned out to be the correct architecture.

Stack