How Etsy tracks millions of packages -- from vendors all over the world using over 100 different shipping providers -- in an efficient and scalable manner.

At Etsy, we track millions of packages daily that get shipped through several shipping carriers worldwide like USPS, FedEx, DHL and several others. Daily updates, from whether a shipping label has been created to whether a package is out for delivery -- are in the order of millions.

Our goal with providing shipping tracking information to our members -- both sellers and buyers -- is to provide accurate, timely tracking data that answers the question “where is my package right now?”

Tracking packages is an example of a general class of problems that require getting frequent updates for large amounts of data. More importantly, as the Etsy marketplace grows, this data set becomes increasingly larger, posing the challenge of being able to flawlessly manage long term and sustained growth.

Using a combination of data sharding, sql, asynchronous processing, faux-queueing and batch processing, we’ve created an efficient, scalable, and resilient system built on existing infrastructure that needs minimal maintenance.

In this talk we will walk you through the problem definition, the architecture we built, and the metrics we use for monitoring the system.