Shopify app developers all contend with one major issue: rate limits. Shopify’s APIs are heavily rate-limited to the point that every app must invest huge amounts of time into careful rate limit management just to get off the ground.

At Gadget, we run a full-stack app platform with a built-in Shopify integration that does this for you. Our goal is to handle all the infrastructure and boilerplate, including the gnarly bits of rate limit management and data syncing, so you can build useful features instead of fighting APIs. Our main strategy to avoid rate limit pain is to sync the data that you need in your app out of Shopify and into your app’s database, so you have unfettered access to a full-fidelity, automatically-maintained, extensible copy of the data. How much you sync and how often you sync is up to you.

Sadly, that means the rate limit problem stops being your problem and starts being ours. We’ve spent many years getting faster and faster at syncing, and recently shipped two big changes we’d like to share:

1. An in-memory streaming system that pulls data from Shopify as fast as possible and is consumed as a buffer independently.
2. A process-local adaptive rate limiter inspired by TCP’s AIMD (Additive Increase, Multiplicative Decrease) algorithm.

The result: faster syncs that saturate Shopify’s API rate limits without stepping on user-facing features or risking 429s.

Here’s how we did it.

The sync problem

Gadget syncs are used for three things:

1. Historical imports and backfills: For example, pulling in every product, order, and customer to populate the database when a shop first installs an app.
2. Reconciliation: Re-reading recently changed data to ensure no webhooks were missed, or recover from bugs.
3. No-webhook models: Some Shopify resources don’t have webhook topics, so scheduled syncs are the only option for copying data out.

In all these cases, developers really care about data latency – if the sync is slow, app users notice missing or mismatched data and complain. But syncing fast is hard for a few reasons:

• Shopify’s rate limits are very low. They just don’t offer much capacity, so you must use what you do get very carefully.
• Shopify will IP ban you if you hit them too hard. If you just blindly retry 429 errors quickly, you can pass a threshold where Shopify stops responding to your IPs, which breaks your entire app for as long as the ban remains in place. Gadget learned this the hard way early on.
• Foreground work competes – Syncs run while the app is still online and doing whatever important work it does in direct response to user actions in the foreground. We want background syncs to go fast, but not so fast that they eat up the entire rate limit and delay or break foreground actions.

The best sync would sustain a nearly-100% use of the rate limit for the entire time it ran, but no more.

Goldilocks zones

Say we’re building a Gadget app to sync product inventory counts to an external system like an ERP. A simple sync flow might be:

1. Fetch a page of products from the Shopify API.
2. Run the actions in the Gadget app for each product, which will send an API call to the ERP.
3. Repeat.

This approach has two major problems:

• If the ERP system is very slow, the sync will run very slowly, because we wait for it to respond for all the products before we move on to fetching the next page of data, leaving performance on the table
• If the ERP system is very fast, the sync can run so fast that it exceeds the Shopify rate limit, maybe dangerously so. If foreground work or other Shopify resources are being synced at the same time, we risk an IP ban.

This means our design criteria for our sync strategy must be:

• The rate at which we read from Shopify is decoupled from the rate at which we can write to external systems, so it can go faster and not wait each iteration.
• The rate at which we read from Shopify must be capped according to the current conditions so it doesn’t go too fast.

We have a porridge situation on our hands: not too fast, not too slow, but just right. Internally, we implemented this by decoupling the data producer (reads from Shopify) from the consumer (a Gadget app running business logic).

Streaming with backpressure

To do this decoupling, we built a simple in-memory streaming approach that reads data from Shopify into a queue as fast as it can, and then consumes from that buffer independently. 

Here’s how it works:

1. A while loop reads a page of data at a time from Shopify as fast as it can, adding to a queue.
2. Gadget’s infrastructure dispatches each unit of work to your Gadget app to run business logic.
3. If the consumer falls behind (because, say, an external system is slow), the queue fills up.
4. Once the queue hits a limit, the producer can’t add more data and is blocked, which prevents excessive rate limit consumption if the consumer is slow.

The producer can spam requests if the rate limit allows, and the consumer can take advantage of Gadget’s serverless autoscaling to process data as quickly as possible within the limits the app has set.

One might ask if it is really worth writing each individual record to a pub-sub queue system just for this decoupling property, and our answer at Gadget is no. We don’t want or need the pain and expense of running Kafka or Pubsub for these gazillions of records. Instead, we use a Temporal to orchestrate our syncs, and model the buffer as a simple p-queue in memory! 

Enter Temporal: Durable syncs with checkpoints

We use Temporal under the hood to run all syncs as complicated, long-running, durable workflows. Each Shopify resource that needs syncing is run as an independent Temporal activity that starts up and is run (and re-run) until the resource has been fully synced. If an activity crashes, times out, or we need to deploy a new version of Gadget, Temporal guarantees the activity will be restarted elsewhere. 

We then use Temporal’s durable heartbeat feature to track a cursor for how deep into the sync we’ve progressed. We use the cursor from the Shopify API for a given resource as our sync cursor. When an activity starts back up, it can continue reading from exactly where the last activity left off. If we’re careful to only update this cursor in Temporal after all the items in the queue have been processed, we can safely leave the queue in memory, knowing that if we crash, we’ll rewind and replay from only the most-recently fully completed cursor.

Adaptive rate limiting (Inspired by TCP)

So, we’ve decoupled producers from consumers. Now the question is: how fast can the producer safely go? Our answer is: it depends. Instead of trying to set a hard limit for the rate we can make API calls, we built an adaptive rate limiter inspired by TCP congestion control.

There are a few key reasons why we must be adaptive:

• Shopify has different limits per store, which you don’t really know ahead of time. Plus, merchants get much higher rate limits, and Enterprise merchants get even higher rate limits after that
• The rate limit conditions can change mid-sync, if another unrelated sync starts, or if the app has high foreground rate limit demand all of a sudden
• We run syncs in parallel (for example, products + orders + customers), and each synced resource contends over the same limit but takes a different amount of time.

Coordinating a global rate limiter across multiple independent processes in a distributed system is annoying and error-prone, as you need some central state store to share who is asking for what and when. It’s especially complicated when you try to account for different processes starting and stopping and wanting some fair slice of the available limit. Instead, we’d like something simpler, and ideally process-local, such that each participant in the system doesn’t need to communicate with all the others each time it wants to make a call.

Luckily, Shopify has implemented a state store for us, over the same communication channel we’re already using! When we make a call, they tell us if we’re over the limit or not by returning a 429. If we are careful not to spam them, we can use Shopify’s own signal to know if we should raise or lower the process-local rate at which we’re making requests.

This problem is very similar to the classic flow control problem in computer networking, and our solution is entirely copied from that world. Gadget’s syncs now throttle their rate limit using TCP’s AIMD (Additive Increase, Multiplicative Decrease) algorithm:

• If things are going well (no 429s), we slowly ramp up request volume.
• If we get a 429, we cut back hard (usually by half).
• Over time, this converges on the real usable rate limit for this process.

If the real usable rate limiter changes, because say a new sync starts and consumes more than before, each process will start witnessing more 429 errors, and will cut back its own process local rate, making room for the new process. If that new process finishes, each remaining process will start witnessing more successful requests and ramp their request volume back up to find a new equilibrium. The equilibrium is ever changing, and that’s the point.

Another great property of AIMD is automatic discovery of the max real rate limit for even single participants in the system, which means high rate limits for Plus or Enterprise merchants are automatically discovered without Gadget hardcoding anything. For example, if an app is syncing only one resource against only one high-rate-limit store, AIMD will continue to raise that one process’s local rate limit until Shopify starts 429-ing, allowing that one process all the resources Shopify will offer.

And finally, AIMD is tunable such that we can target an effective rate limit slightly lower than the real one, so we ensure that we leave rate limit room for foreground actions 

Our AIMD implementation is open source here: https://github.com/gadget-inc/aimd-bucket

Putting It All Together

With this new sync architecture, Gadget apps can:

• Ingest Shopify data at the fastest safe rate
• Avoid polluting Shopify’s API or causing foreground actions to fail
• Process downstream logic (like ERP integrations) at their own pace
• Process reliably in the face of failing computers

It’s fast, durable, and most importantly, something Gadget app developers don’t have to build or maintain themselves going forward, the way infrastructure should be.

Try It Out

These improvements are live today for all Gadget apps syncing Shopify data.

Most apps won’t need to think about it. But for apps installed on lots of Shopify Plus or Enterprise stores, the speedup can be massive. We’ve seen syncs go 4–5x faster on big stores with heavy product or order volume.

If you’re building a Shopify app and are tired of wrangling APIs, OAuth, HMACs, retries, or sync pipelines, check out Gadget.

We’d love your feedback, contributions, or bug reports, and we’re always working to make app development feel like less work.