In this tutorial, we continue building an integration between Shopify and Airtable and focus on managing Airtable’s API rate limits using a Google Cloud Task queue. In part 1, we set up a Shopify connection in Gadget and set up a client to forward product data to Airtable.
When integrating Shopify with Airtable, you may encounter rate limit discrepancies between the two platforms. Shopify and Airtable each have their own rate limits for API requests, and if not managed properly, this can lead to synchronization issues.
Airtable has a rate limit of 5 requests per second. Shopify also has API rate limits that vary based on the API and whether or not the merchant is part of the Shopify Plus program. Although Gadget manages Shopify’s rate limits when syncing historical data, we still need a way to handle Airtable’s rate limit. Forwarding data from Shopify directly to Airtable could exceed Airtable’s API rate limits, especially for Shopify Plus merchants, and exceeding Airtable’s limit results in a 429 response code and prevents you from making additional requests for 30 seconds. We definitely want to avoid a 30-second timeout.
That said, we don’t want to write a whole bunch of custom logic to manage Airtable’s rate limit and retry logic. Instead, Google Cloud Tasks can be used to set up a queue that will not exceed Airtable’s rate limit, ensuring that all our Shopify data is successfully stored in Airtable.
Google Cloud Tasks is a powerful tool that can help you manage the rate limit discrepancy between Shopify and Airtable. With Google Cloud Tasks, you can create a queue for tasks and control the flow of data between different services, ensuring that Airtable can handle the data being sent its way without issues.
Let's dive into the steps required to set up Google Cloud Tasks for your Shopify and Airtable integration.
Reminder: You should complete part 1 of this tutorial before taking the next steps.
Google Cloud Tasks offers a generous free tier, allowing you to get started without incurring any initial costs.
Now you need to enable the Cloud Tasks API on your project.
This limits the events emitted by the queue to match Airtable’s API rate limit of 5 requests per second. By queueing Shopify data events as tasks in the queue, you ensure that you never exceed the Airtable rate limit.
You'll need to create a service account for your Gadget application to interact with Google Cloud Tasks. This service account will handle authentication and task creation.
The service account should be created. You will also need to generate a private key for this account to use in Gadget as an environment variable.
This will download a JSON file to your local machine. This file contains a private_key field that will be used in Gadget.
Your Cloud Tasks project is now ready to queue up requests. You can make changes to your Gadget project to start using the queue.
To take advantage of both the Google Cloud Tasks queue and your Airtable client that was set up in Gadget, you will create a new HTTP route in your Gadget app. Your Cloud Tasks queue will call this route, which will make use of your Airtable client to forward Shopify data to Airtable.
This will just log any incoming requests. You can use it to make sure your Cloud Tasks queue is set up properly.
Configure the necessary environment variables for Google Cloud Tasks integration in your Gadget application. These variables include:
The target service URL is where you want the data to be forwarded. In this setup, it's your Gadget application's route, which will then forward data to Airtable. This <inline-code>TARGET_SERVICE_URL<inline-code> should look like the following:
Now you can modify your existing app to integrate Google Cloud Tasks.
You'll need to modify your Gadget application to work with Google Cloud Tasks. First, you need to install the Cloud Tasks client:
The client will be installed in your Gadget app. Now you can modify the <inline-code>shopifyProduct<inline-code> <inline-code>create<inline-code> action to push tasks to the queue instead of forwarding data directly to Airtable.
This code file sets up the CloudTasksClient and uses it to create a new task with your product data object as the payload. It makes use of the environment variables set up earlier, notably the <inline-code>TARGET_SERVICE_URL<inline-code>, to queue up the data and forward it to your HTTP route, which should write the request payload to your Gadget logs.
Test this out. You can create a new product in your Shopify dev store admin. There is a 10-second delay built into this sample code. This gives you enough time to view your task in the Cloud Tasks queue, if desired. Once the queue has pushed the request, you should see an entry in your Gadget Logs so you know the HTTP route is being called successfully.
Once you see the log message, you can use your Airtable client in the HTTP route to forward data to Airtable.
This code imports your existing Airtable client from Part 1 of this tutorial and uses it to forward product data to Airtable when you create a new Shopify product.
Now you can test out the complete app. Create another product in the Shopify development store admin and, after being queued up by Cloud Tasks, you will see your data appear in Airtable.
Congrats! You have successfully built an app that handles Airtable’s rate limits when forwarding data from Shopify. It‘s now possible to run a historical data sync on your Shopify products so that all existing products are synced to Airtable.
You may want to handle more than just the <inline-code>shopifyProduct<inline-code> <inline-code>create<inline-code> action using Cloud Tasks. To do this, you could either add a unique route for each webhook or action you want to cover or add an additional field to the object sent to Cloud Tasks that contains information about the action. This means that you could use a single HTTP route to handle incoming data for all of my product actions.
Integrating Shopify with Airtable can provide a powerful solution for managing your ecommerce business's data. However, ensuring that data synchronization works seamlessly is essential. By incorporating Google Cloud Tasks into your integration, you can effectively manage rate limits and guarantee that your data moves between platforms without a hitch. This results in a more efficient and reliable integration for your online store.
Stay tuned for Part 3 of this tutorial where changes are synced from Airtable back to Shopify.