Valentin KiselevBackend EngineerTravis TurnerTech EditorComplex file Processing can be made easy with serverless solutions, but when it comes to performing various manipulations on the uploaded files, you also need a robust system that’s easy to integrate. Yet, if you want to create a complex service, you’ll need to know some file processing tools and techniques. To illustrate, we’ll look at Playbook, a platform that handles tons of files every day, and, as a bonus, you’ll see an example serverless app for Google Cloud Platform with a small framework for asynchronous file processing and storing of the results.Let’s explain complex file processing with an example: Ben opens his favorite browser and navigates to your site, a web app for sharing photos. He uploads a picture of his dog and sees a small thumbnail, GPS location, the image dimensions, and an AI-generated description of the picture.In essence, this is file processing in action: this can be straightforward, like showing EXIF info and file size, or it can be something more sophisticated, with image recognition, scaling, color adjustment, and many other AI-inspired operations that can stretch beyond the imagination.However, for those seeking to build a complex service like that, you’ll need to have some file processing tools and techniques in your toolkit.Additionally, you’ll want your users to be happy and you’ll want them to stay happy, too. This is possible when a solution is fast, cheap, and extensible, so it’s worth it to thoroughly think out the infrastructure.It’s very important to make file processing fast and stable if you’re planning to sell it as one of the main features of your application.It’s possible to build a unique system based on modern technologies (like the NoSQL database, retriable queues, and asynchronous code execution) and there are a lot of solutions for every task, you just have to connect them in one app.Serverless functions to the rescue!These functions encapsulate different APIs and technologies within a common interface, and this approach makes development faster and prevents obvious mistakes that would otherwise cause frustration when trying to connect a bunch of services within an app.Let’s talk about the good stuff first: with a serverless solution, the actual code of your app can be simpler since you don’t need to worry about all the peripheral logic like authentication, authorization, or server auto-scaling. Instead, you can just focus on what matters for a particular function; you write the function code and tell the serverless provider when and how to execute it.Naturally, you also don’t need to pay for server wait time because you’re not implementing a server. Instead, you only need to pay for the resources allocated when your code actually runs.This approach can be cheaper because, unlike a dedicated server solution, it completely depends on usage.Further, it’s easy to fit a serverless solution into a provider’s pre-existing infrastructure: for example, you can receive events in AWS Lambda from AWS S3. AWS Lambda will call your functions directly after a file has been uploaded to your S3 storage. In other words, you don’t need to care about delivery because the cloud provider takes care of it for you.Some background: Playbook works with uploaded files and performs various processing, including AI-based image recognition, metadata parsing, file conversion, and thumbnail generation.Evil Martians helped the Playbook team extend and improve their existing file processing service, and we’re ready to share some of the knowledge we gained during this process.The file processing system is deployed as a serverless application that runs on Google Cloud Platform (GCP).Every file upload triggers a processing flow that does some magic with the files and stores the results. Later, these results are fetched and used in the application.Here’s the flow:To facilitate this, a small framework was built on the GCP services which provides a convenient and scalable file processing solution; each file upload triggers the processing pipeline.Here’s how this works:Our roundtable podcast featuring Jessica Ko, co-founder and CEO at Playbook.Once the processing has finished, it’s time to handle the results: the main application runs a Sidekiq worker that fetches the results via a cloud function call, queues up the Sidekiq worker that handles the results, and clears the processed results with one more cloud function call.If some functions haven’t yet processed the file, their results will be fetched on the next worker run. Each processing result is independent and complete, so it’s already safe to handle and clean it.Conducting file processing in Cloud Functions helped Playbook save resources from the main application and delegate processing resources management and scaling to GCP.This division also allowed them to encapsulate the processing logic and reduce the binding between the application and the file processing system.In our example, a small serverless application, we’ll write two functions: one that handles newly created assets on storage and another that processess them. If you want to look at the complete example repository you can find it here: github.com/mrexox/serverless-file-processing-exampleFor the processing, we’ll write a small framework that utilizes the Google Cloud Tasks API and Firestore database for running the processing jobs and storing the results. This will make our solution scalable, so we could call asynchronously lots of processing functions.We need to enable the APIs that we’ll use in GCP. So, in a blank project we’ll do the following in the Google Cloud Console:If something fails deployment later, it can probably be easily fixed by granting the correct permissions or enabling extra APIs.Let’s configure the serverless framework for use with GCP and TypeScript.We’ll store secrets in .env file for convenience and security.Google Cloud Functions only know NodeJS runtime, so we will have to use a special plugin serverless-plugin-typescript for converting the code into JavaScript.We’ll now implement two cloud functions: the processFile function will start the processing flow and parseMetadata function will do the processing:For the parseMetadata function, we’ll assume that there can be many different processing steps, so we can run some processes in parallel, and then enqueue a different cloud function for further processing based on our results.But for our small example let’s add just one simple processing step, parsing file metadata:We’ll build a small framework to allow our functions to be called asynchronously and to store their results somewhere.With this framework, we’ll be able to create any pipeline because it will support both piped and parallel flows.The piped flow is implemented by enqueueing a call to a cloud function from another cloud function.The parallel flow uses Promise to grab the results from async functions that run concurrently.The enqueue function uses Cloud Tasks API to do an asynchronous call to the Cloud Functions. It handles uncatched exceptions and retries the call in this case. This helps with accidental issues not related to the code (for instance, any uncaught network issues if we call other servicess from a cloud function).The processors.call function runs all steps in parallel using promises, then stores the results to Firestore database; these results can potentially be used to enqueue another cloud function.For the sake of simplicity, I’ve omitted the code for getGCSFile. You can go to the sources if you want to check the implementation.The stored results can be fetched later and used in the app. A scheduled Sidekiq worker can be used for this task. This approach allows to fetch the results in a batch and control the load to the database.The parsing is done by metadataParser function. It uses sharp package to get the metadata from a file assuming it is a media file.This is a simplified example of how file processing can be organized using serverless and Google Cloud Platform. The basic idea here is the processing framework: it’s scalable and capable of supporting quite sophisticated processing flows.A few moments after uploadeding the file through the Google Cloud Storage web interface, I see the following record in the Firestore database.Live file processing demonstrationI can fetch it later and delete as soon as my application handles the results. See the functions for fetching and deleting processed results in the example app on Github.At Evil Martians, we transform growth-stage startups into unicorns, build developer tools, and create open source products. If you’re ready to engage warp drive, give us a shout!Get all the new posts delivered directly to your inbox. Unsubscribe anytime.We transform growth-stage startups into unicorns, build developer tools, and create open source products.United States+1 888 400 548577 Sands St.Brooklyn, New York11201Portugal+351 308 808 570Rua Alexandre Oneill, 38,Porto4400-008Japan+81 6 6225 12429F Edobori Center Building, 2-1-1 Edobori, Nishi‑ku,Osaka550-0002Get all the new posts delivered directly to your inbox. Unsubscribe anytime.Designed and developed by Evil Martians