Dynamic Image Serving: imgproxy on AWS ECS
Improving developer experience and decreasing our image payloads by 93%
As part of user experience optimization here at Wowa, we implemented responsive images on our platform, fine-tuning image sizes to save 93% on total bandwidth spent on image payloads on our homepage. In order to deliver these images, we had to find a simple way to resize and compress both existing and future image files. We could have pre-generated versions of our static files and added a handler into our API to generate alternative versions of uploaded user files, but this would have significantly increased the complexity of implementing any future changes. If we defined three responsive image sizes now, but wanted to move to four in the future, we would have to change up our backend and run through our entire image store again and hope for the best.
The simplest solution would be to use a dynamic imaging server. There are self-hosted open source and third-party platforms such as Cloudinary, thumbor, imginary, and imgproxy. We settled on imgproxy for two main reasons:
1. Simplicity of setup: They had a fully functional docker image ready to go.
2. Speed. Benchmarks showed that it had the fastest performance at the lowest resource usage.
3. Open-source. We wanted something that we could control and would scale with us.
Imgproxy provides a ready-to-go docker container which we used AWS ECS to host. I will post a guide on how to do so in the next week, so stay tuned!
In order to gain connectivity to our imgproxy cluster, AWS offers two options: private service discovery and the use of an Elastic Load Balancer to route requests. Private service discovery allows you to assign a private namespace through Route 53 to your cluster and allows you to access the cluster through an easy identifier, but limits access and discoverability to other AWS resources within the same VPC. An Elastic Load Balancer, on the other hand, allows you to route requests either directly from Route 53 or behind Cloudfront.
Utilizing Cloudfront CDN to address caching
Implementing dynamic image resizing can significantly increase latency, especially at a high number of concurrent requests, as resizing and converting images is a non-trivial task. If we processed images at every request, we would lose any advantages gained from a user experience perspective.
To solve this problem, a cache should be used to store previously processed images. While imgproxy offers built-in modules to store processed images in the memory or filesystem, the easiest way to achieve this would be allow CDNs to do what they do best: cache. Placing Cloudfront in front of our imgproxy cluster allows it cache our processed images in its edge datacenters and decrease the latency and load caused by requests.
But what if the image changes?
If you use hash-based file names or versioning, this won’t be a problem as any references to updated images would generate a new URL.
Securing Access to Our imgproxy Cluster
The low number of available concurrent requests means that our imgproxy is highly vulnerable to DDoS attacks. Anybody could use Postman and a link to an extra large image file to cripple the cluster. To secure access to our imgproxy cluster and ensure only authorized use, imgproxy implements and recommends for production signature-based hashing using a secret key and secret salt. The endpoint is hashed and the signature is added directly into the URL.
Unfortunately, this implementation has a striking vulnerability: the secret key and salt have to be stored in our frontend code in order to dynamically generate image endpoint URLs. A determined hacker could go into our code and find the key and salt and sign their own malicious requests. While we could generate the URLs during compilation (for static images) and from our backend (for dynamic user-uploaded files), there are no tools available for the former and the latter would increase the complexity of our backend. Prepack by Facebook seemed like a great concept and may in the future be able to pregenerate static image URLs during build-time, but it’s no longer updated and would still leave the problem of our dynamic images (and even worse, dynamic sizing!).
Rather than using a hashing method, we would love if imgproxy could restrict source image URLs to certain domains or sources. For example, we would define a list of allowed domains:
https://example.s3-website-us-east-1.amazonaws.com
https://example.com
https://other-example.comAnd only images from those sources would be allowed. While this doesn’t prevent exploiting image parameters to cause excessive processing, it does limit the potential damage that can be done. In combination with origin restrictions or CORS, this could prevent a majority of low effort attacks.
WebP Automation
If you’ve used Google Lighthouse or the audit tool in Chrome Dev Tools, you’ll find that Google really really wants you to start using .webp formats for your images. Without legacy browser support for .webp, however, you’ll need to add a fallback. This is simple enough for a single image, but when you use responsive images you’ll end up with a huge chunk of code.
<picture>
<source type="image/webp"
srcset="images/example-300.webp 300w,
images/example-600.webp 600w,
images/example-800.webp 800w,
images/example.webp 1000w"
/>
<source srcset="images/example-300.jpg 300w,
images/example-600.jpg 600w,
images/example-800.jpg 800w,
images/example.jpg 1000w"
/>
<img src="images/example-300.jpg"
alt="Some Picture Description" />
</picture>imgproxy supports returning .webp images by default if no specific encoding is specified and the user’s browser sends an “Accepts” header that includes webp as one of the supported formats. This allows you to reduce the complexity of your code and abstracts away the handling of multiple file formats to imgproxy.
Simplifying Developer Coding
With imgproxy, it is now trivial for our developers to use responsive images. We created a function that accepts either a relative (e.g. /static/some-image.jpg) or absolute URL (https://example.com/some-image.jpg) and returns an imgproxy URL. This allows our developers to quickly code multiple responsive image formats without worrying about generating the image files and lets them quickly iterate through different resolutions. We were also able to remove legacy code from our backend that was used to pre-generate mobile images.
Further Optimizations
At Wowa, we’re always looking to improve. If you have any suggestions or alternative implementations, please let us know in the comments!
