As Director of Systems Engineering for VMIX, it is my job—my department’s job—to scale our infrastructure to meet our clients’ needs, ensuring a high level of service availability and performance.
And each year that goes by, our online video platform usage grows at a pace that could give anyone an ulcer, especially if you don’t come up with a creative solution to scale.
One of the solutions VMIX implemented a little over a year ago was incorporating cloud-based storage for a segment of our infrastructure. After much R&D, we chose to partner with the Amazon S3 service. When we started incorporating S3 to help meet some of our storage needs, we were slightly concerned about the service’s reliability and performance. So our approach was to design a system that would leverage our classic storage model as a buffer or cache for short-term newer content, and utilize S3 for long-term older content.
One year later, I thought I would share our findings with actual data collected from Amazon S3, as well as from the VMIX platform.
The graph below shows three data points (data stored, data transferred in, and data transferred out) on a month-to-month basis. As you will see, we did not experience any limitations on the amount of data stored and the amount of data transferred in and out.
The graph below tracks how many individual objects VMIX stored on a month-to-month basis. Given S3’s object-store model, it was clear that having two buckets in S3 was more than enough to handle what we needed.
The graph below shows the number of transactions on a month-to-month basis. We found that every time we stored, deleted or retrieved an object the performance was very dependable.
In conclusion, with the right architecture, Amazon S3 is an excellent option to help you scale storage. Though performance isn’t perfect (in the past year we’ve seen three to four S3 outages lasting an average of 15 minutes each) we feel you can architect around that as we have.
During this initial period VMIX used Amazon for a segment of its storage needs only—we now feel confident increasing our use of S3 moving forward.