OTT Case Study - Ensure streaming en performance quality

on 2 September 2018 AWS, CLOUD, HLS, JMETER, LOAD_TESTING, MPEG-DASH, OTT, PERFORMANCE, VIDEO and Tags: AWS, case study, CDN, CLOUD, DEVOPS, HLS, Load Test, MPEG DASH, OTT, UX, Video Streaming with 0 comments

How to ensure the best quality of experience during the re transmission of a major sport event around the world ?

By Load Testing video streaming services with UbikLoadPack, Redline13 and AWS Cloud

Recently one of our customer had to ensure his OTT and CDN services would handle correctly the load incurred by re transmission of a major sport event.

He wanted to check multiple things:

Customers user experience (no lags, accuracy of stream, no delay)
CDN caching policy (aggressive caching of parent manifest, low caching of child manifest)
Origin performance delivering child manifests and chunk on time
Load Test his OTT video streaming servers with up to 100 000 Players

His event was a worldwide one where customers would be watching it from:

USA
Latin America
Brasil
Europe (Germany, France, Belgium, Austria …)

So he contacted our professional services to help him ensure he could have this test done in 5 days.

Which tools to use and why ?

His OTT service was composed of a CDN fronting an origin delivering Live HLS and MPEG-DASH manifests, and his CDN was spread accross the world.

Since we needed to run a massive scale Load Test, we used:

UbikLoadPack Video Streaming plugin + Apache JMeter as the tool to simulate users watching Live streams delivered in HLS and MPEG-DASH
Redline13 as the service to run at high scale and distributed accross the world the load test. As a bonus, JMeter and UbikLoadPack Video Streaming plugin are available Right Out Of The Box on their service which would ease the piece of work that was constrained by the very short delays

UbikLoadPack Video Streaming plugin fits customer requirements in the following fields:

Reproduce realistically and very easily a player reading Live stream delivered in HLS, MPEG-DASH, MS Smooth and HDS. There is no need to write complex and unmaintainable JMeter test plan to reproduce the player behaviour, knowing that the multitude of formats and possibilities of Live Streaming makes this very complex and time consuming.
Provide custom metrics to analyze User watching video experience
Ability to add additional metrics thanks to some groovy scripting to analyze CDN caching and accuracy in delivering the child manifests
Ability to graph those metrics
Availability both on Premise and from the Cloud. For the short term, customer wanted to run the test from the Cloud but in the longer term, he also wanted to run those tests as close as possible to his streaming servers in his own Data Center

Redline13 fits customer requirements in the following fields:

Ability to easily deploy JMeter + UbikLoadPack with a single click from a web page on multiple servers in the AWS Cloud
Ability to start those servers from all over the world using AWS datacenters from a simple Web Page
Ability to get live feedback on load test
Ability to get the merged results and metrics in one CSV accross all nodes
Ability to get the JMeter report auto generated at end of test

The tools chosen were:

UbikLoadPack streaming plugin to validate OTT video streaming performances & Redline13 to distribute the high load across the World

Let’s see how simple it is to setup a test on Redline13 using UbikLoadPack.

The results:

Thanks to this load testing campaign , our customer was able to:

Run a load test with 100 000 Players from 10 AWS locations accross the world
Diagnose an issue in the caching policy that triggered lag in the playing. This was correlated thanks to the graphing of Ratio and the addition of Caching metrics that our professional services provided based on CDN headers
Diagnose performance issues affecting customers in the USA and South America
Diagnose 404 issues in the streams from certain locations
Provide his end-user customer metrics showing that their services was delivering streams as per requested SLR