Setup a Video Streaming performance test from the AWS Cloud with UbikLoadPack & Redline13

on 1 October ABR-Streaming, Apple HTTP Live Streaming, AWS, CLOUD, DEVOPS, HLS, JMETER, LOAD_TESTING, MPEG-DASH, PERFORMANCE, REDLINE13, SMOOTH Streaming, VIDEO and Tags: , , , , , , , , with 0 comments

Setup a Video Streaming performance test from the AWS Cloud with UbikLoadPack & Redline13

In this blog, we will show how to run a distributed performance test from AWS using:

  • JMeter + UbikLoadPack as the load injector software allowing to realistically simulate Video Streaming Players (HLS, DASH, SMOOTH, HDS)
  • Redline13 as the web SAAS platform to:
    • easily deploy JMeter and UbikLoadPack
    • monitor your performance test during runs
    • get detailed JMeter HTML report after run including UbikLoadPack dedicated metrics and graphs
    • historize your runs
  • Amazon AWS as the platform for load injectors

The JMeter setup

The JMeter Test plan is very simple:

We have a Thread Group where duration, threads, ramp-up and startup delay have been made variable, this will allow us to control them from the Web Interface of Redline13:

The Thread Group contains ULP Video Streaming Sampler which uses the stream and bitrate variables coming from CSVs and durationRead, conn_timeout and read_timeout defined in UDV_ENV (User Defined Variables):

And 2 CSV files to be able to vary Streams and Bitrates:

  • profiles.csv (loaded by JMeter CSV DataSet CSV-Profiles) contains the bitrates we want to test, this element will fill in ${bitrate} variable on each iteration of CSV with a different value

  • streams.csv (loaded by JMeter CSV DataSet CSV-streams) contains the streams URL suffixes we want to test, this element will fill in ${stream} variable on each iteration of CSV with a different value

AWS setup

AWS setup is very easy and described here.

Redline13 Setup

Note you need a paid subscription in order to have access to some of the features described below.

The configuration is pretty straight forward and is done in one page.

Click Start a New Test:

Start Redline13 test

Select “Jmeter Test” Tab:

  • In Upload JMX , select your JMeter jmx file
  • Upload the UbikLoadPack license file you received by mail after requesting a trial at this page 
  • Upload the 2 CSVs (profiles.csv and streams.csv)
  • Select JMeter version (the most up to date one)

Configure “Advanced Cloud Options” options:

Ensure you fill:

  • Location, in the below example we used Ireland and Virginia as a first step
  • Number of servers
  • Size (type of server)
  • Subnet ID (just click in the field and it’s automatically filled)

In the below example we use 4 Machines:

  • 2 M5.2XLARGE in Virginia
  • 2 M5.2XLARGE in Ireland

AWS Cloud configuration of Agents for OTT Load test

Configure “Advanced JMeter Test” options

  • In “JMeter Plugin support”, check “JMeter Plugin Manager” option
  • Search for our plugin using “UbikLoadPack Video” test in “Plugins from JMeter Plugins”
  • Click on the version of plugin you want to use, 7.1.7 at the time of this blog writing: 

  • You should end up with this:

  • Check plugin “JMeter Report” and fill in the APDEX thresholds you want (200 ms for Satisfied, 400ms for Tolerated in our case)

  • In “Option String” , adjust JMeter properties as you would from command line. For the UbikLoadPack plugin you need to set those ones:
    • For license:
-JULP.videostreaming.licensepath=ubik-streaming-plugin.license
    • For fields saved and needed for the report:
-Jjmeter.save.saveservice.sent_bytes=true 
-Jjmeter.save.saveservice.connect_time=true
-Jjmeter.save.saveservice.subresults=true
-Jjmeter.save.saveservice.url=true
-Jsample_variables=aws_region,bitrate,ulp_buffer_fill,ulp_lag_time,ulp_play_time,ulp_lag_ratio,ulp_lag_ratio_wo_bf,ulp_dwn_time
    • For configuration of test plan used for demo, we will set the properties used by test plan (notice the test plan uses JMeter __P function
-Jduration_read=60 -Jrampup=120 -Jduration=600 -Jthreads=1000

 

You end up with:

      • In the above line , you may need to adjust parameters in bold depending on:
        • Number of Players you want to simulate per machine (threads)
        • Duration of your test (duration)
        • Rampup of load test (rampup)
        • Duration of video read in your test (duration_read property)
  • In JVM_ARGS, you adjust JVM options , example:
    • -XX:+UseG1GC -XX:MaxGCPauseMillis=50  
    • -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false 
    • -XX:G1HeapRegionSize=32m 
    • -Xss256k -Xms6g -Xmx6g -XX:MaxMetaspaceSize=256m 
    • -Djava.awt.headless=true
    • In the above line , you may need to adjust parameters in bold depending on:

      • type of AWS instance and its memory (-Xms6g -Xmx6g)

Give a name to your test and click Start !

The test is now running, you can monitor it in the GUI and view while test is running:

  • Request metrics (Requests per second, KB Per request, Avg response time)

  • Error metrics(Errors per second, KB Per error, Error avg response time)

  • Agent metrics (CPU usage, Network In/Out)

  • The error table

Analyzing the test

Once your test is done, at the end of the test:

  • you get an overview of your test in Summary
  • you have the details of the run (JMeter and Cloud options)
  • you can add Notes and comments on your test
  • you can download jmeter logs from every server
  • More importantly:
    • you can download the HTML JMeter report generated accross all server . Note that this report will contain starting from version 5.0 of JMeter the custom graphs for :
      • Statistics (response time in percentiles, error rate, network…) by type of request by type:
        • Manifest
        • Chunk
        • Subtitles
      • lag ratio graph
      • lag ratio without buffer fill graph
      • buffer fill graph
      • lag time graph
    • you can download the merged CSV accross all server that contains the custom metrics

 

About UbikLoadPack:

You might want to read: