Setup an OTT Video Streaming load test with UbikLoadPack, JMeter and AWS

on 2 September 2018 AWS, CLOUD, DEVOPS, HLS, JMETER, LOAD_TESTING, PERFORMANCE, VIDEO and Tags: , , , , , , , , with 0 comments

Setup an OTT Video Streaming load test with UbikLoadPack, JMeter and AWS

Following our previous blog, we show here how to setup the massive load test using the chosen tools (UbikLoadPack, JMeter, AWS, Redline13).

The JMeter setup

The JMeter Test plan is very simple:

JMeter Test overview

We have a Thread Group where duration, threads, rampup and startup delay have been made variable, this will allow us to control them from the Web Interface of Redline13:

Players configuration

The Thread Group contains ULP Video Streaming Sampler which uses the stream and bitrate variables coming from CSVs and durationRead, conn_timeout and read_timeout defined in UDV_ENV (User Defined Variables):

UbikLoadPack Player configuration

And 2 CSV files to be able to vary Streams and Bitrates:

  • profiles.csv (loaded by JMeter CSV DataSet CSV-Profiles) contains the bitrates we want to test, this element will fill in ${bitrate} variable on each iteration of CSV with a different value

bitrate CSV Data Set configuration

  • streams.csv (loaded by JMeter CSV DataSet CSV-streams) contains the streams URL suffixes we want to test, this element will fill in ${stream} variable on each iteration of CSV with a different value

Stream CSV Data Set configuration

AWS setup

AWS setup is very easy and described here.

Redline13 Setup

Note you need a pro subscription in order to have access to some of the features described below.

The configuration is pretty straight forward and is done in one page.

Click Start a New Test:

Start Redline13 test

Select “Jmeter Test Tab”:

  • In Upload JMX , select your JMeter jmx file
  • Upload the 2 CSVs (profiles.csv and streams.csv)
  • Select JMeter version 4.0 (the most up to date as of July 2018). Use an upper one if available.

Test setup for JMeter for OTT load testing

Configure “Advanced Cloud Options” options:

Ensure you fill:

  • Location, in the below example we used Ireland and Virginia as a first step
  • Number of servers
  • Size (type of server)
  • Subnet ID (just click in the field and it’s automatically filled)

In the below example we use 4 Machines:

  • 2 M5.2XLARGE in Virginia
  • 2 M5.2XLARGE in Ireland

AWS Cloud configuration of Agents for OTT Load test

Configure “Advanced JMeter Test” options

  • In “JMeter Plugin support”, check “UbikLoadPack stream” option
  • Upload your license file
  • Check plugin “JMeter Report”
  • In “Option String” , adjust JMeter properties:
    • -LWARN -Jjmeter.save.saveservice.sent_bytes=true -Jjmeter.save.saveservice.connect_time=true -Jjmeter.save.saveservice.subresults=false -Jjmeter.save.saveservice.data_type=false -Jsummariser.log=true -Jsummariser.out=false -Jsample_variables=stream,aws_region,bitrate,ulp_buffer_fill,ulp_lag_time,ulp_play_time,ulp_lag_ratio,ulp_dwn_time,ulp_hits,ulp_avg_chunk_time,ulp_avg_manifest_time,pm-hit,pm-miss,cm-hit,cm-miss,chunk-hit,chunk-miss -Jduration_read=60 -Jrampup=120 -Jduration=600 -Jthreads=1000

      • In the above line , you may need to adjust parameters in bold depending on:
        • Number of Players you want to simulate per machine (threads)
        • Duration of your test (duration)
        • Rampup of load test (rampup)
        • Duration of video read in your test (duration_read property)
      • Notice we have additional metrics computed in our plan based on customer:
        • pm-hit,pm-miss,cm-hit,cm-miss,chunk-hit,chunk-miss
  • In JVM_ARGS, you adjust JVM options , example:
    • -XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:G1ReservePercent=30 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -XX:G1HeapRegionSize=32m -Xss256k -Xms8g -Xmx8g -XX:MaxMetaspaceSize=256m -Djava.awt.headless=true -Duser.timezone=Europe/London
    • In the above line , you may need to adjust parameters in bold depending on:
      • type of AWS instance and its memory (-Xms8g -Xmx8g)
      • the user timezone you want to use as reference for timestamps (-Duser.timezone=Europe/London)

Advanced JMeter Test Options

Give a name to your test and click Start !

The test is now running, you can monitor it in the GUI and view while test is running:

  • Request metrics (Requests per second, KB Per request, Avg response time)
  • Error metrics(Errors per second, KB Per error, Error avg response time)
  • Agent metrics (CPU usage, Network In/Out)
  • The error table

Analyzing the test

Once your test is done, at the end of the test:

  • you get an overview of your test in Summary
  • you have the details of the run (JMeter and Cloud options)
  • you can add Notes and comments on your test
  • you can download jmeter logs from every server
  • More importantly:
    • you can download the HTML JMeter report generated accross all server . Note that this report will contain starting from version 5.0 of JMeter the custom graphs for :
      • lag ratio
      • lag time
      • hits
      • average manifest time
      • average chunk time
    • you can download the merged CSV accross all server that contains the custom metrics

 

Summary and results provided by Redline13 for OTT Load Test

 

Try it:

Request a trial version HERE.

Ask for a turnkey or consulting service at ubik-load-pack-sales@ubik-ingenierie.com.

 

About us: