Easy and realistic Load Testing of HTTP Live Streaming (HLS) with Apache JMeter

on 12 janvier 2015 DEVOPS, HTML5, JMETER, LOAD_TESTING, MEMORY, PERFORMANCE, PRODUCTIVITY, TUNING, VIDEO and Étiquettes : , , , , , , , , with 3 comments

Easy and realistic Load Testing of HTTP Live Streaming (HLS) with Apache JMeter:

Context:

Videos today are ubiquitous. Nearly every website shows videos, particularly those built for Mobile devices.

The most efficient way to serve video from website and even more for Mobile devices is to stream it as users don’t need to download the full video to start playing it.

Another motivation for streaming is performance and network bandwidth use. As a phone user for example, you don’t want to have to restart your full download because it was interrupted by a phone call or any other simultaneous activity. On the server side, having your users download « progressively » is a big performance benefit.

One of the most popular format for streaming was RTMP (by Adobe), but for iOS this format is not supported. Apple created HLS (HTTP Live Stream) which is becoming increasingly popular even outside of Apple world thanks to new services based on HTML5 Video.

Ensuring you serve videos correctly is a critical matter to ensure infrastructure performance and stability.

HLS (HTTP Live Stream):

HLS has become a standard nowadays for Apple devices, IOS devices but also Desktop. Solutions for playback of HLS on other platforms mostly rely on third-party plug-ins such as Flash or QuickTime. However, the recent development of THEOplayer by OpenTelly now also allows playback of HLS directly on HTML5 platforms without the need of any third-party plug-ins.

Apple initially created HTTP Live Streaming (HLS) to easily deliver audio and video content without server software. As an HTTP-based media streaming protocol, HLS uses regular web servers to send both live and pre-recorded content through a variety of streams at different bit rates, adapting to different network bandwidths. To do so, it breaks the media stream up into shorter sequential downloadable file components of their overall streams.

HTTP Live Streaming architecture

Being HTTP based, HLS can bypass firewalls or proxy servers and can easily be hosted by Content Delivery Networks.

You can read this overview for more in-depth understanding of protocol.

Load Testing of HTTP Live Streaming (HLS) with Apache JMeter:

Load testing HLS realistically is a challenge whatever the tool you use.

For Apache JMeter, whenever you want to load test a video infrastructure serving videos, you can find  blogs showing how to combine Regexp Extractor and HTTP Request to simulate load on servers:

 

 

Unfortunately, these methods have the following drawbacks or limits which will will be at the same time the opportunity to show the features of our plugin:

Unrealistic simulation

Players have a complex behavior in order to reduce pause times (Lag) during the play.

Few examples:

  • They select the playlist depending on the bandwidth
  • They initially buffer few seconds before starting to play
  • They download data while video is being played

Another problem is that your test won’t be able to limit the bandwidth for some of your users based on their network type.

None of the blogs you find tackle these challenges, handling those in JMeter is impossible unless you spend a lot of time on custom code.

Time consuming to create and maintain

Creating an « imperfect simulation » will take you some time and skill as you need to:

  • Master Regular Expression Extractor
  • Ensure you handle correctly pauses
  • Select correctly the playlist depending on bandwidth

And of course you will need to customize it whenever playlist changes.

Finally note that these blogs don’t show how to Load Test playlists that use EXT-X-BYTERANGE to specify byte range media segments, while this tag is now part of the new style playlists and recommended by Apple as it makes it much easier to create playlists and drastically reduces the number of chunks and files to manage on server side.

Metrics and analysis

Once you have created and run your script you have regular Apache JMeter metrics like response time, latency, …

Unfortunately for videos, these metrics are too limited to ensure User experience is OK. Indeed it is very hard to have the following critical informations :

  1. After how many seconds did player start to play video ? Do you want your customers to fly away because your video take too much time to load ?
  2. Did users face pauses during the play ?
  3. What format was chosen by your client ? HD, Medium size or only audio ?

Limited scalability

The described approaches usually use regular JMeter HTTP Request. This sampler is not made for video but for text responses, as a drawback it will store in memory each video chunk (which size can be quite big)limiting the scalability of your script and reducing drastically the number of Virtual Users you can simulate by each JVM.

In this blog I will present an alternative method using Ubik Load Pack Plugin for HLS.

Limited scope

URLs for HLS Streaming can be much harder to analyze than the illustrated examples, so you’ll spend a lot of time finding your regular expressions and building ones that covers all possibilities

UBIK LOAD PACK HLS Plugin

The good news is that there is already a solution for this, and it’s called UbikLoadPack Video Streaming plugin.

Video Streaming plugin is a commercial plugin from Ubik Load Pack solution.

Current version used in this blog is version 1.5.2.

Easy installation:

1) Unzip bundle

To install plugin just unzip ubik-hls-plugin-XXX.zip.

  • Copy lib/ext/ubik-jmeter-hls-plugin-X.X.X.jar to jmeter/lib/ext folder
  • Copy lib/groovy-all-X.X.X.jar to jmeter/lib/ folder (only for JMeter < 3.0)

2) Edit JMeter configuration

Put your license file in a folder of your choice, we will reference this folder as LICENSE_FOLDER, this folder must contain:

  • ubik-hls-plugin.license

Edit user.properties and add the following properties:

#license configuration

ULP.hls.licensepath=<LICENSE_FOLDER>/ubik-hls-plugin.license

#export metrics

sample_variables=ulp_buffer_fill,ulp_lag_time,ulp_play_time,ulp_lag_ratio,ulp_dwn_time

Edit “saveservice.properties” and add at end of file:

_com.ubikingenierie.jmeter.plugin.hls.result.HLSSampleResultConverter=collection

You’re ready to play !

Easy configuration and use:

To simulate video download you need to find the URL of the video and use ULP HLS Sampler:

 

ULP_HLS_TEST_PLAN

 

In this example :

  • video is http://download.oracle.com/otndocs/products/javafx/JavaRap/prog_index.m3u8
  • select Network type or select CUSTOM and enter a value in “Bandwidth”. Bandwidth can also be a variable (extracted from CSV for example ${bandwidth})
  • setup a timeout if you want to stop playing video after some predefined time or leave it to “-1” if you want to wait full download

Plugin will do all the complex stuff we described before for you 🙂

You’re nearly done

 ULP_HLS_SAMPLER

Rich metrics for User experience analysis:

Plugin computes and exposes 5 additional metrics.

These metrics are made available as JMeter Variables using syntax ${variable}:

Metric

Variable name

Description

Buffer Fill time

ulp_buffer_fill

This is the time user waits before video starts playing. During this time he gets a progress roller. During this time the first seconds of the video are downloaded. It’s an important metric to check as users may not wait minutes for your video.

Lag Time

ulp_lag_time

When you face issues, network contention or overloaded server, at some time during the play of the video, there won’t be enough data to play by the player, so it will stop playing and wait for data to be buffered. This negatively impacts your user experience so you need to ensure that this lag time is acceptable regarding your SLA. In ideal case, lag time should be equal to buffer fill time.

Play Time

ulp_play_time

the real time video took to play provided user waits for lags

Lag Ratio

ulp_lag_ratio

Lag ratio is lag time / video total duration. So if user waits 30s for a video of 5 minutes. Lag ratio would be 10%

Download Time

ulp_dwn_time

Download time

These metrics are output in the 2 formats provided by JMeter:

  • CSV
  • XML

Validate your sampler

Plugin being fully integrated with Apache JMeter 2.12, you can use standard JMeter assertions to make checks on responses.

Size Assertion to check Size of downloaded data

With this kind of assertion we can make checks on downloaded bytes.

In the below example, we check for example that size is equal to a variable ${sizeInBytes} which in this case is extracted from Csv DataSet:

ULP_HLS_SIZE_ASSERTION

Note of course you can check different “Type of Comparison”

Response Assertion to check one of the metrics exposed

In this below screen we check that ${ulp_lag_time} is equal to ${ulp_buffer_fill} which means we didn’t get any pause during play of video.

ULP_HLS_RESPONSE_ASSERTION

JSR223 Assertion using Groovy

In this below screenshot you can see script checks that “ulp_play_time” is >= 110000 ms (110s) and if not makes the sampler fail:

if(vars[« ulp_play_time »].toInteger()<110000) {

   // If ulp_play_time is < 110 seconds in this case it’s an issue

   AssertionResult.setFailure(true);

   AssertionResult.setFailureMessage(« video play time should be >= 110s »);

}

ULP_HLS_JSR223_ASSERTION

Other features:

Realistic use of network

As you can see in screenshot, plugin let’s you input a bandwidth (bits/second) in HLS Sampler.

ULP_HLS_BANDWIDTH_SELECTOR

With this feature plugin ensures that a Virtual User with some kind of network (3G or EDGE) will not download at a speed of a User having a 4G Network making test realistic.

Using this feature and JMeter Throughput Controller,(in this below screenshot controller is configured to sample it’s children 30% of the time)

JMETER_TC

you can easily create a test where you split your users into different network populations for example:

  • 50% using 4G netwok
  • 30% using 3G network
  • 20% using Edge network

CDN Compatibility

Thanks to JMeter 2.12 new DNS Cache Manager feature, plugin is able to behave realistically in a CDN based infrastructure.

You only need to add DNS Cache Manager to your script.

Clear results

Plugin will create child SampleResults for each video SampleResult showing the full details of download allowing for better analysis.

You can see in below screenshot how plugin proceeded for downloading the video.

ULP_HLS_SUBRESULTS

ND;LR:

As you can see in the previous examples, using Ubik Load Pack plugin for HLS lets you Load Test video infrastructure in few minutes:

  • Comprehensive metrics for user experience analysis
  • Realistic simulation (Player and network)
  • No coding skills needed
  • Performance and scalability
  • « Apache JMeter » like philosophy which makes plugin very easy to use, no specific training needed
  • Full compatibility with Core JMeter
  • CDN compatibility thanks to JMeter 2.12 DNS Cache Manager
  • Our plugin runs on any cloud provider(Amazon WS, CloudWatt, Google Cloud, Microsoft Azure). This can be useful when you need to make a massive load test that consumes important network bandwidth.
  • Our plugin can also run on PAAS by our friends from Flood.IO, this can be useful when you need to make a massive load test that consumes important network bandwidth and don’t want to manage your own cloud servers.
  • Very competitive pricing as you only pay the plugin, not the free core based on Apache JMeter

So try it ASAP here. Trial is free and lasts 10 days.

About author:

Philippe Mouawad works as an Architect and technical expert for Ubik-Ingenierie where he leads among other things the development of UbikLoadPack a set of Commercial Plugins for Apache JMeter allowing to load test different protocols like HLS, MPEG-DASH, GWT, Java Serialization, Flex.
Philippe is a committer on the Apache JMeter project and member of the PMC since October 2011.
Finally he is a co-author of a french book on JMeter called Maîriser JMeter: Du test de charge à Devops.

About us:

3 Comments

    Comments are closed.