on 12 May 2017 JMETER, JMETER-PLUGINS, LOAD_TESTING, Microsoft-Smooth, PERFORMANCE, PRODUCTIVITY, SMOOTH Streaming, VIDEO and Tags: ApacheJMeter, HLS, LOADTESTING, MICROSOFT SMOOTH STREAMING, SMOOTH STREAMING, Video Streaming with 0 comments

Load Testing Smooth Streaming Video with JMeter

The format

Microsoft Smooth Streaming is an adaptive video streaming format created by Microsoft and used to stream content through HTTP requests.

This streaming format was first used during the 2008 Beijing Summer Olympics. Like other adaptive HTTP streaming technologies (Apple HLS, MPEG-DASH, Adobe HDS …), it allows the user to enjoy the best quality for the media being played.

You can see the general process of adaptive HTTP streaming with this schematic :

The Microsoft Smooth Streaming can be used for Live streaming but also for Video-On-Demand (VOD) streaming (and of course, our plugin for Apache JMeter can handle both !).

Microsoft Smooth Streaming uses a Manifest file that contains all the informations about the media. The video and audio channels are de-multiplexed, which means that the support for multi-language will be easier if you use Microsoft Smooth Streaming.

Some metrics on Video Streaming

You still have some doubt about load testing your videos? Here are some numbers that will convince you :

81% of internet and mobile audiences watched more live video in 2016 than in 2015.
Live video is more appealing to brand audiences: 80% would rather watch live video from a brand than read a blog, and 82% prefer live video from a brand to social posts.
The average conversion rate for websites using video is 4.9%, compared to 2.9% for those who don’t.
The global VOD market in 2016 is about $16.3 billion
4X as many consumers would rather watch a video about a product than read about it.
73% of B2B organizations report positive ROI from video marketing.
23% of people who have been presented with a poor quality video experience would hesitate to purchase from that brand.
Companies using video enjoy 41% more web traffic from search than non-users.
Mobile video traffic accounted for 60% of total mobile data traffic in 2016.

All the datas are from livestream and cisco.

Why load test your video streaming servers ?

This is a legitimate question. And there are several reasons for that :

Nowadays, the page load speed is critical for customers experience. The longer a customer waits for the page to load, the higher are the chance that he will leave your website. The problem is the same (and might be even bigger) with videos. If a customer can’t watch a video that describes your main product on your website or faces lags or very long initial buffering, because the traffic is too high on it you will loose a potential buyer. See an example on how load time impacts website abandonment here.
Same issue if you didn’t provide the correct format for his device.
Load testing your servers allows you to see what an end user will receive under circumstances that you can customize. You will see if the user will face some lags, if the initial buffering takes time, etc … And with those results, you will be able to fine tune your servers to improve the user experience.
Secondly, the test will tell to you whether your server infrastructure is able to handle the load incurred by the users traffic. Maybe you need to add another server, add more RAM to your existing servers to avoid lag …
As you can also see here, it’s quite important to correctly tune your server, our plugin can help you ensure your server serves content with optimal Lag Ratio.
With streaming, a positive user experience and sustained audience engagement is highly correlated with video playback performance.

Why do you need our plugin ?

Video Streaming load testing is very complex for at least 2 reasons:

Scripting many different format, the most frequent ones being Apple Http Live Streaming, MPEG-DASH, Smooth. Doing that requires a lot of time, skills and has a cost
Simulating how a Player downloads the streams is even harder if not impossible with OOTB JMeter

The new version of our Video Streaming plugin for JMeter add the support of Microsoft Smooth Streaming format to existing Apple Http Live Streaming and MPEG-DASH !

Moreover, it will realistically simulate how a player would stream the media files from your servers

No more scripting work on your side, the plugin will do it for you, along with a simulation that plays the media.

Even more, it will provide you precious metrics to understand the Real User experience, about the simulation, that you can use to do assertions with JMeter.

Read this blog for more on those metrics.

Testing this new format in Apache JMeter with our plugin is a matter of few minutes as you only need to provide the url of the manifest file, the plugin will auto-detect the format used (Apple HLS, MPEG-DASH or Microsoft Smooth Streaming) and will process it accordingly.

How to create a valid load test ?

To create a valid load test, you need to follow those steps :

Script
- It’s unfortunately not a matter of record and playback approach ! If you do that, it will simply request the same chunk with the same quality on the same content.
- Without our plugin, you need to manually do all the work, what does it means:
  - Ensure it downloads correctly chunks for all the possible video streaming format. Every format has its own syntax, so extraction of every possible format is a big work
  - Request different media qualities
  - Ensure you don’t always download the same media
  - Add the required pauses to ensure you act like a player
  - Finally compute the video related metrics (see below)
- The good news is that our plugin does all that work for you. It is built to do relevant tests
- It’s just up to you to insert the good assertions, to check the parameters that you want to improve etc …
Run
- Once everything is ready, you can start to run your test !
- Ensure you follow JMeter best-practices
Analyze
- During the test, thanks to tools like Dynatrace, AppDynamics or New Relic you can monitor your servers to detect the root cause of your performance issues.
- And thanks to the custom metrics (described below) you’ll be able to improve the key metrics to ensure your users have a nice experience
- Now it’s time to work a little! With all the data you gathered, you will have to check if the test goes well, what part of it went wrong …
Improve
- With all provided metrics, you fine tune your server settings, add potentially some servers to your infrastructure, increase the network bandwidth of your hoster, encode differently your videos …

Sample Demo

First get a valid license HERE.

Execute the preliminary configuration steps :

1 – Unzip bundle

Unzip ubik-videostreaming-plugin-XXX.zip.
Copy lib/ext/ubik-jmeter-videostreaming-plugin-X.X.X.jar to jmeter/lib/ext fold

2 – Edit JMeter configuration

Put your license file in a folder of your choice, we will reference this folder as LICENSE_FOLDER, this folder must contain ubik-streaming-plugin.license file
Edit “user.properties” and add the following properties:

#license configuration

ULP.videostreaming.licensepath=<LICENSE_FOLDER>/ubik-streaming-plugin.license

#export metrics

sample_variables=ulp_buffer_fill,ulp_lag_time,ulp_play_time,ulp_lag_ratio,ulp_dwn_time

Edit “saveservice.properties” and add at end of file:

_com.ubikingenierie.jmeter.plugin.videostreaming.result.VideoStreamingSampleResultConverter=collection

3 – Launch JMeter !

Add a Thread Group Element under Test Plan

Add a ULP Video Streaming Sampler to your test.

Now we can start a debug test. Select a Microsoft Smooth Playlist URL, for example :

http://playready.directtaps.net/smoothstreaming/TTLSS720VC1/To_The_Limit_720.ism/Manifest

By clicking the “Show Playlist” button, a pop-up will appear, and show you all the bit rates available for the video:

In play options, depending on your test, you can read the entire video or only part of it.

For this example, let’s choose play only part of videos and enter 10 seconds.

In network options leave auto or choose a bandwidth you would like to test.

As you can see, a list with normalized values (that you can see here) is provided by the plugin.

Add a listener to your test, a view result tree for example (Right-click your test –> Add –> Listener –> View Result Tree).

Finally click the green play button in the bar above.

If you chose 10 seconds like we did, after that duration, you should be able to see your results in the View Results Tree.

You can see the manifest has been downloaded with several video chunks and notice the plugin has computed the metrics:

bufferFillTime
lag ratio
…

You can also do some assertions about the video. As you can see above the assertion that checks that Lag time is equal to buffer fill has failed as we had lags during play time.

We can also use a JSR223 Assertion (Right click on ULP Video Streaming Sampler –> Add –> Assertions –> JSR223 Assertions) to check if the response time for manifest is below 500ms, if the response time for chunks is below 5000s and if that “ulp_lag_ratio_wo_bf” is > 0,5%

In real-world example, URL, Bandwidth and Size would come from a CSV file and you can use a CSV DataSet to load it in JMeter and use it with the plugin.

All the assertions provided by JMeter can be used with the plugin !

Once you’ve checked all this, you’re ready to start your test.