UbikLoadPack : Advised tuning for injector when load testing Video Streaming servers
Recently one of our customer asked the following question on Stackoverflow:
What are the recommended configuration in terms of:
- System tuning
We thought that we should be answering this question with as much details as possible.
So here is the answer.
It is hard to answer such generic question as behaviour depends on at least following factors:
- The type of Stream : HLS will perform better than MPEG-DASH and HSS, the 2 latest being more CPU consuming due to XML format.
- The duration of the Manifest: This is the total duration it holds, the biggest it is the more memory and parsing it will require
- The refresh period : This is the period at which a Live Stream manifest will be reloaded by player, the shortest it is, the more it will consume memory and CPU
So the best advice is to calibrate based on your Manifests, meaning to run this for 50 to 100 threads, see how much CPU and Heap it consumes and then extrapolate.
Physical Machine or Virtual Machine:
- Ensure you have enough bandwidth in your data-center between injector and target servers
- You should usually load test as close as possible to target to avoid network parasitizing your results. And only when you’re sure your server are correctly handling the load load test from Cloud servers. But you can do the latter if you can’t do the first step.
- Ensure your network card is correctly configured
- With “Average Manifest” , you should have no issue load testing 700 to 1500 Threads (Virtual Users) with machine of type m5.2xlarge (8 vCPU / 16 to 32 Go)
- We advise you to use Linux/MacOS instead of Windows OS as you will get more Threads on same virtual/physical configuration.
- But all OSes work since Java is used
For Linux apply following tuning:
net.ipv4.tcp_syncookies=1 net.core.netdev_max_backlog=5000 net.ipv4.tcp_max_syn_backlog=4096 net.core.somaxconn=5000 net.core.rmem_max=8388608 net.core.wmem_max=8388608 net.ipv4.tcp_no_metrics_save=1 net.ipv4.tcp_keepalive_probes=2 net.ipv4.tcp_keepalive_intvl=20 net.ipv4.tcp_fin_timeout=15 net.ipv4.tcp_max_syn_backlog=5000 net.ipv4.tcp_keepalive_time=300 net.ipv4.tcp_tw_recycle=0 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_sack=1 net.ipv4.tcp_dsack=0 net.ipv4.ip_local_port_range= 1025 65000 vm.swappiness = 5 net.ipv4.tcp_slow_start_after_idle = 0
Ensure you tune correctly the user that runs JMeter limits:
run this to check user limits:
Ensure you increase :
- open files (-n)
- max user processes (-u)
Edit /etc/security/limits.conf and add the lines:
* soft nofile 65535 * hard nofile 65535 * soft nproc 88000 * hard nproc 88000
Edit /etc/pam.d/login, adding the line:
session required /lib/security/pam_limits.so
The system file descriptor limit is set in /proc/sys/fs/file-max. The following command will increase the limit to 65535:
echo 65535 > /proc/sys/fs/file-max
You should then be able to increase the file descriptor limits using: ulimit -n unlimited The above command will set the limits to the hard limit specified in /etc/security/limits.conf.
- Ensure you use last minor version of Java 8 or 9. This is for reliability , security and performance
- Ensure you use Java 64 bits to be able to allocate more than 1.5/ 2GB memory. Check this by running
- Allocate enough heap based on your calibration, for example to allocate 6g Heap:
- Check in jmeter.log that the memory is really what you allocated, something like this, for 2g you would get:
2018-03-14 21:55:09,662 INFO o.a.j.JMeter: Max memory =2058354688
- Some advised GC tuning:
-XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:G1ReservePercent=30 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -XX:MaxMetaspaceSize=256m -XX:G1HeapRegionSize=32m -Xss256k
- Run headless by adding this to JVM options:
- To set those options on Linux, using JMeter 4.0:
- create a file called setenv.sh and set:
- GC_ALGO for GC tuning
- HEAP for Memory tuning and other tunings
- create a file called setenv.sh and set:
As always use latest version of Apache JMeter as you’ll get:
- Best performances
- Best reliability
- More features
- More security
Our solution does not require any scripting except if you want to export some variables, make some custom scripting or customize number of Thread for rampup.
In this case, for your scripts :
NEVER EVER USE the below language as they will limit JMeter scalability in a huge way:
What is UbikLoadPack
If you’ve never heard about our solution and are coming here for the first time for another reason, then you should know that UbikLoadPack is a commercial set of JMeter plugins that allow load testing complex protocols or technologies realistically and easily.
Our solution is used by very big names in all fields and has been around since 2013.
We offer solutions for load testing:
- Video Streaming servers delivering HLS, MPEG-DASH, MS Smooth. Our solution test realistically simulating real players.
- Complex enterprise solution like Oracle JD Edwards, Oracle PeopleSoft, Oracle Siebel, Vaadin…
- GWT RPC
- Java Serialization based applications
- FLEX/AMF application
So come and get a free trial version HERE.
Trial is free for 10 days.
Our plugin can be used on Cloud providers like:
- Handy checklist for ensuring User Experience when delivering Video Streams at Large Scale 4 February 2019
- Reporting feature in JMeter 17 January 2019
- Shift left your performance tests with JMeter and Maven 15 January 2019