UbikLoadPack : Advised tuning for injector when load testing Video Streaming servers
Recently one of our customer asked the following question on Stackoverflow:
What are the recommended configuration in terms of:
- System tuning
We thought that we should be answering this question with as much details as possible.
So here is the answer.
It is hard to answer such generic question as behaviour depends on at least following factors:
- The type of Stream : HLS will perform better than MPEG-DASH and HSS, the 2 latest being more CPU consuming due to XML format.
- The duration of the Manifest: This is the total duration it holds, the biggest it is the more memory and parsing it will require
- The refresh period : This is the period at which a Live Stream manifest will be reloaded by player, the shortest it is, the more it will consume memory and CPU
- Whether stream is Live or VOD, Live will consume more CPU, VOD might consume more memory
So the best advice is to calibrate based on your Manifests, meaning to run this for 50 to 100 threads, see how much CPU and Heap it consumes and then extrapolate.
Physical Machine or Virtual Machine:
- Ensure you have enough bandwidth in your data-center between injector and target servers
- You should usually load test as close as possible to target to avoid network parasitizing your results. And only when you’re sure your server are correctly handling the load load test from Cloud servers. But you can do the latter if you can’t do the first step.
- Ensure your network card is correctly configured
- With “Average Manifest” , you should have no issue load testing 700 to 1500 Threads (Virtual Users) with machine of type m5.2xlarge (8 vCPU / 16 to 32 Go)
- We advise you to use Linux/MacOS instead of Windows OS as you will get more Threads on same virtual/physical configuration.
- But all OSes work since Java is used
For Linux apply following tuning:
net.ipv4.tcp_syncookies=1 net.ipv4.tcp_no_metrics_save=1 net.ipv4.tcp_keepalive_probes=2 net.ipv4.tcp_keepalive_intvl=20 net.ipv4.tcp_fin_timeout=15 net.ipv4.tcp_max_syn_backlog=5000 net.ipv4.tcp_keepalive_time=300 net.ipv4.tcp_tw_recycle=0 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_sack=1 net.ipv4.tcp_dsack=0 net.ipv4.ip_local_port_range= 1025 65000 vm.swappiness = 5 net.ipv4.tcp_slow_start_after_idle = 0 #Adjust those ones based on your machine specification: net.core.netdev_max_backlog=5000 net.ipv4.tcp_max_syn_backlog=10240 net.core.somaxconn=5000 net.core.rmem_max=25165824 net.core.wmem_max=25165824
Ensure you tune correctly the user that runs JMeter limits:
run this to check user limits:
Ensure you increase :
- open files (-n)
- max user processes (-u)
Edit /etc/security/limits.conf and add the lines:
* soft nofile 65535 * hard nofile 65535 * soft nproc 88000 * hard nproc 88000
Edit /etc/pam.d/login, adding the line:
session required /lib/security/pam_limits.so
The system file descriptor limit is set in /proc/sys/fs/file-max. The following command will increase the limit to 65535:
echo 65535 > /proc/sys/fs/file-max
You should then be able to increase the file descriptor limits using: ulimit -n unlimited The above command will set the limits to the hard limit specified in /etc/security/limits.conf.
- Ensure you use last minor version of Java 8 or 11. This is for reliability, security and performance
- Ensure you use Java 64 bits to be able to allocate more than 1.5/ 2GB memory. Check this by running
- Allocate enough heap based on your calibration, for example to allocate 6g Heap:
- Note you should reserve around 400 Mb for your system and not allocate the full available memory
- If your calibration shows that JVM does not require the whole memory, then don’t allocate too big heaps
- Check in jmeter.log that the memory is really what you allocated, something like this, for 6g you would get:
2018-03-14 21:55:09,662 INFO o.a.j.JMeter: Max memory =6442450044
- Some advised GC tuning:
-XX:+UseG1GC -XX:MaxGCPauseMillis=50 -XX:G1ReservePercent=30 -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false -XX:MaxMetaspaceSize=256m -XX:G1HeapRegionSize=32m -Xss256k
- Run headless by adding this to JVM options:
- To set those options on Linux, using JMeter 5.3:
- create a file called setenv.sh and set:
- GC_ALGO for GC tuning
- HEAP for Memory tuning and other tunings
- create a file called setenv.sh and set:
As always use latest version of Apache JMeter as you’ll get:
- Best performances
- Best reliability
- More features
- More security
Add to user.properties:
hread_pool_size=<Max Number of Threads * 2>
hread_pool_queue_size=<Max Number of Threads * 2>
Note you can pass the 2 latest on command line:
-Julp_streaming.sampler.player.thread_pool_size=<Max Number of Threads * 2> -Julp_streaming.sampler.player.thread_pool_queue_size=<Max Number of Threads * 2>
Our solution does not require any scripting except if you want to export some variables, make some custom scripting or customize number of Thread for rampup.
In this case, for your scripts :
NEVER EVER USE the below languages as they will limit JMeter scalability in a huge way:
What is UbikLoadPack
If you’ve never heard about our solution and are coming here for the first time for another reason, then you should know that UbikLoadPack is a commercial set of JMeter plugins that allow load testing complex protocols or technologies realistically and easily.
Our solution is used by very big names in all fields and has been around since 2013.
We offer solutions for load testing:
- Video Streaming servers delivering HLS, MPEG-DASH, MS Smooth. Our solution test realistically simulating real players.
- Complex enterprise solution like Oracle JD Edwards, Oracle PeopleSoft, Oracle Siebel, Vaadin…
- GWT RPC
- Java Serialization based applications
- FLEX/AMF application
So come and get a free trial version HERE.
Trial is free for 10 days.
Our plugin can be used on Cloud providers like:
- Update JMeter plugins from CLI in CI/CD 20 December 2021
- JMeter and the Log4j vulnerability 14 December 2021
- Case Study from Healthcare sector: Performance Test video streaming services with UbikLoadPack 8 December 2021