You want to make a Load Test that will use hundreds to thousands of Threads and you’ve read on so many blogs that :
- JMeter consumes a lot of memory…
- JMeter generates OutOfMemory…
- JMeter consumes a lot of CPU…
- JMeter can only cope with 100 threads and not much more…
- JMeter is fine for playing but once it becomes serious use another tool :-)
This has become a kind of “Urban Legend” partly due:
- to issues that have been fixed for a while now
- and partly in my opinion to some default configuration parameters that lead to these issues (UPDATE 22th July 2014 : Defaults have been improved in versions 2.10 & 2.11)
- misuse of elements that are to be used during scripting and never during Load Test
Hopefully, You’re in the right place, Everything you read is WRONG. :-)
In this short article, we will see how to get the most out of JMeter.
JMeter Performance Basic concepts
Understand important concepts about JMeter:
- The more it computes to create or process your samples, the less time it spends sampling, so when doing custom coding, ensure it is efficient
- Listeners receive Sample Results and do some processing with it, this takes resources (memory, CPU) so during Load Testing, a very simple rule is to REMOVE ALL LISTENERS ! , you will ask How can I use my results then ? , the answer is, you will do it after the run. So NEVER EVER have one of these listeners in your Test Plan during a Load Test :
- View Results in Table => OutOfMemory guarantee in GUI Mode
- View Results in Tree => OutOfMemory guarantee in GUI Mode
- Graph Results => Performance issues
- Assertion Results => OutOfMemory guarantee
- Comparison Assertion Visualizer
- Distribution Graph (alpha) => Performance issues
- Graph Results => Performance issues
Here are our tips to get the most out of JMeter:
Always use the last version of JMeter
JMeter performances and memory usage have been highly improved in last 3 versions and upcoming 2.9 version will bring new set of improvements regarding memory and CPU:
- https://issues.apache.org/bugzilla/show_bug.cgi?id=54268 (fixed in 2.9)
- https://issues.apache.org/bugzilla/show_bug.cgi?id=54412 (UPDATE 22th JULY 2014 : Fixed in 2.10)
Use NON-GUI for Load Test
JMeter has 2 modes:
- GUI mode is for creating the test plan, checking it, debuging it BUT NOT FOR MASSIVE LOAD TEST . AWT Event Thread will disrupt your Load Test.
- NON-GUI mode is for massive load testing, it is as simple as:
<JMETER_HOME>/bin/jmeter -t <Path to Test Plan> -n -l <path to results>/results.csv
Configure JMeter Java options to meet your requirements
Default JMeter java configuration comes with 512 Mo and very little GC tuning.
First ensure you set -Xmx option value to a reasonable value regarding your test requirements.
Then change MaxNewSize option in jmeter file to respect the original ratio between MaxNewSize and -Xmx.
Finally try tuning GC options only if you master this domain.
Use CSV as output for SaveService
XML is verbose, it takes resources to be written (CPU and memory) and for analysis, CSV is great so forget about XML. Furthermore, for massive load tests there are many result data you don’t need.
So, in user.properties, add:
jmeter.save.saveservice.output_format=csv jmeter.save.saveservice.data_type=false jmeter.save.saveservice.label=true jmeter.save.saveservice.response_code=true jmeter.save.saveservice.response_data.on_error=false jmeter.save.saveservice.response_message=false jmeter.save.saveservice.successful=true jmeter.save.saveservice.thread_name=true jmeter.save.saveservice.time=true jmeter.save.saveservice.subresults=false jmeter.save.saveservice.assertions=false jmeter.save.saveservice.latency=true jmeter.save.saveservice.bytes=true jmeter.save.saveservice.hostname=true jmeter.save.saveservice.thread_counts=true jmeter.save.saveservice.sample_count=true jmeter.save.saveservice.response_message=false jmeter.save.saveservice.assertion_results_failure_message=false jmeter.save.saveservice.timestamp_format=HH:mm:ss jmeter.save.saveservice.default_delimiter=; jmeter.save.saveservice.print_field_names=true
Use Post-Processor and Assertion efficiently
Post-Processor and Assertions have a cost.
Ensure you use them when required and use the ones that consume the less memory and CPU:
Use Regular Expression Extractor
Use Regular Expression Extractor for extracting data BUT never ever check Body (unescaped), choose among:
- Response Code
- Response Message
Use efficient Regular expressions and extract as less data as possible
Avoid XPath Extractor when possible
XPath builds a DOM tree so it consumes CPU and memory, prefer Regular Expression extractor or CSS/JQuery Extractor (since JMeter 2.9) for HTML content
Use Response Assertion or Size assertion
These 2 implementations fit 99% of requirements.
Avoid costly ones as:
- XML Assertion
- XML Schema Assertion
- XPath Assertion
Use JSR 223 + Groovy for Scripting
You have a lot of options to do scripting with JMeter:
Although you can be lazy and choose the language you know, FORGET ABOUT IT.
Use the most efficient option, which is JSR223 + Groovy + Caching (supported since JMeter 2.8 in external script and since JMeter 2.9 also supported with embedded scripts).
Using Groovy is as simple as adding groovy-VERSION-all.jar in <JMETER_HOME>/lib folder.
But of course ensure your script is necessary and efficiently written, DON’T OVERSCRIPT
Generate Reports AFTER the Run
Always generate Graphs and Reports AFTER the Load Test and NEVER EVER DURING IT
You can use JMeter reports:
- Aggregate Graph
- Response Time Graph
- JMeter Plugins graphs
Update 22th July 2014, we contributed this GraphsGeneratorListener, read our blog:
You can alternatively choose a SAAS solution to generate your report, see:
- http://loadosophia.org/ (Provider is Team Leader of great JMeter-Plugins project)
Distributed (Remote) Testing
Once you reach the limits of one machine, you can switch to distributed or remote testing.
But JMeter defaults before 2.9 were not fine for efficient remote testing, so in user.properties, add:
Update 22th July 2014, this mode is now the default since JMeter 2.9.
- remove some data from the SampleResults as the response body, but do you need response body during a High Load Test, NO, DEFINITELY NO !
- will send Sample Results as Batches and not for every sample reducing CPU, IO and network roundtrips
Following those recommendations you can reach easily up to 5000 threads on a standard basic machine, 10000 threads on a m3.xlarge …
When it’s not enough
- Use our Perf Sampler plugin, see UBIK LOAD PACK
- Use one of the Cloud Solutions that support Apache JMeter (4 at least now)
- Deploy it on the cloud yourself:
- It works pretty fine since 2.11 with distributed testing (N Servers , 1 controller)
- Or use this project https://github.com/oliverlloyd/jmeter-ec2 by Oliver Lloyd