Load-Testing an API with JMeter: Most of the Work Was Reading the Errors

I needed to know whether an app’s backend API, on Node.js, could hold up with many users signing in and opening their data at once. I wrote a JMeter suite and, over four days, produced 59 sets of results. Writing the script wasn’t what took the time. Working out what the numbers meant was.

One plan, three profiles

The test does what a user does after opening the app: signs in, opens their profile and a few history lists, opens a QR code, reads their messages, and optionally updates their profile. Accounts come from a CSV file, with one file per environment.

I didn’t want to maintain three nearly identical scripts, so there’s one test plan. Everything that changes between runs comes from a profile through JMeter’s ${__P(name,default)}:

# stress.properties
threads=150
rampUpSec=30
perApiRpm=115
loginRpm=115
durationSec=1800
enableWrites=true

	Threads	Duration	Requests per minute, per endpoint
Smoke	100	3 minutes	115
Load	50	1 hour	23
Stress	150	30 minutes	115

Each run writes its HTML report, raw .jtl results, and log into a timestamped folder. That’s the only reason I can still compare those runs now.

Writes sit behind enableWrites, so a run can be read-only. All three profiles have it on, though, so read-only is something you have to choose.

Control requests per minute, not threads

Thread count sets how many users are active, not how many requests reach the server. A fast endpoint just makes threads loop faster. So each endpoint has a Constant Throughput Timer that caps its requests per minute, and that makes runs comparable.

It also means the 17 requests per second in my stress report is the load I set, not what the server can handle. I never ramped up to find a ceiling, so this suite can’t tell you the maximum.

Logging in from the script

The API doesn’t accept plaintext passwords. A client fetches an RSA public key, encrypts the password, and submits that. So each thread has to do the same during setup, in a Groovy pre-processor:

import java.security.KeyFactory
import java.security.spec.MGF1ParameterSpec
import java.security.spec.X509EncodedKeySpec
import javax.crypto.Cipher
import javax.crypto.spec.OAEPParameterSpec
import javax.crypto.spec.PSource

def pem = vars.get('publicKeyPem')
    .replaceAll('-----(BEGIN|END) PUBLIC KEY-----', '')
    .replaceAll('\\s', '')
def key = KeyFactory.getInstance('RSA')
    .generatePublic(new X509EncodedKeySpec(Base64.decoder.decode(pem)))

def cipher = Cipher.getInstance('RSA/ECB/OAEPWithSHA-256AndMGF1Padding')
def oaep = new OAEPParameterSpec('SHA-256', 'MGF1', new MGF1ParameterSpec('SHA-256'), PSource.PSpecified.DEFAULT)
cipher.init(Cipher.ENCRYPT_MODE, key, oaep)

def encrypted = cipher.doFinal(vars.get('password').getBytes('UTF-8'))
vars.put('encryptedPassword', Base64.encoder.encodeToString(encrypted))

The explicit OAEPParameterSpec matters. The server uses SHA-256 for both hashes, and Java’s default for that transformation name uses SHA-1 for MGF1, despite how it reads.

The first day was mostly fixing the script: Base64 handling, a JSON extractor, Groovy syntax, a timestamp. That’s what the smoke test is for. It tells you whether the script works before you start suspecting the server.

The stress runs weren’t a straight line

Here are the stress runs I kept reports for, picked to show the swings:

Run	Error rate	p90
Oct 27, 17:58	23.69%	30.4 s
Oct 27, 23:00	0.76%	2.3 s
Oct 28, 02:30	53.81%	2.8 s
Oct 28, 02:48	50.86%	2.2 s
Oct 28, 18:13	34.57%	30.4 s
Oct 29, 01:03	9.56%	12.0 s
Oct 29, 23:09	3.79%	10.1 s
Oct 29, 23:56	2.23%	2.3 s

I wouldn’t draw much from any one row. The script and settings changed between runs, some were much shorter than others, and I didn’t record which deployed version each one hit. Two details are worth noticing, though. A p90 of 30 seconds is the response timeout, so those requests were hitting the limit. And 50% errors with a p90 of 2.2 seconds is a different failure, since those requests failed quickly.

An error rate on its own doesn’t tell you who to blame.

Breaking down the last run

To trace errors I’d added request and response logging to the plan, which let me go from an aggregate number back to specific requests. The last stress run had 703 errors out of 31,588 requests, at 2.23%:

Type	Count
HTTP 500	387
HTTP 401	282
Connection errors and timeouts	34

The 401s were mostly my script. One thread’s request for the public key timed out, so it had no key. The plan is set to continue on error, so that thread sent the literal text ${encryptedPassword} as its password and got a 401 on login. Then it got a 401 on every later request. That one thread produced 209 of the 703 errors: 23 on each of the eight downstream endpoints, 24 on login, and the key request itself.

The 500s pointed upstream. Some data comes through a proxy to a third-party system. The logs showed that proxy failing for about 100 seconds during the run, and the API passing its failure along as 500.

The other 34 I can’t assign. Connection timeouts and resets could be the network, the test machine, or the server. So I can say the last run’s errors were mostly not the API’s own, but I can’t say all of them were.

One thing I got wrong in my own analysis at the time. I saw 97 login 401s in JMeter and 98 in the server logs, and put the gap down to the sampling window. Going back to the raw results for this post, there were 97 under the login label and one more under the separate initial login label. It was a labeling difference.

The real bug it found

The most useful result of all this came from an earlier run. Under load, the API hit Prisma’s P2028, where a transaction can’t start or finish in time. A colleague fixed it by moving a lookup out of the transaction and removing a $disconnect() that ran at the end of each request.

The final numbers came after that fix, going by the dates, though without recorded versions I can’t prove each run included it:

	Error rate	Mean	p90	p95
Stress (150 threads, 30 min)	2.23%	1.5 s	2.3 s	2.7 s
Load (50 threads, 1 hour)	0.22%	1.3 s	2.0 s	2.2 s

What I’d change

• Stop a thread that can’t get its key. Continuing on error turned one timeout into 209 fake errors. I found this and wrote it down, but the plan I have still continues on error.
• Record the deployed version with each run. Without it, I can’t fully connect a fix to a result.
• Report upstream errors separately from the API’s own 500s.
• Ramp up to a ceiling. The fixed rate answers “does it hold at this load”, not “how much can it take”.
• Hardcoded timeouts. Connect and response timeouts are still fixed in the plan, not in the profiles.