Skip to content

Load testing the booking path

Performance results says what one request costs. This page says how many people can book at once before the site stops coping, and what gives way first. They are different questions and they need different tools: a functional test issues one request at a time, so it cannot answer the second even in principle.

The answer that transfers is a named resource, not a number. One container on one machine gives a count of simultaneous bookers for that machine and nothing else. What carries from a run to a real site is the shape: which resource saturates first, where the latency knee is, and whether a change moved the ceiling up or down.

What the harness is

Two halves. The site's half is the yoyaku_load_test module, which builds the hall and reads back what a run left behind. The driver's half is three PHP scripts under tests/load/, which speak HTTP and know nothing of Drupal, so they run anywhere PHP and its curl extension do.

Piece What it does
drush yoyaku:load-seed Builds a venue, a hall of places and the slots offering it, and prints the manifest the driver reads.
drush yoyaku:load-clear Gives every place back between arms. --truncate empties the tables instead of deleting each booking, which is the rig's reset and leaves the caches to you.
drush yoyaku:load-check Reads back what the run left: no place held twice, and the hall holding what the driver was told it held. Fails the run when it does not.
tests/load/drive.php Walks the journey with many visitors at once and writes one line per request.
tests/load/sample.php Watches the web server's workers, the database's threads and row lock waits, and memcache, on a fixed tick.
tests/load/aggregate.php Turns the samples into throughput and percentiles per step.

The journey, and why it is not one endpoint

Hammering one endpoint measures that endpoint. The driver walks the journey as it is actually walked, because the expensive parts are not evenly spread:

  • map, arriving on the place map, which is a render;
  • map_document, the hall the page draws, served apart from it so a browser and a proxy can keep it. A run that fetched only the page would report an arrival costing half what an arrival costs;
  • token, the CSRF token the first click fetches in a request of its own;
  • open, the opening click, which writes the order and is measurably dearer than the ones after it;
  • click, the recurring click, the one most visitors repeat;
  • avail, the availability feed the map refreshes from;
  • cart, the basket.

Visitors come in a mix, set with --mix=browse:book:abandon. Most arrivals browse, fewer book, some wander off mid-basket and leave the sweep to clean up. The ratio changes which bottleneck arrives first, so it is part of the scenario and is recorded with every figure.

Every visitor has a cookie jar of its own and their requests are in flight together. It is not a process per visitor, which no PHP process could hold: the visitors are a state machine driven by one curl_multi loop, so a run costs one process however many visitors it walks. Where one process cannot keep up, run several with different --seed values; aggregate.php takes any number of sample files.

Refusals are the majority case

Capacity is finite, so a run books the hall out and then every further hold is refused. That is not the harness breaking down, it is the harness reaching the interesting part: during a real rush most people meet a refusal, not a success. Measuring only successful holds measures the minority.

So the harness has to be explicit about capacity, and the three ways of being explicit answer different questions:

  • seed a hall the run cannot exhaust, and measure the cost of holding;
  • empty it between rungs, and measure each rung on the same starting state;
  • exhaust it on purpose, and measure the refusal path under load.

A refusal is also not a failure, and telling them apart is not something a status code can do here. A run of clicks is answered with 200 and a refused list naming the place and why, so the driver reads the body. It reports refused_taken for a place somebody else got, refused_policy for a place this booker may not have, refused_gone for a basket that is no longer there, and keeps fail_http and fail_transport for the site actually giving way. In a status code count those all look alike and they mean opposite things.

What a run records

  • Throughput, holds per second, and the arrival rate at which it stops rising.
  • Latency percentiles per step, p50, p95 and p99 separately for the render, the opening click and the recurring click. A mean over the three is a number about nothing.
  • Outcome mix, refusals apart from failures, as above.
  • What saturates first. The web server's busy and idle workers, the database's running threads and connections, its row lock waits and what those waits are waiting ON, memcache's hit rate and evictions, and what the machine itself has left. That last one is not padding: a run whose requests start failing because the site reached its limit and one whose requests start failing because the box ran out of memory look identical from the outside, and only one of them is about yoyaku.

The last is the real deliverable, and it is the reason sample.php reads performance_schema.data_lock_waits rather than only the summary counters: the waits file names the table, the index and the row, which is what turns "the run was slow" into "the waiting was here".

Running it

Build a site of its own. Not a site anybody is browsing, and not one sharing a database or a memcache with anything else: hit rates and lock counters have to be this run's and nobody else's. Turn off any step debugger, which otherwise attempts a connection on every request, and leave the opcode cache on.

Seed the hall, once:

drush en yoyaku_load_test -y
drush yoyaku:load-seed --rows=80 --per-row=100 --slots=4 > hall.json

That hall carries a capacity and a tariff quota, both its own size, which is the tightest configuration a slot can be given and what the published register was measured against. A seated house usually carries neither, because its seats are what bound it, and the difference is not cosmetic: those two numbers are what decide how much of a slot one hold has to lock. Pass --bounded=0 for a hall bounded by its seats alone, and say which of the two a figure came from.

Then one rung: empty the hall, rebuild the caches, start the sampler, drive.

drush yoyaku:load-clear --truncate && drush cr

php tests/load/sample.php --mysql-host=DB --status-url=http://SITE:8080 \
  --memcache-host=CACHE --seconds=95 --out=/tmp/rung-40 &

php tests/load/drive.php --base=http://SITE --manifest=hall.json \
  --bookers=40 --seconds=60 --warmup=30 --arm=single \
  --label=rung-40 --out=/tmp/rung-40

php tests/load/aggregate.php /tmp/rung-40.samples.tsv
drush yoyaku:load-check 1,2,3,4

Then walk the rungs: 5, 10, 20, 40, 80, 160 simultaneous bookers. Throughput flattening while p95 climbs is the knee, and it answers the question in the units the question is asked in.

The warmup is not padding. Emptying the hall means rebuilding the caches, and a cold start under concurrency is its own event: requests pile onto the semaphore rows core uses to stop two of them rebuilding the same cache collector at once. Measured at a ten second warmup, that stampede was still inside the window and it put a fourteen second reading into the map page's p99. Thirty seconds puts it outside. If a run reports an outlier on a render step, look in the waits file for semaphore before believing it is about booking.

The arm that drives an area rather than seats

A seat is claimed once and two bookers conflict only over the same seat. An area offered by quantity is counted, and a count is what a race can spoil, so it is worth an arm of its own. Seed the area with --pool, then drive it with --units instead of seats:

drush yoyaku:load-seed --rows=1 --per-row=1 --bounded=0 --pool=200 > pool.json

php tests/load/drive.php --base=http://SITE --manifest=pool.json \
  --bookers=60 --seconds=45 --warmup=0 --units=3 --clicks=4 --mix=0:100:0 \
  --label=pool --out=/tmp/pool

drush yoyaku:load-check <slot> --pool-section=<section> --pool-capacity=200

Two things differ from a seated arm. The warmup is zero, because the driver's count of units has to be comparable with the area's own and a warmup would take units it could not account for; warm the page by hand with a single request instead. And a click can be served in part: asked for three units with two left, the endpoint gives the two and refuses the rest in the same answer. The driver counts what a click actually got, by the difference in what the visitor holds of the area, rather than counting answers, because counting answers gets exactly that case wrong.

The two arms that settle the lock question

Taking a place does SELECT ... FOR UPDATE on the slot row, serializing every hold on that slot by design. That is the module's central promise, that a rush cannot overbook. Two arms at the same total concurrency separate what serializes from what merely collides:

  • --arm=single, every visitor on one slot, where holds serialize on that slot's row by design;
  • --arm=spread, visitors across several slots, which do not serialize against each other yet still write the same cache tag row.

Run both with --mix=0:100:0 so the render is out of the way and the holds are what is under load.

Reading the answer

aggregate.php gives throughput and percentiles. The saturation answer is in the two files sample.php writes. The tick file is the timeline: busy workers against idle, threads running, row lock waits per tick, memcache hits, misses and evictions. The waits file is written only while something is actually waiting, one line per wait, naming the row.

Count the waits by table. That count is the answer to "what gives way first", and it is an answer with a name.

Caveats to record with every figure

  • One container on one machine. The absolute count of simultaneous bookers does not transfer. The shape does.
  • The driver shares the box with the site. Every sample carries two durations: ms, which is the site's, and wall_ms, which adds the time the request waited for the driver's own loop. When the second runs away from the first, the run is measuring PHP on the driver's side and the figures are spent.
  • Name the scenario completely: the hall's size and shape, the number of slots, the mix, how many places a booking visitor takes, the think time, the database, and what was already booked. A count without them means nothing.
  • The status endpoint counts itself. One busy worker at idle is the sampler's own request.
  • Say measured or inferred, every time, and never carry a figure from this harness into a claim about a live site.