Sizing for a rush¶
What to decide before a rush: how much machine to bring, how many workers to run, and which of yoyaku's own settings change what a rush costs.
Performance results records the figures and Load testing the booking path describes the harness that produced them. This page is the operational reading of the same runs: not what the numbers are, but what to do about them.
How to read the numbers on this page¶
Absolute throughput does not survive between runs. Ratios do.
Four ladder runs of the identical configuration produced one-slot throughput of 60.6, 55.0, 50.0 and 36.0 holds a second: a spread of 68 percent, with no change to the code, the hall, the load or the worker cap, and with memory headroom, worker saturation and driver latency all confirming the runs were valid.
Normalize each ladder to its own first rung and the same runs agree closely. So every claim here is a ratio measured inside a single run, and a difference between two runs smaller than about 40 percent is not a finding, it is the rig.
This is the discipline the page follows, and it is worth adopting for any load work: compare arms within one run, never a number from today against a number from yesterday.
What a worker costs¶
This is a direct measurement of a process rather than a throughput comparison, so it does not suffer from the drift above.
A PHP worker that has served a booking click keeps the memory it needed, and does not give it back. So the working figure is not what a request allocates, it is what a worker settles at after it has served the biggest page the site has.
Measured on this rig, Apache prefork with mod_php, opcode cache on:
| worker state | resident |
|---|---|
| started, nothing served | about 72 MB |
| after a click on a 1440 place hall | about 114 MB |
| after a click on an 8000 place hall | about 150 MB |
The last row is the one to size on, because a worker that has served the big hall once is a big-hall worker for the rest of its life. A cap of 80 workers on this hall wants about 12 GB for the web server alone, before the database, the cache and the operating system.
Two measurements of worker memory, both true, only one useful
Summing per process RSS across workers overstates it badly, because RSS counts every shared page once per process and these processes share a PHP binary, an opcode cache and every library between them. Asking the container what an extra worker actually adds gave about 7 MB each on a small hall, which understates it just as badly: that is the marginal cost of a worker that has not yet touched a large page. Neither is the sizing number. Size on what a worker settles at once it has served your largest hall, which is what the table above measures.
Two consequences worth stating plainly:
- Recycle workers. Apache's
MaxRequestsPerChild, orpm.max_requestsunder PHP-FPM, is what returns a big hall's memory to the machine. Without it the web server's footprint only ever ratchets up. - Measure after a click, not after a warm-up. A rig that has only served the front page reports a worker cost that has nothing to do with a rush.
Raising the ceiling: give the rush more rows¶
Taking a place locks the slot's row, so holds on one slot are served one at a time. Spread the same load over more slots and it lands on more rows.
Three separate ladder runs, each normalized to its own one-slot rung, 8000 place hall, 160 bookers, 80 workers, holds only:
| slots | run A | run B | run C |
|---|---|---|---|
| 1 | 1.00 | 1.00 | 1.00 |
| 4 | 1.98 | 1.85 | 2.23 |
| 16 | 2.84 | 2.62 | 2.65 |
Four slots carry about twice the holds of one, and sixteen about two and three quarter times. Three runs whose absolute throughput differed by 68 percent agree on that to within a tenth, which is the strongest result on this page.
The returns fall off, as they must once the waiting has moved somewhere else. Sixteen times the rows is not sixteen times the throughput and never could be.
In operational terms:
- Several performances beat one. This is the largest lever available, and it is a configuration choice, not a code change.
- A hall that is one slot is one queue, however large the hall. An 8000 seat house booked as a single slot has the same ceiling as a 200 seat one.
- More sections do not mean more parallelism by themselves. What sets the effective number of lanes is how the demand spreads. A house with forty sections but four price bands concentrates its rush into far fewer lanes than its section count suggests.
- Staggered opening times per slot spread the arrival as well as the lock, and cost nothing to arrange.
Where the queue forms, in order¶
Relieving one bottleneck reveals the next, so the order matters more than any single figure. These are counts of lock waits inside one run, which the drift above does not touch.
First, the slot row. Taking a place locks the slot's row, which is
what makes overbooking impossible and also what serializes the rush. On one
slot it is essentially all of the waiting: on the ladder run, 13871 of 13877
recorded wait edges were on a single yoyaku_slot row, and at the top rung
every one of 38392 edges was.
Give the load more slots and that waiting collapses by orders of magnitude.
Wait edges on yoyaku_slot over one, four and sixteen slots in one run:
162688, then 2581, then 483.
Then, the cache tag row. What holds queue on next is the cachetags row for
yoyaku_booking_list, which every hold invalidates. It is small in a
single-slot rush and the majority of the waiting once slots stop
colliding, because slots that do not serialize against each other still
write that one row. Whether moving it is worth doing is discussed below; on
this rig the question could not be settled.
Then, worker memory, as above.
Nothing else came near its limit on any run: memcache evicted nothing at any rung, and the machine's free memory was never the constraint except when it was the constraint entirely and the database was killed.
Twenty times more people than workers¶
What happens when 2000 people try to book and the box runs 80 workers.
They queue, and queueing is the cheap part. A connection waiting in the kernel's accept backlog costs kilobytes; a connection waiting in a PHP worker costs 150 MB. The whole design goal under a rush is to keep people in the first queue until a worker can actually serve them.
- Let the backlog be deep and let the worker cap be the throttle. This is the opposite of raising the cap until nobody queues.
- The arrival paths are page cached and answer in single digit milliseconds, so a crowd refreshing the event page is not competing with the crowd trying to book.
- A refusal costs about what a success costs: within one run, 16 ms at the median for a refused seat click against 16 for one that got its seat, 35 against 35 on a counted area. During a rush most clicks are refusals, and the refusal path being the cheap one is why the site stays up while the hall empties.
- The guarantee holds at the boundary. A hall of 200 places under 60 simultaneous bookers gave out exactly 200 places, 4281 refusals, and no failures of any kind. This is a count, not a rate, so it is exact.
What this does not promise is that everybody gets served quickly. It promises that the site degrades into a queue rather than into errors, which is the behavior to design the front of the site around: tell people they are queueing.
Does doubling the workers help?¶
Not on its own, and the honest answer is weaker than it looks.
Comparing a 160 worker ladder against an 80 worker one, each normalized to its own one-slot rung:
| slots | 80 workers | 160 workers |
|---|---|---|
| 1 | 1.00 | 1.00 |
| 2 | 1.48 | 1.84 |
| 4 | 1.98 | 2.56 |
| 8 | 2.29 | 3.04 |
| 16 | 2.84 | 3.50 |
Read this as how much of its own ceiling each configuration reaches at one slot: the 80 worker run starts at 35 percent of its 16-slot figure, the 160 worker run at 29 percent. The extra workers gain more where the load is genuinely parallel and less where it is queueing on one row, which is what the lock model predicts.
What cannot be claimed from this rig is the absolute comparison, that 160 workers serve more or fewer holds a second than 80. Those are two different runs, and two different runs of the same configuration differed by more than any gap in that table.
The practical rule does not depend on resolving it. Set the worker cap from the memory budget, since 80 workers already want 12 GB on a large hall, and do not expect extra workers to relieve a single-slot rush, because what they are waiting for admits one at a time whatever their number.
Settings that change what a rush costs¶
Row level locking is required, not preferred¶
The guarantee is a SELECT ... FOR UPDATE inside a transaction, and a driver
that does not implement the clause does not fail, it just does not lock.
Drupal's SQLite driver drops it silently, which leaves the locking read an
ordinary read and every figure on this page meaningless.
Run MySQL with InnoDB. Everything measured here was, and on a normal site
this needs nothing done: core's MySQL driver creates every table with
ENGINE = InnoDB and installation fails where InnoDB is unavailable. It is
worth confirming only where somebody has converted tables to another engine by
hand, since a table that is not InnoDB takes no row locks. PostgreSQL implements
the clause, and the test suite passes on it, though continuous integration runs
on MySQL. This is not
a tuning note: a site on the wrong driver does not have the guarantee it thinks
it has, and nothing tells it so.
Cache tag checksums in memcache: measured, and it is the tail it fixes¶
The cachetags row is the bottleneck a house meets once its own locks are in
order, and on this rig it was worth more than any tuning on this page.
Every booking saved invalidates yoyaku_booking_list, one row of core's
{cachetags} table. Core writes it after the transaction commits rather than
inside it, so no hold keeps that row while it works, and what forms instead is a
convoy: each invalidation is a short single row update, and at a hundred holds a
second they queue for their turn. It makes no difference how large the house is
or how many parts it has, because every booker on the site shares that one row.
Seen from the database, on a house of 80,000 seats in 80 parts with every booker naming its own part, so the parts were independent by construction:
| at 160 bookers | waiting | granted |
|---|---|---|
cachetags PRIMARY |
107 to 154 | - |
yoyaku_booking, yoyaku_slot_place, yoyaku_lock_anchor |
none | 6 |
Nothing was short of processor or memory when that was sampled: the web container ran at six to nine cores of fourteen, the database at about one, and the harness itself at twelve per cent of one core. The bookers were queueing and not working. Asked which transaction held the row, the database named one that was itself waiting, with its own update pending and nothing held for longer than a moment: a queue rather than a hostage.
Moving the checksums to memcache changes nothing else, and what it changes most
is how long a booker waits. Same rig, same house, same driver, same rung, arms
alternated, with the {cachetags} counter read before and after each run to
prove which arm ran: it moved by sixteen and twenty-three thousand on the
database arm and by nothing at all on the memcache one.
| 80 bookers | holds/s | median hold | 95th percentile |
|---|---|---|---|
| database checksums | 184 and 256 | 244 to 249ms | about 795ms |
| memcache checksums | 282 and 295 | 154 to 160ms | 276 to 290ms |
Read the latency, not the rate. The median falls by about a third and the ninety-fifth percentile by about two thirds, tightly in both rounds, which is what a queue looks like when it goes away. The throughput difference is real but between ten and sixty per cent depending which rounds are paired, because the database arm itself swings from 184 to 256 on this machine, and this page's warning about drift applies to its own figures as much as anybody's.
One service, in the same $settings['container']['services'] section that wires
the memcache bins:
'cache_tags.invalidator.checksum' => [
'class' => 'Drupal\memcache\Cache\TimestampCacheTagsChecksum',
'arguments' => ['@memcache.timestamp.invalidator.tag'],
'tags' => [['name' => 'cache_tags_invalidator']],
],
The memcache module documents a services file for the same thing; either way in reaches the same service. The figures above were measured with the section above.
What it costs you is that memcache stops being optional. Tag invalidation becomes memcache's job, so a site that loses memcached loses the record of what has been invalidated. The direction of that failure is the safe one, since a missing timestamp reads as invalidated and the caches rebuild, but it does mean memcache belongs in the same class as the database for a site sized this way: monitored, resident, and not restarted during a rush.
The module side of this is [#3618572]: a hold invalidating the whole site's booking list is also why one slot's rush throws away every other slot's availability, and that is worth fixing whatever the checksums are stored in.
The other memcache bins are worth having¶
Nothing here argues against memcache generally. The default, render and dynamic page cache bins were in memcache for every run on this page, and the arrival paths being single digit milliseconds is that working.
A checklist before a rush¶
- Serve one click on your largest hall and measure a worker. That figure times the worker cap is the web server's memory, and it must fit with the database and cache still resident.
- Recycle workers so a big hall's memory goes back to the machine.
- Do not raise the worker cap to meet the crowd. Past the lock it does not relieve a single-slot rush, and it costs memory whether it helps or not.
- Count the slots the rush will land on. One is the worst case and the most common mistake.
- Put cache tag checksums in memcache. Without it every booker on the site queues on one row whatever else you tune, and that queue is most of the tail: the ninety-fifth percentile fell by about two thirds when it was measured.
- Confirm InnoDB.
- Run the harness against a copy with your hall and your mix, compare arms inside one run, and record the caveats with the figures. A number without the hall's shape beside it is not a number, and a number compared against yesterday's run is not a comparison.