Skip to content

gate: run a background job only while there is room

The problem

Background jobs share something with the person at the keyboard: an API quota, a disk, a CPU, a monthly budget. Scheduled blindly, they eat the share the person needs. Checking the number first sounds easy and breaks quietly: ten jobs on one minute all ask the source together and some are refused; the source goes down and every job skips, or every job runs; a limit right for Monday morning is wrong on Saturday night.

The method

One command reads the number, decides, and keeps a history.

  • Read anything. Any shell command; the first number it prints, or a field picked from JSON.
  • Retry once, staggered. A failed read is retried after a pause that differs per process, so jobs started together do not collide twice.
  • Fresh enough beats live. Still failing, use the last recorded reading if it is under 15 minutes old.
  • Fail closed. No number, no go.
  • Learn the week. Every reading lands in a plain TSV. gate project turns it into a growth rate for each of the 168 hours of the week (resets and gaps skipped) and projects where a growing counter will stand at a deadline, so a job can run hot in a quiet week and hold back in a busy one.

Sample

The decision, in Python:

v, err = read_number(cmd, field)
if v is None:
    time.sleep(3 + os.getpid() % 7)          # stagger the retry
    v, err = read_number(cmd, field)
if v is None:
    t, v = last_reading(name)
    if v is None or now - t > max_age:
        return WAIT                          # unknown never means go
else:
    record(name, v, now)
return GO if v < limit else WAIT

Use it:

gate check --read "df -P / | awk 'NR==2{print \$5}'" --name disk --under 90 -- ./backup.sh
# disk: 46 of 90: go
gate check --read "curl -fsS https://quota.example/api" --field windows.0.pct --under 60
# quota: 72 of 60: wait
gate project --name quota --at 1767225600
# in front of a heartbeat job (a skipped wake is not counted)
HEARTBEAT_GATE="gate check --read 'sysctl -n vm.loadavg' --name load --under 4"

Limits

  • The reading is taken once, at the start; a long job is not stopped when the number climbs. Gate short jobs, or each step.
  • An hour of the week needs half an hour of close readings before its own rate is trusted; others use the overall mean.
  • project suits counters that only grow until a reset (quotas, spend), not numbers that wander.

Code

methods/gate/ in the methods repository: the script, a README and a test.