October 22, 202611 min read

Reducing Kubernetes API Traffic in the Ptah Controller

How we removed repeated enrollment and result reads while keeping publication authority and deletion checks live, then measured the released operator.

Nathan BrooksTechnical Writer

Ptah helps you plan, review, and apply database migrations. Try in your browser

A migration Job can finish correctly while the controller spends too much of the Kubernetes API budget watching it. During Ptah Operator qualification, we found repeated reads around work that had already completed: enrolling a Pod, reading an immutable result, and checking whether a result could be collected.

Those reads existed for a reason. The operator must bind a result to the approved operation and its exact Job and Pod. It must store the result before acknowledging delivery. It must retain evidence needed by an active or unresolved Apply. Removing a request meant establishing which decision it served, and whether that decision could have changed.

We reduced repeated reads within those boundaries, then changed how running read-only Jobs wait for progress. The focused tests proved specific reductions. The later release qualification established the operating result: Ptah Operator 0.2.4 stayed within the API budget in all measured soak and backlog cells. The supporting records separate those kinds of evidence.

The request path we followed

Consider a Job that reads a database’s migration history. The controller creates the Job and enrolls its producing Pod. The Pod publishes its result to an authenticated receiver. The receiver checks who may publish, persists the result, and returns a receipt. The controller consumes the result and advances the resource. Collection eventually removes records whose retention and recovery obligations have ended.

A history Job goes through Pod enrollment, publication authorization, durable result storage, consumption, and eventual collection. A short-lived enrollment hint skips repeated enrollment reads. Publication and deletion still have their own live authority checks.

Each step can involve several API requests. Job progress events and timed reconciliation revisit the path while the same Pod is still working. A read that is reasonable once becomes unnecessary work when repeated on every pass. The standby manager also performs control-plane work, so counting only the leader would hide part of the installation’s demand.

The qualification profile makes that repetition visible: ten PtahSchemas and ten PtahMigrations across two namespaces, two manager replicas, and two-minute refresh intervals. It includes retained plans, changes, replacement resources, and unrelated Kubernetes objects, with real PostgreSQL and MySQL execution. The early focused diagnostics used PostgreSQL on native Linux amd64 and Kubernetes 1.37. They were development measurements, not the full profile.

Remember enrollment, then check publication again

Enrollment establishes the first Pod allowed to produce a particular operation’s result. Once its immutable pin existed, later Job progress passes still listed the producer Pod and read that pin again.

We added a bounded hint after successful enrollment. Its key contains the resource identity and generation, execution and input bindings, operation, and Job identity. A replacement Job cannot reuse it. The hint expires after one minute, and each resource family retains at most 64 entries. Failed enrollment is never remembered. A restart or an expired hint returns to direct reads.

This is the decision the hint makes: skip enrolling this exact operation again. It does not answer whether the Pod may publish now.

The receiver and admission still check the persisted first-Pod pin and current authority when a result is published. An operation may have been revoked since enrollment. A Pod may have been replaced. Reusing an earlier authorization would miss those changes, whether that earlier observation came from an informer or a direct read.

The regression test performs five repeated progress passes without another enrollment read, then advances the clock to expiry and requires the original Pod and pin reads again. Separate cases keep the hint warm while revoking the claim, replacing the Pod, or making the API unavailable. Publication is refused in every case. The test results retain those assertions.

Read an immutable result once within a call

Small results already had a simpler storage path. A payload of up to 256 KiB fits atomically in one immutable intent record. Larger results, and small results whose chunked publication had already started, retain the chunked path. Acknowledgment still waits for complete, verified durable storage.

But publishing or loading could read the same intent twice within one call. The outer function had already read the persisted record and checked its binding. The internal loader discarded that observation and fetched it again.

We passed the verified persisted intent into the loader. The loader still checks identity, binding, manifest, digest, and length. Chunked results still require their completion record and every chunk to agree with that exact intent UID. We did not replace readback with a CREATE response.

The controlled inline-publication case changed from two intent GETs to one. The load tests also require each persisted record to be read once. Damaged stored records remain refusal cases: a missing UID, wrong owner, wrong record type, deletion, or corrupt payload must not yield a receipt or a result. This is reuse within a call, without a process-wide cache of publication data.

Postpone collection without remembering permission to delete

Retention provides a useful early answer: before the deadline, deletion is forbidden regardless of the current pin state. Rechecking every recovery pin cannot change that answer. We reused the retirement marker already verified in the current pass and deferred work while the deadline alone required us to keep the result.

Once the result becomes eligible for deletion, the answer depends on current state again. The collector checks live pins, and admission independently checks each DELETE. A recovery pin restored after the collector’s read must still prevent deletion.

Expired collection also reloaded payloads whose identity and version were already present in a read or metadata listing. We used that UID and resourceVersion as DELETE preconditions. If the object changed or was replaced after the scan, the delete is refused and a later scan must reassess it. The controlled expired two-chunk case changed from nine full record GETs to three. Admission’s live checks remain outside that collector-only count.

Reused observation	What can be skipped	What still needs current state
Successful enrollment for the exact operation and Job	Repeated Pod enumeration and enrollment-pin reads while the hint is valid	Permission to publish
Verified immutable intent within one call	A second GET of the same intent	Complete payload and binding validation
Verified retirement deadline in the current pass	Work that cannot authorize deletion before the deadline	Recovery pins and admission when deletion is eligible
UID and resourceVersion from a scan	Reloading a payload only to identify a DELETE	Preconditions and per-delete admission

These changes remove duplicated work. They do not turn an earlier permission into a standing permission.

Use Job events before polling read-only work

Reducing the work in a pass did not remove unnecessary passes. A running read-only Job without a Lease could still trigger another Job and intent read every five seconds, even when nothing had happened.

Job watches already deliver progress. We kept those events and gave this narrow waiting state a 30-second fallback. The released helper makes the Lease boundary explicit:

func activeJobPollInterval(holdsLock bool) time.Duration {
  if holdsLock {
    return maxLockContentionPoll
  }
  return 30 * time.Second
}

Apply keeps its five-second Lease renewal poll. Plan, post-Apply proof, credential enrollment, and result-read retries keep their existing timers. The regression tests deliver a terminal Job event before advancing the clock to the fallback and require the result to be consumed immediately.

A client-side rate limiter would address throughput by delaying requests. It would not remove the work. We separately bound throttling, queue delay, and observation freshness, so shifting request demand into those delays could still fail the profile. More CPU would not change a request-count budget.

The result transport explains some of the cost. Moving results out of Pod logs removed a durability dependency but introduced persistent API objects and collection work. The first transport also created per-operation credential Secrets and exhausted the namespace’s 128-Secret quota. Pod-bound tokens removed that allocation. Solving the log and Secret constraints did not make the new request path inexpensive by itself.

A scrape bounds a peak; it does not locate every request

The budget was originally 6,000 manager requests in every sliding sixty-second window. We changed it to 7,000 as an explicit profile decision during qualification. Raising the limit was not a performance improvement, and the earlier verdicts remain attached to their original limits.

We sampled the managers’ request counters every five seconds. For any window inside the scenario, a completed scrape before its start and a scrape begun after its end enclose all its traffic. Their difference is an upper bound. The extra edge traffic means it may exceed the budget even when the exact window does not. Conversely, a counter difference contained wholly within a minute supplies a lower bound that can prove an exceedance.

A requested sixty-second interval lies inside a wider pair of counter reads. The outer difference includes edge traffic and gives an upper bound; a contained difference gives a lower bound. A bound crossing the budget alone leaves the exact peak undecided.

Scrape start and finish times matter because the replicas are read sequentially. Counter resets, process replacements, gaps, or a missing edge cannot be smoothed into an apparent pass. We sum across all manager identities and keep measurement and fixture traffic separate.

For exact counts, we used Metadata audit from every API server. We counted request arrival timestamps, deduplicated audit IDs, and included WATCH requests. Counting completed requests would move slow requests to the wrong window and could omit a watch still running when measurement ended. Kubernetes defines those audit fields and stages. Per-process counter deltas also have to agree with the audited requests inside the narrow and wide spans of their counter reads.

The release still had to pass the full workload

A short diagnostic that fits the API bound does not establish freshness, retention behavior, or the full soak. The development work exposed missing persisted deadlines at read-only handoffs; those needed their own correction. Another run exposed uncertainty between the sampling host’s clock and the manager’s clock. Passing a request bound could not make either problem pass.

The released 0.2.3 MySQL soak later made the distinction concrete: its driver completed, but an exact audit counted 7,236 requests in one sliding minute. The API requirement failed. The PostgreSQL cells and shorter successful diagnostics did not reverse that result.

Resources created together were refreshing together. The change shipped in 0.2.4 spreads a fleet’s refresh starts into slots, rather than repeatedly concentrating the same work. It preserves the scheduling and freshness constraints; it does not delete authorization checks to make a peak smaller.

The released artifacts then ran the full frozen workload. Each cell used a fresh lab with two managers. The soak included update rounds, churn, maintenance, retention faults, a registry outage, a delayed database, and a restart of all managers. The retained assessments record these exact peaks:

Platform and database	Soak: requests per sixty seconds	Approval backlog: requests per sixty seconds
Linux amd64, Kubernetes 1.35, PostgreSQL	6,735	5,890
Linux arm64, Kubernetes 1.35, PostgreSQL	6,483	6,161
Linux arm64, Kubernetes 1.37, MySQL	6,627	5,839

The three Ptah Operator 0.2.4 soak peaks and three approval-backlog peaks are below the 7,000-request budget. The plot shows separate measured cells, with no inferred speedup between them.

All nine capacity cells passed, including the overload cases. Overload doubles the fleet to 40 resources, outside the admitted profile. Its API peaks are retained as observations; that scenario is judged by refusal, safety, and recovery after returning to the supported workload. It is not a claim that 40 resources fit the 7,000-request budget.

The request budget also has a real boundary: API-server restarts can provoke a manager burst outside the measured profile. That follow-up remains tracked in #657.

The result is a qualified operating point, not a matched before-and-after speedup. The exact comparisons are the duplicated reads removed in controlled tests. The release measurements establish what the resulting operator did under the declared workload. Our qualification account covers the wider acceptance decision and deployment limits.

Put Ptah to work

Example files

← All articles