October 18, 20264 min read

Replace an Embedding Model and Keep a Way Back

Build a second vector generation with a real local model, compare retrieval, switch the application, and roll back when a corpus query ranks worse.

Simon HartSolutions Engineer

Ptah helps you plan, review, and apply database migrations. Try in your browser

The new model produced valid vectors. Our six-case evaluation passed. Then a query from the evaluation corpus ranked “Cancel your subscription” above “Request a subscription refund.” We returned the active pointer to the old generation and got the previous ranking back.

This article follows the first pgvector deployment with an actual model replacement. We used Ptah 0.13.0, PostgreSQL 17.11, pgvector 0.8.6, and Ollama 0.34.2 on ten synthetic help articles. It demonstrates building, evaluating, switching, and reversing the change; it is not a general ranking of the models.

Keep the old query path available

Our first generation uses the public all-minilm:22m model with 384-dimensional vectors in embedding_v1. The candidate uses nomic-embed-text:v1.5 with 768-dimensional vectors in embedding_v2. Both ran in our local model runtime. Their measured digests are retained in the model inventory and specifications.

Each specification names the model, endpoint, vector dimension, source fields, target column, and consistency policy. A new vector dimension needs its own compatible storage and application query. Changing only the endpoint would leave those parts of the application contract unresolved.

The source is mutable and uses a revision field with outbox consistency. This run did not modify the articles during the replacement; the earlier search article exercises source edits and stale-vector exclusion.

We fixed the evaluation corpus before asking either model. It contains six queries with required article IDs. We kept the same corpus and thresholds after seeing the results.

Build the candidate beside the active generation

After creating and activating the first generation, we prepared the second:

ptah inference probe --spec spec-v2.yaml --format json
ptah inference prepare --spec spec-v2.yaml --db-url "$DB_URL" --run-id help-v2
ptah inference backfill --spec spec-v2.yaml --db-url "$DB_URL" --run-id help-v2 --batch-rows 4
ptah inference catchup --spec spec-v2.yaml --db-url "$DB_URL" --run-id help-v2
ptah inference index --spec spec-v2.yaml --db-url "$DB_URL" --run-id help-v2
ptah inference verify --spec spec-v2.yaml --db-url "$DB_URL" --run-id help-v2

backfill finished: 10 scanned, 10 embedded, 0 skipped

The probe returned 768 finite values for its test input. Backfill embedded all ten source rows, the index became valid, and deterministic verification passed. The verification report explicitly says it did not read the stored vector values back; retain that distinction when interpreting “passed.”

While the candidate was ready but inactive, the application still used the old model and column. Its refund query returned:

2 | Request a subscription refund
3 | Cancel your subscription
1 | Reset your password

Building the candidate did not itself change the active pointer. This is a sequential check during the build workflow, not a sustained traffic test.

Compare under the same search settings

We froze six synthetic questions before evaluation. The refund question used by the application is one of them. Both measurements use explicit hnsw.ef_search=40,ivfflat.probes=1 connection settings, so the candidate is compared with the baseline under the same search configuration:

ptah inference evaluate --spec spec-v2.yaml --db-url "$EVAL_DB_URL" --corpus corpus.yaml --baseline "$PREVIOUS_GENERATION" --baseline-spec spec-v1.yaml --min-recall 1 --max-ndcg-regression 0.02

generation 933c952ecec7aad81dd644f7a0c49a02556f738cfde637518e60dfbe271bb74d against corpus 5f57e2e3c625
  - recall 1.000, MRR 0.917, NDCG 0.938 over 6 cases
  - the index agrees with an exhaustive search on 1.000 of results, over 6 cases
  - measured under hnsw.ef_search=40,ivfflat.probes=1

The matched-settings run exited 0. Both models scored recall 1.000, MRR 0.917, and NDCG 0.938 on this small corpus. The README shows how EVAL_DB_URL sets the same parameters for both measurements. A passing aggregate over six cases leaves plenty of queries untested.

Approve the pointer change

The specification requires approval of the exact plan. The first cutover attempt writes a plan and refuses to proceed:

ptah inference cutover --spec spec-v2.yaml --db-url "$DB_URL" --run-id help-v2 --stabilize-for 1h --plan-file cutover-v2.txt

plan b4ebf2e26b04
  - plan written to cutover-v2.txt; sign it with "ptah schema approve --plan cutover-v2.txt --key <key>"
cutover refused:
  - this policy requires an approval and none was given
error: cutover refused

We reviewed that plan and passed its complete plan: digest as PLAN_DIGEST. The short digest printed to the terminal is only a display abbreviation.

ptah inference cutover --spec spec-v2.yaml --db-url "$DB_URL" --run-id help-v2 --stabilize-for 1h --approve "$PLAN_DIGEST" --approver synthetic-release

The cutover result moved the pointer and retained the previous generation as a rollback candidate. This example uses a typed approval; it does not demonstrate cryptographic approver identity. The model migration guide describes the workflow and its evidence boundaries.

Check what the application actually returns

The application must use the model and column belonging to the selected generation. Its query also filters by the active pointer, current source revision, and upsert state. The Python client shows those predicates and the mapping from generation to model and column.

For the corpus question “Can I get my money back for the subscription?”, the new generation returned:

3 | Cancel your subscription
2 | Request a subscription refund
9 | Update the payment card

The refund article is still present, but it is no longer first. The corpus requires it within the first three results, so this ordering still passes. Equal aggregate scores can conceal a worse rank for an individual query. The visible change gives us a concrete reason to exercise rollback; this is not a held-out test or evidence of worse retrieval on every query.

Feed the previous generation, then return to it

A retained column is useful only while its data is current. We ran catch-up for the old generation within the stabilization window, then rolled back:

ptah inference catchup --spec spec-v1.yaml --db-url "$DB_URL" --run-id help-v1 --maintain-for 1h
ptah inference rollback --spec spec-v2.yaml --db-url "$DB_URL" --to "$PREVIOUS_GENERATION" --window 1h

queries now read ea3f652ee237abefa7a3a56b759dd81e4bf68de76bb6b95a8351193096ee9327, which replaced 933c952ecec7aad81dd644f7a0c49a02556f738cfde637518e60dfbe271bb74d

The old-model application query returned its original ranking:

2 | Request a subscription refund
3 | Cancel your subscription
1 | Reset your password

Both vector columns remained present: ten rows, 384 dimensions in v1, and 768 in v2. We did not retire either generation during this demonstration. The rollback and retirement guide explains when removing an old generation closes that recovery path.

The example separates vector validity, measured retrieval quality, and one application-visible result within the same corpus. Use a larger corpus drawn from your own requirements before a real rollout; keep unexposed queries for confirmation rather than tuning thresholds until a candidate passes. The complete record retains the successful checks and the reasons this run chose to return to v1.

Put Ptah to work

Example files

← All articles