October 23, 20265 min read

Bring Spark's Partitioned SQL into the Diff with Ptah

Spark keeps its partitioned-table definition outside the Ent diff. Ptah brings it into schema review while Spark keeps control of daily partitions.

Simon HartSolutions Engineer

Ptah helps you plan, review, and apply database migrations. Try in your browser

Spark keeps the partitioned definition of signing_nonces in SQL. Its documented Atlas Community workflow builds the migration diff from Ent. The SQL definition sits outside that diff.

That leaves a gap in schema review. An engineer can change the SQL file and still get a clean Ent diff. The result says nothing about whether the live table matches the partitioned definition in the repository.

Ptah brings that definition into the comparison alongside the Ent tables. A change to the SQL parent becomes a plan the engineer can inspect and apply. Spark continues to create and retire its daily partitions.

This proposal uses Ptah 0.15.0 and PostgreSQL 16.15. The configuration, integration patch, and reproduction records are available with the example. We have opened an issue and a draft pull request in the Spark repository. This is a proposal to the Spark maintainers, not an adoption announcement.

The table is visible; its partitioning is missing

Ent generates Spark’s client for signing_nonces, but its model does not express the table’s range partitioning. The external SQL file does. Spark’s migration guide explains the compromise: composing Ent and SQL requires Atlas Pro, so the Community configuration reads Ent alone and accepts the difference.

Keeping the SQL in Git documents the intended structure. It does not make the migration tool compare that structure with the database. Skipping DROP TABLE operations does not add the missing partition definition either. The engineer still has to check that part of the schema separately.

The daily partitions need a different boundary. Spark creates and removes signing_nonces_YYYYMMDD tables as retention advances. A schema tool should check the parent without treating those changing children as tables to remove.

Spark’s Ent-only comparison leaves the SQL partition definition outside the diff. Ptah combines Ent tables with the SQL parent, produces a reviewable change, and leaves dated partitions with Spark.

Put the SQL parent into the desired schema

We give Ptah the Ent tables and the external SQL as inputs to one desired schema. A small exporter reads Spark’s generated Ent metadata and omits its duplicate definition of signing_nonces. The SQL file supplies the parent, including PARTITION BY RANGE (id). Ent still generates the same Go client.

The composition is explicit:

data "composite_schema" "operator" {
  schema "public" {
    url = data.external_schema.ent.url
  }
  schema "public" {
    url = "file://external/schema.sql"
  }
}

The result contains 49 Ent-defined tables and the SQL-defined parent. Our full configuration excludes signing_nonces_[0-9]* from live comparison, leaving the parent in scope. The exporter uses Spark’s existing open-source Ent/Atlas libraries; this composition does not need Atlas Pro.

From spark/so/ent, we can now compare the live database with both sources:

ptah-compat schema diff --env local --from "$COMPOSITION_OPERATOR_DB_URL" --to env://src --schema public

Schemas are synced, no changes to be made.

Here, a clean diff includes the SQL parent’s partition definition. The application’s dated children can change without creating schema noise.

Turn a parent change into reviewed SQL

Suppose we add a nullable field to the partitioned table. In the example, case_study_note is a synthetic field added to a copy of the external SQL. The Ent export stays unchanged.

Ptah compares the combined desired schema with the live database and saves a plan. $LAB is the example workspace containing those schema files:

ptah schema plan \
  --db-url "$COMPOSITION_OPERATOR_DB_URL" \
  --schema-file "$LAB/operator-ent-identifiers.hcl" \
  --schema-file "$LAB/external-additive.sql" \
  --exclude 'signing_nonces_[0-9]*' \
  --output "$LAB/qualified-additive.plan.json"

The plan contains one statement:

ALTER TABLE "signing_nonces" ADD COLUMN "case_study_note" TEXT;

The SQL edit is now part of a reviewable database change. The plan contains no child-table deletion or detachment. We apply that saved plan:

ptah schema apply --db-url "$COMPOSITION_OPERATOR_DB_URL" --plan "$LAB/qualified-additive.plan.json" --auto-approve

PostgreSQL adds the column to the parent and its attached children. Their OIDs, partition bounds, and existing rows stay unchanged. Spark’s existing PurgeAndCreateSigningNoncePartitions function then advances retention: it retires the October 19 partition and creates October 22. The October 20 row stays in its original child.

Ptah’s next plan is empty. The parent change has converged, and Spark still controls the partition lifecycle.

Keep the migration history and the next Ent change

Bringing SQL into schema review does not require replacing Spark’s Ent clients or starting its migration history over. Ptah Compat applies the existing 187 operator migrations and the separate ephemeral history. Both keep their original SQL and atlas.sum files. Atlas Community and Ptah can read and continue each other’s recorded history.

The next ordinary Ent change also follows the familiar path: edit the schema, regenerate Ent metadata, and generate a migration. We add a nullable field to WalletSetting, then run:

ptah-compat migrate diff composition_probe --env authoring --dir "file://$LAB/qualified-authored-operator-history" --dev-url "$SCHEMA_DEV_DB_URL"

After replaying the original history, Ptah produces:

ALTER TABLE "public"."wallet_settings" ADD COLUMN "case_study_note" varchar;

Applying the migration preserves the existing row and constraints. A repeat diff against the extended directory reports:

The migration directory is synced with the desired state, no changes to be made

The authoring environment has a deliberate scope. The original history creates an ordinary signing_nonces table; a separate manual cutover makes it partitioned. Ordinary Ent migration generation therefore excludes that parent. The live comparison and parent-change plan include it. Every historical SQL statement still runs during replay.

Converting a populated ordinary table to a partitioned one remains a reviewed manual cutover. Ptah refuses that mismatch and an in-place change to the partition method or key. The data movement needs its own plan.

The integration patch connects these workflows to Spark’s startup commands, migration tasks, Docker image, and database test helper. It keeps Ent generation and application partition management in place. To follow the example, start with the complete configuration and setup.

Put Ptah to work

Example files

← All articles