Schema evolution

Source schemas change — a team adds a column, widens a number, retires a field. Schema evolution is how Elglide reacts to those changes on the next sync without you rebuilding the pipeline by hand. You set the policy per task under Schema → Schema Evolution.

Elglide is net-additive: your destination is never silently destroyed. Columns are added to the warehouse when the source gains them, but a column is never automatically dropped and a change that could truncate data never applies on its own — the run stops and tells you instead.

The three settings

Each task has three independent controls, one per kind of column change. All three ship with a conservative default; you only change the ones you want to behave differently.

On new source column(s)

What happens when a column appears at the source that the destination doesn't have.

Refresh destination table (default) — add the column to the destination and reload the table so its history is populated.
Ignore — leave the column out; the sync continues without it. The column is marked excluded so it isn't re-flagged every run. Switch back to Refresh and re-sync to pull it in later.
Stop and notify — pause the run so a human decides.

On dropped source column(s)

What happens when a column that used to exist disappears from the source.

Stop and notify (default) — the conservative choice: pause so someone can confirm the drop was intentional.
Mark destination column deprecated — keep syncing the remaining columns and leave the destination column in place, so its historical data is preserved. It simply stops receiving new values.

On source column type change

What happens when an existing column changes type. Because every destination is Snowflake, many source type changes don't actually change the destination type at all — see what counts as a type change below.

Widen or keep destination (never narrow) (default) — a genuine widening (for example a whole number becoming a decimal, a date becoming a timestamp, or any column becoming text) rebuilds the destination column at the wider type and reloads so the values repopulate. A change that would narrow the column (decimal back to whole number) keeps the destination at its current, wider type — the narrower values still fit, so the sync just continues. Only a truly incompatible change (for example text back to a number, with no safe common type) falls through to stop and notify.
Stop and notify — pause on any real type change.

Widening always follows the supertype rule: the destination column is only ever moved to a type that holds both the old and the new values. Elglide never narrows a column automatically, so a type change can't quietly truncate what's already landed.

Renamed columns

A rename can't be told apart from an unrelated drop-and-add with certainty, so Elglide doesn't guess — guessing wrong would corrupt data. A renamed source column is handled as a drop plus an add, governed by the two settings above: the new name flows through On new source column(s) and the old name through On dropped source column(s).

With Refresh + Mark deprecated, a rename flows through hands-off: the new column is added and backfilled while the old one is kept with its history frozen. With the defaults, it stops and notifies so you can confirm.

File sources

For a file set the saved Schema Map is the snapshot and the headers of the files a run reads are the live schema. Before any row is written, every file in the batch is compared with the map, and the first two settings apply exactly as they do for a database:

What the files did	Refresh / Mark deprecated	Ignore	Stop and notify
A new column appears	Added to the destination at the type the files suggest (a numeric column lands as a number, not text), added to the map, and the run continues.	Recorded in the map as excluded; the files keep carrying it and it is never read.	The run fails naming the file and the column.
A column vanishes	Marked deprecated: excluded from the map, the destination column and its history kept, the run continues with NULL for it.	—	The run fails naming the file and the column (the default).

There is no type-change setting for files: a CSV declares no types and a JSON value is typed per record, so there is nothing to compare. The setting is not shown on a file task.

Two cases always stop, whatever the settings say, because there is nothing to evolve to. Files that disagree with each other in one batch have no single shape; narrow the file set. A delimited file with no header row is read by position, so a column cannot be matched by name; re-run Detect Schema to accept the new shape, or give the files a header.

REST API sources

For a REST API endpoint the saved Schema Map is the snapshot and the fields of the records a run reads are the live schema. There is no catalog to ask, so every run samples its first page — with the same watermark the run itself uses — and compares the fields it finds, with the types inferred from their values, against the map before any row is written. All three settings apply:

What the records did	Refresh / Mark deprecated / Widen	Ignore	Stop and notify
A new field appears	Added to the destination at the type its values suggest, added to the map, and the run continues.	Added to the map excluded; the destination is untouched. Tick it on the Schema tab to include it.	The run stops, naming the field.
A mapped field is absent from every sampled record	Marked deprecated: excluded from the map, the destination column and its history kept, the run continues with NULL for it.	—	The run stops, naming the field and saying how many records were sampled.
A field's values changed type	Widened when the new type holds the old values (a number that is now text); kept as is when the destination already holds the new values (text that is now numbers); stops only on a change with no common type.	—	The run stops, naming the field and both types.

A JSON record may omit an optional field, so “absent” is judged across every record of the sampled page and the run log says how many that was. An API that omits fields is best paired with Mark deprecated, which keeps the column and its history; a field that comes back can be ticked again on the Schema tab. An excluded field is known to the map, so it is never reported as new again.

What counts as a “type change”

The destination is always Snowflake, and Elglide maps source types onto a small set of generous Snowflake types. So the most common source type changes land on the same Snowflake type and need no action — the sync just continues:

int → bigint — both are the same Snowflake number.
varchar(50) → varchar(100) — both are the same Snowflake text.
Even a source narrowing like bigint → int — the destination column already holds it.

The On source column type change setting only governs the rarer cases where the destination type genuinely has to move — a whole number becoming a decimal, a date becoming a timestamp, and so on.

Picking a policy

Hands-off, additive pipeline — Refresh + Mark deprecated + Widen: new and widened columns flow through automatically; nothing is ever destroyed.
Change-controlled pipeline — keep the Stop-and-notify defaults so every real schema change pauses for review.
Fixed column set — Ignore new columns so only the columns you mapped ever reach the warehouse.

See it in the app

Open any task, go to Schema → Schema Evolution. Set the policy per column-change type; it takes effect on the next sync. Open your tasks (sign-in required).