Data quality audits

A sync that reports success has moved the rows it believed it needed to move. An audit is the independent check on that belief: after the sync, Elglide asks the source a question, asks Snowflake the same question, and compares the two answers. You configure it per pipeline under Task → Data Quality.

An audit reads scalars, not rows. Each rule runs one aggregate query on each side and compares the results. It reads no rows and writes none, so audits never affect your rows-synced usage.

Audit rules

Every pipeline starts with one rule you can’t remove: COUNT(*). It’s the cheapest check that catches the failure that matters most — rows that never arrived. You can edit its threshold, and you can add as many rules beside it as the table warrants.

Built-in functions

Pick an aggregate and a column; Elglide writes the SQL for both sides.

COUNT(*) — did every row land?
COUNT(DISTINCT column) — did the cardinality survive? Catches duplicates a plain count can’t.
COUNT(*) WHERE column IS NULL — did nulls appear (or vanish) in transit?
SUM(column) / AVG(column) — the value check. A row count can match while the money doesn’t.
MIN(column) / MAX(column) — the boundary check. MAX on a timestamp is the fastest way to see that a pipeline has quietly stopped advancing.

Target column names aren’t always source column names — Snowflake uppercases unquoted identifiers, and columns can be renamed in the column map. Elglide resolves the mapping for you, so a rule on ModifiedDate queries MODIFIEDDATE on the target without you thinking about it.

Custom SQL

When the question is business-shaped rather than column-shaped — “do shipped orders total the same on both sides?” — write the source statement yourself. Elglide validates it, then shows you the Snowflake statement it will run against the target so nothing is translated behind your back. Give the rule a name; the full SQL stays available as its tooltip in the rule list.

SELECT SUM(Quantity * UnitPrice) FROM dbo.OrderDetails WHERE Status = 'shipped'

Thresholds

A threshold is how much difference you’re willing to call normal. Two forms:

Percentage — the delta as a share of the source value. Right for counts and sums that grow.
Absolute — a flat number. Right when “5% of a billion” is not a rounding error.

A custom rule with no threshold of its own inherits the default rule’s — the rule list marks those (inherited). Date-typed MIN and MAX rules are exact-match only: a timestamp either agrees or it doesn’t, so no threshold applies and none is offered.

Retiring a rule

A rule that has never run can be deleted outright. Once a rule has verdict history, it can only be made inactive — it stops running but keeps its history, so the record of what you were checking, and what it said, doesn’t disappear. Re-check Active to bring it back.

Verdicts

Verdict	What it means
Pass	The two scalars agree within the threshold. If both sides evaluate to NULL — an all-NULL or empty column faithfully synced — that is also a pass, not a failure.
Mismatch	The delta exceeds the threshold. This is the only verdict that drives the on-failure ladder below.
Inconclusive	The check couldn’t produce a verdict — exactly one side evaluated to NULL, or the audit query itself failed (timeout, connection blip, SQL error). Deliberately not a mismatch: a check that couldn’t run is not evidence your data is wrong, so it never triggers a retry and never fails the run.

Each verdict is recorded with the source value, the target value, and the delta. Date MIN/MAX results are shown as timestamps, not raw tick counts.

When audits run

Audits run after a sync completes — scheduled or manual, it makes no difference. Rules run in order and stop at the first mismatch: the failing rule has already started the on-failure ladder, and running the rest would only pile up duplicate retries of the same pipeline. An inconclusive rule doesn’t stop anything; the remaining rules still run.

Audit Now

Audit Now runs every rule on demand, right now. It compares live source data against the target as of the last completed sync — so anything written to the source since then shows up as drift. That’s the point: it tells you how far the target has fallen behind. To compare like with like, run a sync first.

An ad-hoc audit deliberately skips the whole failure ladder — no retry, no escalation, no notification. It answers a question you asked; it doesn’t start work you didn’t.

Row filters and audits

If a pipeline has a source row filter, the audit applies the same filter — but only when the mechanism the run actually used was Full or RowHash. Those are the only two that can observe a row’s absence and remove it from the target. Under Watermark or a change stream, a row that falls out of the filter is never revisited, so the filter can’t be enforced end-to-end — Elglide suppresses it on the audit side too, and says so in the run log, rather than reporting phantom drift forever.

On audit failure

A mismatch is often transient: the sync read the source mid-transaction, or a late-arriving batch landed a second after the extract. So Elglide’s first response is to try again, and only then to tell you. Three controls, on the Data Quality tab:

Auto retry <mechanism> up to N times — re-runs the pipeline with its normal mechanism. The original run is attempt 1, so the retries are attempts 2 through N+1. Several rules failing at once collapse into a single retry, never a stampede.
Last full sync if all retries fail — one final attempt forced to a Full load, which reconciles drift an incremental mechanism can’t see. This is runtime-only: it does not rewrite the pipeline’s configuration, so the next scheduled run resumes the mechanism you chose. Hidden when the pipeline already runs Full — there’s nothing to escalate to.
Auto-notify after all retries fail — raise the notification once the ladder is exhausted.

One incident, one email. The notification fires only when the ladder is terminal. A mismatch that a retry reconciles stays silent — you hear about drift that survived every attempt to fix it, not about every wobble on the way there.

The whole sequence is written to the run log, whatever the outcome: each retry attempt, the Full escalation, and the notification — including the case where auto-notify is off, so “why didn’t I get an email?” has an answer in the log rather than in a mailbox.

Notifications

A terminal mismatch raises the Audit mismatch event, carrying the source and target values. Who receives it is a per-user choice under Account → Notifications:

Set your baseline per event type — it applies to every pipeline.
Add pipeline exceptions for the ones that shouldn’t follow that baseline, including muting a pipeline entirely.

Two deliberate defaults worth knowing. Audit mismatch is opt-in — only sync failure is on for a user who has never touched the page, because defaulting everything on trains people to filter the mail, which loses the failures too. And audit mismatch is never throttled: repeat sync failures inside a short window are the same incident restated, but a second audit mismatch is a genuinely new finding, so it is never collapsed into the first.

Reading the history

Each rule carries its own verdict table — newest first, so the top row is the current state. The Audit History selector switches every table at once between the last 5, the latest only, the last 10, or a time range (24 hours, 7 days, 30 days). Sync History carries an Audit column too, so you can see a run’s verdict without leaving the run list.

See it in the app

Open any task and go to Data Quality. Add rules with the + beside “Audit Rules”, set the failure policy under “On Audit Failure”, and use Audit Now to check a pipeline immediately. Open your tasks (sign-in required).