File Connector

Load CSV (and TSV or TXT delimited text), JSON, NDJSON, XML, Parquet and Excel files — plain, gzipped or zipped — into Snowflake from an SFTP server, an Amazon S3 bucket, a Windows or NAS share, or a folder on a machine in your network. Files are read in place: nothing on the source is copied, moved, renamed or deleted.

Setup at a glance

1. Create a connection to where the files are, and press Test Connection.
2. Add a file set for each table you want: which folder, which files, which format.
3. Create the pipeline: pick the destination, tick the file sets, save.
4. Choose how each task syncs: Full, or Incremental with Replace, Upsert or Append.
5. Sync and check the run's Files view.

Each step is covered below, followed by reference details and troubleshooting.

Choose your source

A hybrid agent runs inside your network, so the files and the credentials never leave it; only the rows you load reach Snowflake. Direct means Elglide's cloud connects to the source itself, with nothing to install.

Step 1 — Create the connection

Go to Connections → Add and pick the connector from Connector Category. Fill in the fields for your source below, then press Test Connection. A good test reads “✓ Found N file(s) in M folder(s) under …” and names the place it actually read; that is the quickest proof that everything after this step will work.

Create Connection runs the same test and saves only when it passes. If the source is briefly unreachable, Save Anyway saves it untested. Passwords, keys and secrets are write-only: once saved they are never shown again, and leaving the box blank when you edit keeps the saved one.

SFTP server

Amazon S3 bucket

The identity only needs to list the bucket and read objects. A minimal policy for one prefix:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::your-bucket",
      "Condition": { "StringLike": { "s3:prefix": "exports/daily/*" } } },
    { "Effect": "Allow", "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::your-bucket/exports/daily/*" }
  ]
}

Signing in with an IAM role (no stored keys)

1. Choose IAM role as the Method. The form shows Elglide's Principal and your account's External ID.
2. In AWS, create a role whose trust policy admits that Principal with that External ID:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Principal": { "AWS": "<Principal shown on the form>" },
       "Action": "sts:AssumeRole",
       "Condition": { "StringEquals": { "sts:ExternalId": "<External ID shown on the form>" } }
     }]
   }

3. Attach the list-and-read policy above to the role, and paste its ARN into Role ARN.

The External ID is fixed for your account, so another Elglide customer who learns your role's ARN still cannot use it. An IAM role works with Direct only; through a hybrid agent, use an access key.

Windows or NAS share (SMB)

No mounting and no agent configuration: type the share and an account, and the agent you pick opens it.

Folder on the Agent Machine

The connection itself only needs a Name (a label, e.g. Finance exports) and the Agent group. Which folder it reads is set on the agent's machine, by whoever installed the agent — deliberately not in the portal, so nobody signed in can point it at an arbitrary path.

The quickest start: the agent bundle comes with FILE_ROOT=./files. Create a files folder next to the bundle's agent.sh, put your files in it, and Test Connection will list them. The agent only ever reads from it.

Using a different folder

Set FILE_ROOT in the bundle's .env, then stop and start the agent:

FILE_ROOT=/srv/exports          # Linux / macOS
FILE_ROOT=C:\Elglide\files      # Windows: a real folder on this PC

./agent.sh stop && ./agent.sh start

A network share mounted on the agent's machine

Most shares are easier as a Network Share (SMB) connection. Mount one on the agent's machine only when its server requires Kerberos.

• Linux: mount the share with /etc/fstab or a systemd unit (so it survives a reboot) and point FILE_ROOT at it. Mount several shares under one parent folder and point FILE_ROOT at the parent: each share becomes a subfolder you can pick per file set.
• Windows: let Docker mount the share. In docker-compose.connectors.yml, uncomment the file-share volume and swap the volumes: line for the commented one; set FILE_SHARE_USER and FILE_SHARE_PASSWORD in .env. These stay on your machine.
• Kubernetes: there is no .env; point the elglide-source-files PersistentVolumeClaim at your storage.

Step 2 — Add file sets

A file set says which files make one table. Open the connection, and in its File Sets list press + (Add a file set). Add one file set per destination table; a connection can have as many as you like.

Patterns that work

Press Preview before saving: it lists exactly the files the patterns select, read live from the source, and warns when nothing matches.

Step 3 — Create the pipeline

1. Open the connection's Pipelines tab and choose the Snowflake destination in Select destination….
2. Under Schema Map, set the Snowflake schema the tables go into (it starts as FILES).
3. Tick the file sets to load, and press Save. Each ticked file set becomes a task, loading one table.
4. Optional: pick a Default Schedule. Manual only means a task runs when you press Sync.

Columns and types are detected from the newest files. Review them on a task's Schema tab, and press Refresh after the files' layout changes. A detected type is a proposal; it never overwrites a type you saved.

Step 4 — Choose how each task syncs

On a task, open Settings → Sync Mode. Pick Full or Incremental first:

• Full — every run reads every matching file and rebuilds the table in one transaction. The table always equals the folder; the only choice that removes rows of files that were deleted.
• Incremental (default) — every run reads only files that are new or changed since the last run, then the sync mode below decides what a changed file does to its earlier rows.

Example

orders_mon.csv (orders 1–3) and orders_tue.csv (orders 4–5) are loaded: five rows. Before the next run, orders_wed.csv arrives with orders 6–7, orders_tue.csv is regenerated with order 4 corrected and order 5 removed, and orders_mon.csv is deleted.

Safety built in

• A load that fails part-way rolls back; the table keeps its previous rows.
• A Full run never empties a table because a folder is empty: if no file matches, the run fails and the table is untouched.
• A Full task skips the run when nothing in the folder changed, so a frequent schedule over a quiet folder costs nothing.
• Upsert refuses a row with an empty key, rather than inserting a duplicate on every run.
• Switching a task from Append to Replace leaves earlier rows alone; they are replaced as each file is next read. To start a table over, use Sync → Full Sync.

Step 5 — Sync and check

Press Sync on a task. When it finishes, open the run in Sync History and choose Files to see every file it read, with its size, modified time, row count and why it was read (new, changed, grown, or full).

Every loaded table also gets four columns that trace each row back to its file:

JSON and NDJSON tables also get _ELGLIDE_RAW, the whole record as VARIANT, unless the file set's KeepRaw option is false.

How Elglide knows what is new

After a successful load, each file is remembered by its path, size and modified time; the next run skips it without opening it. A file with a new size or modified time is read again, and the sync mode decides what happens to its earlier rows. Nothing is remembered until the load commits, so a failed run simply picks the same files up next time.

• A file edited in place that keeps both its size and its timestamp looks unchanged. Some copy tools preserve timestamps; if yours does, write a new filename instead.
• A file deleted from the folder is not noticed by Incremental tasks: its rows stay. Only a Full task removes them.
• A file re-added under a new name is new work and loads again.
• To read everything again, use Sync → Full Sync.

Files that grow

A log or a running export is often appended to in place. With Changed files → Are appended to ticked, a file that grew is checked first: if its earlier contents are unchanged, only the new rows are read, and _ELGLIDE_LINE continues from the last number. If the earlier part changed, the whole file is read, as for any changed file.

Only Append saves the work: Replace and Full exist to replace a file's earlier rows, so they always read whole files, and compressed files are always read whole.

Format options

Optional JSON in the file set's Format options box. Leave it blank for the defaults — except XML, which must name its record element.

The CSV format reads .csv, .tsv, .txt and .tab files; set the Delimiter for anything not comma-separated.

Excel workbooks

• A file set reads one sheet of each .xlsx or .xls workbook. To load two sheets, add two file sets.
• Macro-enabled .xlsm and binary .xlsb workbooks are not read.
• Formulas load the value last saved in the workbook. Merged cells load their value in the first cell only.
• Blank rows after the data are ignored; a blank row between data rows loads as NULLs.

The run stops and names the sheet and cell for an error cell (#N/A, #REF!, #DIV/0!), a value under a column with no header, or a password-protected workbook.

Compressed files

Gzip and zip files are read in place on every kind of source; nothing needs unpacking first.

• gzip (orders.csv.gz): one file; the name under .gz tells the format.
• zip: every member is read as the file set's format. _ELGLIDE_FILE names the member, e.g. orders.zip!2026-09/orders.csv.
• A zip must hold only files of that format. __MACOSX/, .DS_Store and Thumbs.db are ignored; use an exclude pattern such as *.pdf to skip anything else.
• Archives inside archives, tar, bz2 and PGP files are not read.

When the files' columns change

Before loading, every file is checked against the task's saved columns. What happens next is set in the task's schema evolution settings: by default a new column is added to the table and a vanished column stops the run, naming the file and the column.

• CSV and Excel with a header are matched by column name, so reordered columns are fine.
• CSV and Excel without a header are matched by position, so any added or removed column stops the run.
• JSON, NDJSON, XML and Parquet are matched by name; a record missing a field gets NULL.

Run limits

A file set can carry per-run caps. The connector's defaults suit almost everyone; when a cap applies:

• Files per run and Bytes per run: files beyond the cap wait for the next run. A Full run ignores these, since a partial rebuild would lose rows.
• Rows per file written: rows per file uploaded to Snowflake.

Troubleshooting