Overview of Continuous and Automated Data loading in Snowflake using Snowpipe

  • Blogs
  • Data Analytics
  • Overview of Continuous and Automated Data loading in Snowflake using Snowpipe
  • Share Post:

Snowpipe is Snowflake’s continuous data ingestion service.

Snowpipe loads data from files as soon as they are available in a staging area based on the COPY statement defined in a referenced pipe. The COPY statement identifies the source location (e.g. AWS S3 Bucket) of the data files and a target table to be used by Snowpipe to load data into Snowflake table.

Below mechanisms are available for detecting and loading the staged files:

1. Automating Continuous Data Loading Using Cloud Messaging
2. Calling Snowpipe REST Endpoints to Load Data

Automating Continuous Data Loading Using Cloud Messaging – Automated data loads uses event notifications (e.g. AWS SQS) for cloud storage to inform Snowpipe of the arrival of new data files to load. Snowpipe then copies the files into a queue, from which they are loaded into the target table in a continuous, serverless fashion based on parameters defined in the pipe.

Calling Snowpipe REST Endpoints to Load Data – Your client application calls a public REST endpoint with a list of data filenames and a referenced pipe name. If new data files matching the list are discovered in the stage, they are queued for loading. Snowflake-provided compute resources load data from the queue into a Snowflake table based on parameters defined in the pipe.

DataTerrain, with years of experience and reliable experts, is ready to assist. We have served more than 250-customers in the US and over 70 customers worldwide. We are flexible in working hours and do not need any long-term binding contracts.