Snowpipe is Snowflake’s continuous data ingestion service.
Snowpipe loads data from files as soon as they are available in a staging area based on the COPY statement defined in a referenced pipe. The COPY statement identifies the source location (e.g. AWS S3 Bucket) of the data files and a target table to be used by Snowpipe to load data into Snowflake table.
Below mechanisms are available for detecting and loading the staged files:
- Automating Continuous Data Loading Using Cloud Messaging
- Calling Snowpipe REST Endpoints to Load Data
Automating Continuous Data Loading Using Cloud Messaging – Automated data loads uses event notifications (e.g. AWS SQS) for cloud storage to inform Snowpipe of the arrival of new data files to load. Snowpipe then copies the files into a queue, from which they are loaded into the target table in a continuous, serverless fashion based on parameters defined in the pipe.
Calling Snowpipe REST Endpoints to Load Data – Your client application calls a public REST endpoint with a list of data filenames and a referenced pipe name. If new data files matching the list are discovered in the stage, they are queued for loading. Snowflake-provided compute resources load data from the queue into a Snowflake table based on parameters defined in the pipe.
DataTerrain, with years of experience and reliable experts, is ready to assist. We have served more than 250-customers in the US and over 70 customers worldwide. We are flexible in working hours and do not need any long-term binding contracts.