Category: Snowflake

Overview of Continuous and Automated Data loading in Snowflake using Snowpipe

Snowpipe is Snowflake's continuous data ingestion service

Snowpipe is Snowflake’s continuous data ingestion service. Snowpipe loads data from files as soon as they are available in a staging area based on the COPY statement defined in a referenced pipe. The COPY statement identifies the source location (e.g. AWS S3 Bucket) of the data files and a target table to be used by Snowpipe to load data into Snowflake table. Below mechanisms are available for detecting and loading the staged files: Automating Continuous Data Loading Using Cloud Messaging Calling Snowpipe REST Endpoints to Load Data Automating Continuous Data Loading Using Cloud Messaging – Automated data loads uses event notifications (e.g. AWS SQS) for cloud storage to inform Snowpipe of the arrival of new data files to load. Snowpipe then copies the files into a queue, from which they are loaded into the target table in a continuous, serverless fashion based on parameters defined in the pipe.Calling Snowpipe REST Endpoints to Load Data – Your client application calls a public REST endpoint with a list of data filenames and a referenced pipe name. If new data files matching the list are discovered in the stage, they are queued for loading. Snowflake-provided compute resources load data from the queue into […]

Read More +

HOW TO CHANGE TRACKING USING TABLE STREAMS IN SNOWFLAKE

Streams in Snowflake

A stream is a Snowflake object that provides change data capture (CDC) capabilities to track the changes in a table. It records changes made to a table, including information about inserts, updates and deletes as well as metadata about each change. This process is referred to as change data capture (CDC). Types Of Streams: Standard table stream tracks all DML changes to the source table, including inserts, updates, and deletes (including table truncates). Append-only table stream tracks row inserts only. Update and Delete operations (including table truncates) are not recorded. An append-only stream returns the appended rows only and therefore can be a better choice than a standard stream for extract, load, transform (ELT) and similar scenarios that depend exclusively on row inserts. Append-only table streams can be combined with Snowflake tasks for continuous ELT workflows to process recently changed table rows. STALE Streams: A stream becomes stale if its point of offset gets positioned at a point earlier than the table’s data retention period. In order to avoid staleness we should ensure that data from the stream is consumed regularly. Consuming stream data advances the stream offset point as a result of which it avoids stream becoming stale. ENABLE […]

Read More +

An overview Snowflake Analytic Datawarehouse

Snowflade as a Saas

Snowflake is an analytic data warehouse provided as Software-as-a-Service (SaaS). It is not a packaged software offering that can be installed by a user.Snowflake manages all aspects of software installation and updates.Snowflake runs completely on cloud infrastructure. It cannot be run on private cloud infrastructures (on-premises or hosted).All components of Snowflake’s service (other than an optional command line client), run in a public cloud infrastructure. It uses virtual compute instances for its compute needs and a storage service for persistent storage of data. Snowflake provides a data warehouse that is faster, easier to use, and far more flexible than traditional data warehouse offerings.Snowflake’s data warehouse is not built on an existing database or “big data” software platform such as Hadoop. It uses a new SQL database engine with a unique architecture designed for the cloud. To the user, Snowflake has many similarities to other enterprise data warehouses, but also has additional functionality and unique capabilities. Snowflake’s data warehouse is a true SaaS offering. More specifically: ● There is no hardware (virtual or physical) for you to select, install, configure, or manage. ● There is no software for you to install, configure, or manage. ● On-going maintenance, management, and tuning is […]

Read More +