Manage clusters and servers

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

In Snowflake, a warehouse is required for queries and for loading data into tables.^[3] Snowflake warehouses can be started and stopped at any time, and resized at any time, even while running.^[3] Snowflake's auto-suspend is enabled by default and suspends a warehouse that is inactive for a specified period, so a warehouse is not left running and consuming credits when there are no incoming queries.^[3] Snowflake uses per-second billing for warehouses, with a 60-second minimum each time a warehouse starts.^[3] Group the steps that need a warehouse and suspend it once at the end, rather than suspending and resuming it between short steps. (Meridian's view)

Apache Hop's Snowflake warehouse manager fails to resume a warehouse that is not in a suspended state, or to suspend one that is not in a started state.^[1] Make a step that starts or stops a warehouse tolerate one already in the wanted state, or check its state first, so that a rerun does not fail on it. (Meridian's view) In Snowflake, the additional resources from resizing a warehouse do not impact queries already running, and become available to queries that are queued or newly submitted.^[3] Over-provisioning, allocating more resources than required, may incur extra costs for unused or underutilised resources.^[4] A step that starts a cluster needs a matching step that stops it on every path out of the pipeline, failures included. (Meridian's view)

The method

A computer cluster is a group of autonomous computers that work together so that they can be viewed as a single system.^[5] The components of a cluster are usually connected through fast local area networks, with each node running its own instance of an operating system.^[5] The newest manifestation of cluster computing is cloud computing.^[5] In computing, elasticity is the degree to which a system adapts to workload changes by provisioning and de-provisioning resources in an autonomic manner, so that available resources match current demand as closely as possible.^[4] Elasticity is a defining characteristic that differentiates cloud computing from previously proposed distributed computing paradigms, such as grid computing.^[4]

In Snowflake, a warehouse's size specifies the compute resources available per cluster, and each step up to the next larger size doubles the credit usage for each full hour the warehouse runs.^[3] Talend's tAmazonRedshiftManage component manages Amazon Redshift clusters and snapshots.^[6]

Compared with the alternatives

Infrastructure as code (IaC) is the process of managing and provisioning computer data centre resources through machine-readable definition files, rather than through physical hardware configuration or interactive configuration tools.^[7] IaC definitions may exist in a version control system and may be scripts or declarative definitions, though IaC more often employs declarative approaches.^[7] In Snowflake, resizing a warehouse is primarily intended for improving query performance, and multi-cluster warehouses are recommended for fully automated scaling for concurrency.^[3]

Sources

1. [1]Snowflake warehouse manager | Apache Hopdocumentation
2. [2]Amazon EMR components | Talend Components for Jobs Helpdocumentation
3. [3]Overview of warehouses | Snowflake Documentationdocumentation
4. [4]Elasticity (computing) - Wikipediasecondary source
5. [5]Computer cluster - Wikipediasecondary source
6. [6]Amazon Redshift components | Talend Components for Jobs Helpdocumentation
7. [7]Infrastructure as code - Wikipediasecondary source